DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities:
In paragraph 0044 line 9, “screen region 212” should read “screen region 210”.
In paragraph 0044 line 11, “application 210” should read “application 212”.
In paragraph 0056 line 4, “toe” should read “the”.
Appropriate correction is required.
The use of the term “Adobe”, “Firefly”, “GPT-4”, “Dall-E”, “Codex”, and “Wi-Fi” in paragraphs 0026 and 0075, which is a trade name or a mark used in commerce, has been noted in this application. The term should be accompanied by the generic terminology; furthermore the term should be capitalized wherever it appears or, where appropriate, include a proper symbol indicating use in commerce such as ™, SM , or ® following the term.
Although the use of trade names and marks used in commerce (i.e., trademarks, service marks, certification marks, and collective marks) are permissible in patent applications, the proprietary nature of the marks should be respected and every effort made to prevent their use in any manner which might adversely affect their validity as commercial marks.
Claim Objections
Claims 12 and 20 are objected to because of the following informalities:
Claim 12 recites the limitation "the standalone content editing application" in line 10. There is insufficient antecedent basis for this limitation in the claim. For the sake of examination, line 2 of claim 12 is read as “a standalone content editing application”.
In claim 12 line 12, “based in” should read “based on”.
In claim 20 line 2, “at least on” should read “at least one”.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 16-17 and 20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Acha et al. (US 20260045010 A1), hereinafter Acha.
Regarding claim 16, Acha teaches one or more computer-readable storage media storing instructions that, responsive to execution by a processing device, causes the processing device to perform operations comprising (Paragraph 0169, 0171 – “The processor may be a special-purpose processor selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including, but not limited to, any type of disk including optical disks… a computer-usable or computer-readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device”):
receiving input image data as at least one layer taken from digital content displayed in a user interface (Paragraph 0026, 0054, 0057 – “The initial image and the original prompt (and optionally user input) are provided… The initial image includes one or more objects…the user interface module 202 generates presets that are displayed with an initial image”; Note: the initial image and original prompt are equivalent to the input image data, and the different objects in the image are different layers. It is received and displayed by the user interface module);
forming a prompt that includes the at least one layer for processing using generative artificial intelligence (AI) by a machine-learning model (Paragraph 0026 – “The initial image and the original prompt (and optionally user input) are provided to an LLM or other text-generation model…The LLM rewrites the prompt. A rewritten prompt corresponds to the respective original prompt, i.e. specifies the same one or more (image modification) tasks for modifying the initial image…If the user input is not provided and/or a user did not select an object in the initial image, the LLM may generate a rewritten prompt that associates the original prompt with the only object in the image or a most prominent object in the image”; Note: a rewritten prompt is formed for processing by a machine learning model. The prompt includes an object in the image, which is a layer);
obtaining generative digital content from the machine-learning model responsive to the prompt (Paragraph 0027, 0039, 00059 – “A machine-learning model is selected from a set of machine-learning models to be used for generating an output image… The selected machine-learning model generates an output image in response to the rewritten prompt… The user interface module 202 generates graphical data for displaying an output image”; Note: the output image, which includes generative digital content, is generated by the machine-learning model responsive to the prompt and is received by the user interface module);
and communicating the generative digital content to a source of the digital content (Paragraph 0034, 0051-0052 – “an initial image may be captured by the user device 115a and transmitted with user input and a prompt to the media application 103a on the media server 101, which generates an output image that is transmitted to the media application 103b on the user device 115a for display…media application 103, stored in memory 237, that includes a user interface module 202…The user interface module 202 generates graphical data for displaying a user interface that includes images”; Note: the output image, which is equivalent to the generative digital content, is communicated/transmitted for display in the media application, which is on the user device that captured the input image) as an additional layer for inclusion as part of the digital content and display in the user interface (Paragraph 0061-0062 – “FIG. 3A illustrates an example user interface 300 with an initial image 302 that includes a fence 304 in front of a subject 306 and a preset 308 for removing the fence, according to some embodiments described herein. The user interface module 202 performs object recognition on the initial image 302 and identifies that the initial image 302 includes the fence 304 and the subject 306. The user interface module 202 generates a “fence removal” preset 308 that, when selected, commands the machine-learning model to generate an output image without the fence 304…FIG. 3B illustrates an example user interface 350 with an output image 355 that includes the subject 357 (corresponding to the subject 306 of FIG. 3A) without the fence 304 of FIG. 3A”; Note: the filled region without the fence is equivalent to the generative digital content in this case, and it is added as a region/layer to the initial image in order to produce an output image without the fence).
Regarding claim 17, Acha teaches the one or more computer-readable storage media as described in claim 16. Acha further teaches wherein the digital content is displayed in a content-editing window (Paragraph 0061 – “FIG. 3A illustrates an example user interface 300 with an initial image 302 that includes a fence 304 in front of a subject 306 and a preset 308 for removing the fence, according to some embodiments described herein. The user interface module 202 performs object recognition on the initial image 302 and identifies that the initial image 302 includes the fence 304 and the subject 306. The user interface module 202 generates a “fence removal” preset 308 that, when selected, commands the machine-learning model to generate an output image without the fence 304”; Note: the initial image, which is the digital content, is displayed in a window for content-editing, as shown in Fig. 3A) and the communicating causes the generative digital content to be added to the content-editing window as the additional layer (Paragraph 0061-0062 – “FIG. 3A illustrates an example user interface 300 with an initial image 302 that includes a fence 304 in front of a subject 306 and a preset 308 for removing the fence, according to some embodiments described herein. The user interface module 202 performs object recognition on the initial image 302 and identifies that the initial image 302 includes the fence 304 and the subject 306. The user interface module 202 generates a “fence removal” preset 308 that, when selected, commands the machine-learning model to generate an output image without the fence 304…FIG. 3B illustrates an example user interface 350 with an output image 355 that includes the subject 357 (corresponding to the subject 306 of FIG. 3A) without the fence 304 of FIG. 3A”; Note: the filled region without the fence is equivalent to the generative digital content in this case, and it is added as a region/layer to the initial image in the editing window in order to produce an output image without the fence).
Regarding claim 20, Acha teaches one or more computer-readable storage media as described in claim 16. Acha further teaches detecting an edit to the at least one layer of the digital content (Paragraph 0037, 0069 – “a user may circle an object in the initial image and provide a textual request to change a color of the object to a different color, change features of a region to different features, or replace an original object with a new object…FIG. 5B illustrates an example user interface 525 that receives user input 531 on the initial image 527 and a text field for receiving an original prompt 533 from a user, according to some embodiments described herein. The user provides the user input 531 by circling the dog and adding “Pink” to the text field to create an original prompt 533. By circling the dog and adding “Pink” to the text field 533, the user is indicating that the user wants to change the dog to a pink dog. The user selects the arrow button 545 to generate the output image”; Note: the process occurs responsive to the user editing (circling) digital content of a layer (the object/dog) in the initial image) and wherein the receiving is performed automatically and without user intervention responsive to the detecting (Paragraph 0004, 0025-0026, 0043 – “A computer-implemented method includes receiving an initial image and an original prompt from a user, wherein the original prompt includes a request to modify the initial image…User input may also be provided, such as selection of an object (e.g., where a user taps different objects in an initial image until the object that the user wants to modify is highlighted, circling an object, etc.). The initial image and the original prompt (and optionally user input) are provided to an LLM or other text-generation model…Processor 235 can be one or more processors and/or processing circuits to execute program code and control basic operations of the computing device 200…processor may perform its functions in real-time”; Note: the inputs are received automatically (by a computer/processor on its own) in real-time after an edit (circling an object) is detected).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-7, and 11-15 are rejected under 35 U.S.C. 103 as being unpatentable over Acha in view of Kumar et al. (US 11217005 B1), hereinafter Kumar.
Regarding claim 1, Acha teaches a method (Paragraph 0004 – “computer-implemented method”) comprising:
receiving, by a processing device, input image data, the input image data describing pixels displayed on a display device (Paragraph 0026, 0057, 0073 – “The initial image and the original prompt (and optionally user input) are provided… the user interface module 202 generates presets that are displayed with an initial image… The segmenter 204 segments initial images…the segmenter 204 generates a segmentation mask that identifies object pixels or region pixels associated with the one or more objects or a region based on segmenting the one or more objects or the region”; Note: the initial image and original prompt are equivalent to the input image data. It is received by the user interface module, which is part of a processing device shown in Fig. 2. The image, comprising pixels, are displayed on the device);
forming, by the processing device, a prompt based on the input image data for processing using generative artificial intelligence (AI) by a machine-learning model (Paragraph 0026 – “The initial image and the original prompt (and optionally user input) are provided to an LLM or other text-generation model…The LLM rewrites the prompt. A rewritten prompt corresponds to the respective original prompt, i.e. specifies the same one or more (image modification) tasks for modifying the initial image, but the rewritten prompt further meets at least one of the following criteria: it is more clear instruction to the machine-learning model, it is a more concise instruction to the machine-learning model, and/or its wording/instruction(s) improves the performance of the machine-learning model”; Note: a rewritten prompt is formed based on the original prompt and initial image, which are equivalent to the input image data, for processing by a machine learning model);
obtaining, by the processing device, generative digital content from the machine-learning model responsive to the prompt (Paragraph 0027, 0039, 00059 – “A machine-learning model is selected from a set of machine-learning models to be used for generating an output image… The selected machine-learning model generates an output image in response to the rewritten prompt… The user interface module 202 generates graphical data for displaying an output image”; Note: the output image, which includes generative digital content, is generated by the machine-learning model responsive to the prompt and is received and displayed by the user interface module);
and presenting, by the processing device, the generative digital content for display in a user interface concurrently with at least a portion of the pixels (Paragraph 0061-0062 – “FIG. 3A illustrates an example user interface 300 with an initial image 302 that includes a fence 304 in front of a subject 306 and a preset 308 for removing the fence…The user interface module 202 generates a “fence removal” preset 308 that, when selected, commands the machine-learning model to generate an output image without the fence 304…FIG. 3B illustrates an example user interface 350 with an output image 355 that includes the subject 357 (corresponding to the subject 306 of FIG. 3A) without the fence 304 of FIG. 3A, according to some embodiments described herein. The user interface module 202 receives the output from the machine-learning module 208 (e.g., from the non-structure and non-shape preserving machine-learning model) and displays the output image 355”; Note: the output image, which includes generative digital content, is presented on the user interface module for display. The area with the removed fence is equivalent to the generative digital content, and it is presented concurrently with the pixels from the initial image that are not modified).
Acha does not teach receiving input image data from a frame buffer. However, Kumar teaches receiving input image data from a frame buffer (Col. 14 lines 58-59 – “the method may include reading a current value for a pixel in a frame buffer”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Acha to incorporate the teachings of Kumar to receive the input image data from a frame buffer because frame buffers store pixel and color data of images for display, and thus, accessing the images through frame buffers makes it easier to obtain and use information about images that are already displayed, such as when a user is selecting input images from a digital image gallery.
Regarding claim 2, Acha in view of Kumar teaches the method as described in claim 1. Acha does not teach wherein the pixels are rendered to the frame buffer through execution of a standalone content editing application. However, Kumar teaches wherein the pixels are rendered to the frame buffer through execution of a standalone content editing application (Col. 4 lines 43-44 and 53-54, Col. 11 lines 8-16, Col. 12 lines 12-14 and 26-27 – “The rendering manager 112 can allocate a frame buffer 116…The frame buffer is the size of the display and stores a value for each pixel to be rendered on the display…the content design system 602 also includes rendering manager 112. In general, rendering manager 112 marks, fills in, textures, and/or colors pixels of a vector graphics object using rendering state machine 400 and the triangle information generated by tessellation manager 104. For example, the rendering manager 112 uses control triangles, interior triangles, and color triangles to render a vector graphics object using the rendering state machine based on a current state of frame buffer 116…the components 604-612 of the content design system 602 may, for example, be implemented as one or more stand-alone applications…the components of the content design system 602 may be implemented in a digital image editing application”; Note: execution of the content design system, which is a stand-alone editing application, causes pixels to be rendered to a frame buffer for display). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Acha to incorporate the teachings of Kumar to have a standalone editing application render pixels to a frame buffer because in order to efficiently display an image on a screen, the pixels would need to be stored in a frame buffer. Moreover, image editing applications, like the media application in Acha, usually require rendering and displaying images on the screen, and thus, it would be useful for the application to render the image pixels to a frame buffer.
Regarding claim 4, Acha in view of Kumar teaches the method as described in claim 1. Acha further teaches wherein the receiving, the forming, the obtaining, and the presenting are performed, automatically and without user intervention, in real time (Paragraph 0004, 0043 – “A computer-implemented method includes receiving an initial image and an original prompt from a user, wherein the original prompt includes a request to modify the initial image. The method further includes selecting, based on the original prompt, a machine-learning model from a set of machine-learning models. The method further includes providing the original prompt and the initial image as input to a large language model (LLM). The method further includes receiving, from the LLM and based on the original prompt and the initial image, a rewritten prompt. The method further includes providing the rewritten prompt and the initial image as input to the selected machine-learning model. The method further includes generating, by the selected machine-learning model, an output image that satisfies the rewritten prompt…Processor 235 can be one or more processors and/or processing circuits to execute program code and control basic operations of the computing device 200…processor may perform its functions in real-time”; Note: the operations are performed automatically (by a computer/processor on its own) in real-time) responsive to detecting an edit to digital content associated with the pixels as displayed in the user interface (Paragraph 0037, 0069 – “a user may circle an object in the initial image and provide a textual request to change a color of the object to a different color, change features of a region to different features, or replace an original object with a new object…FIG. 5B illustrates an example user interface 525 that receives user input 531 on the initial image 527 and a text field for receiving an original prompt 533 from a user, according to some embodiments described herein. The user provides the user input 531 by circling the dog and adding “Pink” to the text field to create an original prompt 533. By circling the dog and adding “Pink” to the text field 533, the user is indicating that the user wants to change the dog to a pink dog. The user selects the arrow button 545 to generate the output image”; Note: the process occurs responsive to the user editing (circling) digital content associated with the pixels in the initial image).
Regarding claim 5, Acha in view of Kumar teaches the method as described in claim 4. Acha further teaches communicating the generative digital content, automatically and without user intervention, for display in a window in the user interface associated with a source of the input image data (Paragraph 0030, 0034, 0051-0052 – “the media server 101 sends and receives data to and from one or more of the user devices 115a, 115n via the network 105…an initial image may be captured by the user device 115a and transmitted with user input and a prompt to the media application 103a on the media server 101, which generates an output image that is transmitted to the media application 103b on the user device 115a for display…media application 103, stored in memory 237, that includes a user interface module 202…The user interface module 202 generates graphical data for displaying a user interface that includes images”; Note: the output image, which is equivalent to the generative digital content, is communicated/transmitted for display in the media application, which includes a user interface associated with the source of the input image data since the user interface is on the user device that captured the input image. The communication occurs automatically (by the server)).
Regarding claim 6, Acha in view of Kumar teaches the method as described in claim 1. Acha further teaches receiving text data describing the generative digital content to be generated and wherein the forming of the prompt includes the text data (Paragraph 0026 – “The initial image and the original prompt (and optionally user input) are provided to an LLM or other text-generation model…The LLM rewrites the prompt. A rewritten prompt corresponds to the respective original prompt, i.e. specifies the same one or more (image modification) tasks for modifying the initial image”; Note: the original prompt is equivalent to the text data, and forming the rewritten prompt includes the original prompt).
Regarding claim 7, Acha in view of Kumar teaches the method as described in claim 6. Acha further teaches selecting the machine-learning model from a plurality of candidate machine learning models based on the input image data, the text data, or a user selection (Paragraph 0027 – “A machine-learning model is selected from a set of machine-learning models to be used for generating an output image. In some embodiments, the machine-learning model is selected by a media application based on the original prompt. For example, if the user input circles a cat and the original prompt is ‘make pink,’ the media application selects a shape-preserving machine-learning model”).
Regarding claim 11, Acha in view of Kumar teaches the method as described in claim 1. Acha further teaches wherein the pixels correspond to a layer of digital content (Paragraph 0061, 0073 – “FIG. 3A illustrates an example user interface 300 with an initial image 302 that includes a fence 304 in front of a subject 306 and a preset 308 for removing the fence, according to some embodiments described herein. The user interface module 202 performs object recognition on the initial image 302 and identifies that the initial image 302 includes the fence 304 and the subject 306. The user interface module 202 generates a “fence removal” preset 308 that, when selected, commands the machine-learning model to generate an output image without the fence 304… the segmenter 204 generates a segmentation mask that identifies object pixels or region pixels associated with the one or more objects or a region based on segmenting the one or more objects or the region”; Note: in the initial image, there are pixels in a region surrounding the fence, which is a layer since it is separate from the modifications of “fence removal”) and wherein the presenting of the generative digital content is added as an additional layer to the digital content (Paragraph 0061-0062 – “FIG. 3A illustrates an example user interface 300 with an initial image 302 that includes a fence 304 in front of a subject 306 and a preset 308 for removing the fence, according to some embodiments described herein. The user interface module 202 performs object recognition on the initial image 302 and identifies that the initial image 302 includes the fence 304 and the subject 306. The user interface module 202 generates a “fence removal” preset 308 that, when selected, commands the machine-learning model to generate an output image without the fence 304…FIG. 3B illustrates an example user interface 350 with an output image 355 that includes the subject 357 (corresponding to the subject 306 of FIG. 3A) without the fence 304 of FIG. 3A”; Note: the filled region without the fence is equivalent to the generative digital content in this case, and it is added as a region/layer to the initial image in order to produce an output image without the fence).
Regarding claim 12, Acha teaches a system (Paragraph 0041 – “Computing device 200 can be any suitable computer system, server, or other electronic or hardware device. In one example, computing device 200 is media server 101 used to implement the media application 103a. In another example, computing device 200 is a user device 115”; Note: the computing device is equivalent to the system) comprising:
a content editing application executable by a processing device to edit digital content and display the digital content in a content-editing window in a user interface (Fig. 8A, Paragraph 0044, 0051, 0059 – “Memory 237 can store software operating on the computing device 200 by the processor 235, including a media application 103… media application 103, stored in memory 237, that includes a user interface module 202, a segmenter 204, a prompt engine 206, and a machine-learning module 208… The user interface module 202 generates graphical data for displaying an output image. In some embodiments, the user interface module 202 includes options for enabling multiple edits to an initial image”; Note: the media application is equivalent to the content editing application, which is executable by the processor of the computing device. Fig. 8A shows an example of a content-editing window in a user interface);
PNG
media_image1.png
417
215
media_image1.png
Greyscale
Screenshot of Fig. 8A (taken from Acha)
one or more machine-learning models configured to implement generative artificial intelligence to produce generative digital content as a digital image (Paragraph 0120 – “The machine-learning module 208 trains machine-learning models to generate output images based on rewritten prompts and initial images. In some embodiments, the machine-learning module 208 receives a command from the prompt engine 206 to generate the output image based on a machine-learning model selected by the prompt engine 206 along with the initial image, the rewritten prompt, and user input if available. In some embodiments, the machine-learning model is selected from a structure-preserving machine-learning model, a shape-preserving machine-learning model, or a non-structure and non-shape preserving machine-learning model”);
and a generative artificial intelligence (AI) manager system executable by the processing device to perform operations including (Paragraph 0041, 0043, 0051, 0120 – “Computing device 200 can be any suitable computer system, server, or other electronic or hardware device. In one example, computing device 200 is media server 101 used to implement the media application 103a. In another example, computing device 200 is a user device 115…Processor 235 can be one or more processors and/or processing circuits to execute program code and control basic operations of the computing device 200…FIG. 2 illustrates an example media application 103, stored in memory 237, that includes a user interface module 202, a segmenter 204, a prompt engine 206, and a machine-learning module 208…The machine-learning module 208 trains machine-learning models to generate output images based on rewritten prompts and initial images. In some embodiments, the machine-learning module 208 receives a command from the prompt engine 206 to generate the output image based on a machine-learning model selected by the prompt engine 206 along with the initial image, the rewritten prompt, and user input”; Note: the computing device is also a generative AI manager system):
receiving input image data (Paragraph 0026, 0034 – “The initial image and the original prompt (and optionally user input) are provided…receive an initial image captured by the user device”; Note: the initial image and original prompt are equivalent to the input image data);
forming a prompt for processing by the one or more machine- learning models to produce the generative digital content based on the input image data (Paragraph 0026, 0039 – “The initial image and the original prompt (and optionally user input) are provided to an LLM or other text-generation model…The LLM rewrites the prompt. A rewritten prompt corresponds to the respective original prompt, i.e. specifies the same one or more (image modification) tasks for modifying the initial image, but the rewritten prompt further meets at least one of the following criteria: it is more clear instruction to the machine-learning model, it is a more concise instruction to the machine-learning model, and/or its wording/instruction(s) improves the performance of the machine-learning model… The selected machine-learning model generates an output image in response to the rewritten prompt”; Note: a rewritten prompt is formed based on the original prompt and initial image, which are equivalent to the input image data, for processing by a machine learning model);
displaying the generative digital content in a generative window in the user interface (Paragraph 0061-0062 – “FIG. 3A illustrates an example user interface 300 with an initial image 302 that includes a fence 304 in front of a subject 306 and a preset 308 for removing the fence…The user interface module 202 generates a “fence removal” preset 308 that, when selected, commands the machine-learning model to generate an output image without the fence 304…FIG. 3B illustrates an example user interface 350 with an output image 355 that includes the subject 357 (corresponding to the subject 306 of FIG. 3A) without the fence 304 of FIG. 3A, according to some embodiments described herein. The user interface module 202 receives the output from the machine-learning module 208 (e.g., from the non-structure and non-shape preserving machine-learning model) and displays the output image 355”; Note: the output image, which includes generative digital content, is displayed on the user interface module in a window shown in Fig. 3B. The area with the removed fence is equivalent to the generative digital content);
and communicating the generative digital content generated by the one or more machine-learning models to the standalone content editing application (Paragraph 0034, 0039, 0051-0052 – “an initial image may be captured by the user device 115a and transmitted with user input and a prompt to the media application 103a on the media server 101, which generates an output image that is transmitted to the media application 103b on the user device 115a for display… The selected machine-learning model generates an output image in response to the rewritten prompt…media application 103, stored in memory 237, that includes a user interface module 202…The user interface module 202 generates graphical data for displaying a user interface that includes images”; Note: the output image, which is equivalent to the generative digital content, is communicated/transmitted for display in the media application, which is the content editing application) for inclusion in the content-editing window (Paragraph 0061-0062 – “FIG. 3A illustrates an example user interface 300 with an initial image 302 that includes a fence 304 in front of a subject 306 and a preset 308 for removing the fence…The user interface module 202 generates a “fence removal” preset 308 that, when selected, commands the machine-learning model to generate an output image without the fence 304…FIG. 3B illustrates an example user interface 350 with an output image 355 that includes the subject 357 (corresponding to the subject 306 of FIG. 3A) without the fence 304 of FIG. 3A, according to some embodiments described herein. The user interface module 202 receives the output from the machine-learning module 208 (e.g., from the non-structure and non-shape preserving machine-learning model) and displays the output image 355”; Note: the output image is included in the content-editing window, as shown in the transition from Fig. 3A to Fig. 3B).
PNG
media_image2.png
393
493
media_image2.png
Greyscale
Screenshot of Fig. 3A and 3B (taken from Acha)
Acha does not teach receiving input image data rendered from digital content associated with the standalone content editing application. However, Kumar teaches receiving input image data (Col. 14 lines 58-59 – “the method may include reading a current value for a pixel in a frame buffer”) rendered from digital content associated with the standalone content editing application (Col. 4 lines 43-44 and 53-54, Col. 11 lines 8-16, Col. 12 lines 12-14 and 26-27 – “The rendering manager 112 can allocate a frame buffer 116…The frame buffer is the size of the display and stores a value for each pixel to be rendered on the display…the content design system 602 also includes rendering manager 112. In general, rendering manager 112 marks, fills in, textures, and/or colors pixels of a vector graphics object using rendering state machine 400 and the triangle information generated by tessellation manager 104. For example, the rendering manager 112 uses control triangles, interior triangles, and color triangles to render a vector graphics object using the rendering state machine based on a current state of frame buffer 116…the components 604-612 of the content design system 602 may, for example, be implemented as one or more stand-alone applications…the components of the content design system 602 may be implemented in a digital image editing application”; Note: the image data is rendered for display from the content design system, which is a stand-alone editing application). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the received input image data from the user device of Acha (Paragraph 0034 – “receive an initial image captured by the user device”) could have been substituted for the received input image data from the standalone editing application of Kumar because both the received input image data from the user device and received input image data from the standalone editing application serve the purpose of processing the image data. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of generating new or different image data based on the input image data. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the received input image data from the user device of Acha for the received input image data from the standalone editing application of Kumar according to known methods to yield the predictable result of generating new or different image data based on the input image data.
Regarding claim 13, Acha in view of Kumar teaches the system as described in claim 12. Acha further teaches wherein the operations of the generative artificial intelligence manager system further include receiving text data describing the generative digital content to be generated and wherein the forming of the prompt includes the text data (Paragraph 0026 – “The initial image and the original prompt (and optionally user input) are provided to an LLM or other text-generation model…The LLM rewrites the prompt. A rewritten prompt corresponds to the respective original prompt, i.e. specifies the same one or more (image modification) tasks for modifying the initial image”; Note: the original prompt is equivalent to the text data, and forming the rewritten prompt includes the original prompt).
Regarding claim 14, Acha in view of Kumar teaches the system as described in claim 12. Acha further teaches wherein the operations of the generative artificial intelligence manager system further include selecting the machine-learning model from a plurality of candidate machine learning models based on the input image data or text data entered via a user interface describing the generative digital content to be generated (Paragraph 0027 – “A machine-learning model is selected from a set of machine-learning models to be used for generating an output image. In some embodiments, the machine-learning model is selected by a media application based on the original prompt. For example, if the user input circles a cat and the original prompt is ‘make pink,’ the media application selects a shape-preserving machine-learning model”).
Regarding claim 15, Acha in view of Kumar teaches the system as described in claim 12. Acha further teaches wherein the operations of the generative artificial intelligence manager system further include detecting an edit to the digital content (Paragraph 0037, 0069 – “a user may circle an object in the initial image and provide a textual request to change a color of the object to a different color, change features of a region to different features, or replace an original object with a new object…FIG. 5B illustrates an example user interface 525 that receives user input 531 on the initial image 527 and a text field for receiving an original prompt 533 from a user, according to some embodiments described herein. The user provides the user input 531 by circling the dog and adding “Pink” to the text field to create an original prompt 533. By circling the dog and adding “Pink” to the text field 533, the user is indicating that the user wants to change the dog to a pink dog. The user selects the arrow button 545 to generate the output image”; Note: the process occurs responsive to the user editing (circling) digital content associated with the pixels in the initial image) and wherein the receiving is performed automatically and without user intervention responsive to the detecting (Paragraph 0004, 0025-0026, 0043 – “A computer-implemented method includes receiving an initial image and an original prompt from a user, wherein the original prompt includes a request to modify the initial image…User input may also be provided, such as selection of an object (e.g., where a user taps different objects in an initial image until the object that the user wants to modify is highlighted, circling an object, etc.). The initial image and the original prompt (and optionally user input) are provided to an LLM or other text-generation model…Processor 235 can be one or more processors and/or processing circuits to execute program code and control basic operations of the computing device 200…processor may perform its functions in real-time”; Note: the inputs are received automatically (by a computer/processor on its own) in real-time after an edit (circling an object) is detected).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Acha in view of Kumar and Jonsson (US 9058699 B2), hereinafter Jonsson.
Regarding claim 3, Acha in view of Kumar teaches the method as described in claim 2. Acha does not teach communicating the generative digital content to the standalone content editing application for display in a window associated with the standalone content editing application that includes the pixels, the communicating performed from a window associated with the presenting. However, Jonsson teaches communicating the generative digital content to the standalone content editing application for display in a window associated with the standalone content editing application that includes the pixels, the communicating performed from a window associated with the presenting (Col. 5 lines 49-50, Col. 14 lines 40-51, Col. 18 lines 18-21 – “an image compositing application (e.g., a stand-alone application…a user interface window 600 of an image compositing application displays various frames that may be visible to a user during an image compositing operation…active images (i.e. those currently available for selection and merging into a composite image) are displayed in a frame on the left side of window 600, user interface elements (e.g., slider bars) usable to invoke various image editing tools (e.g., to adjust the scale, color, or other parameters of an image or an object thereof) are displayed in a frame along the right side of window 600, and the composite image in progress is displayed in a large central frame in window 600…image compositing application 820 may be configured to render merged and/or modified images into a different window than input images”; Note: digital content is communicated to a window of the image compositing application, which is a stand-alone editing application. The communicating occurs from a separate window where other images are presented. Generative digital content was previously taught by Acha in the rejection of claim 1). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Acha to incorporate the teachings of Jonsson to communicate the generative digital content to a window of the standalone content editing application for display for the benefit of differentiating the final output and making it easier for the user to use or manually edit it separate from the machine learning model.
Claims 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over Acha in view of Kumar and Kang (CN 111143015 B), hereinafter Kang.
Regarding claim 8, Acha in view of Kumar teaches the method as described in claim 1. Acha does not teach presenting a plurality of options specifying a source of the input image data and wherein the receiving is performed using a select option from the plurality of options. However, Kang teaches presenting a plurality of options specifying a source of the input image data and wherein the receiving is performed using a select option from the plurality of options (Paragraph 0031-0032, 0034, 0036 – “Step 102: In response to the first input, display multiple screenshot category options. Based on the user's initial input, multiple screenshot category options can be displayed on the currently displayed target interface… Step 103: Receive the user's input for the target category option, which can be any one of the multiple screenshot category options… Step 104: In response to the input selection, generate a screenshot image containing the target content; wherein, the target content is the content in the target interface corresponding to the target category option”; Note: the target category options are equivalent to the plurality of options specifying a source of the input image data, which is the screenshot in this case. After the user selects an option, a screenshot is received). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Acha to incorporate the teachings of Kang to present options specifying a source of the input image data to receive the input image data for the benefit of making it more accessible and user-friendly for the user to be able to import the desired data. “This increases the selectivity of screenshot methods by displaying multiple screenshot category options, reduces the tedious operations brought about by image editing, better meets the different screenshot needs of users, is convenient and fast, improves the user experience, and facilitates the differentiation and organization of screenshot images” (Kang: paragraph 0019).
Regarding claim 9, Acha in view of Kumar and Kang teaches the method as described in claim 8. Acha does not teach wherein the plurality of options includes a full screen option, a select window option, a select screen region option, or an application option usable to select a content editing application. However, Kang teaches wherein the plurality of options includes a full screen option, a select window option, a select screen region option, or an application option usable to select a content editing application (Paragraph 0031-0032, 0044– “Step 102: In response to the first input, display multiple screenshot category options. Based on the user's initial input, multiple screenshot category options can be displayed on the currently displayed target interface… When a notification bar pop-up message is present on the interface, the target category option can be a notification bar option. The specific process of receiving the user's selection input for the target category option is as follows: when the notification bar is displayed on the target interface, the user's selection input for the notification bar option is received”; Note: the options include a notification bar option, which is a type of selection screen region option since the notification bar is a region of the screen). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Acha to incorporate the teachings of Kang to have the options include a select screen region option because it helps the user choose a specific portion of the screen they may be interested in as input data, which is convenient and user-friendly. “This increases the selectivity of screenshot methods by displaying multiple screenshot category options, reduces the tedious operations brought about by image editing, better meets the different screenshot needs of users, is convenient and fast, improves the user experience, and facilitates the differentiation and organization of screenshot images” (Kang: paragraph 0019).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Acha in view of Kumar and Meng et al. (SDEDIT: GUIDED IMAGE SYNTHESIS AND EDITING WITH STOCHASTIC DIFFERENTIAL EQUATIONS), hereinafter Meng.
Regarding claim 10, Acha in view of Kumar teaches the method as described in claim 1. Acha further teaches presenting a control that is user selectable via the user interface (Paragraph 0057 – “the user interface module 202 generates presets that are displayed with an initial image. The user interface module 202 generates a preset as a selectable icon that, when selected, causes an output image to be generated that satisfies the description in the preset. In some embodiments, the user interface module 202 provides the same set of presets in response to a user selecting an edit button and/or a suggestions button”; Note: presets are controls that are user selectable). Acha does not teach specifying an amount of fidelity to be applied by the machine-learning model in generating the generative digital content and wherein the prompt includes the amount. However, Meng teaches presenting a control via the user interface to specify an amount of fidelity to be applied by the machine-learning model in generating the generative digital content and wherein the prompt includes the amount (Paragraph 3 on Page 5 – “We note that the quality of the guide may affect the overall quality of the synthesized image. For reasonable guides, we find that t0 ∈ [0.3,0.6] works well. However, if the guide is an image with only white pixels, then even the closest “realistic” samples from the model distribution can be quite far, and we must sacrifice faithfulness for better realism by choosing a large t0. In interactive settings (where user draws a sketch-based guide), we can initialize t0 ∈ [0.3,0.6], synthesize a candidate with SDEdit, and ask the user whether the sample should be more faithful or more realistic; from the responses, we can obtain a reasonable t0 via binary search”; Note: faithfulness is equivalent to fidelity, and t0 is equivalent to the prompt in this case. The user interface is implied since a user response is obtained). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Acha to incorporate the teachings of Meng to have a control for specifying an amount of fidelity to be applied by the machine-learning model for the benefit of creating an output image that better adheres to user preferences. It is also important to include the amount in the prompt to ensure that the machine-learning model knows to apply that specific amount when generating the output.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Acha in view of Jonsson.
Regarding claim 18, Acha teaches the one or more computer-readable storage media as described in claim 17. Acha does not teach displaying the generative digital content in a window separate from the content-editing window responsive to the obtaining and before the communicating. Jonsson teaches displaying the generative digital content in a window separate from the content-editing window responsive to the obtaining and before the communicating (Col. 5 lines 49-50, Col. 14 lines 40-51, Col. 18 lines 18-21 – “an image compositing application (e.g., a stand-alone application…a user interface window 600 of an image compositing application displays various frames that may be visible to a user during an image compositing operation…active images (i.e. those currently available for selection and merging into a composite image) are displayed in a frame on the left side of window 600, user interface elements (e.g., slider bars) usable to invoke various image editing tools (e.g., to adjust the scale, color, or other parameters of an image or an object thereof) are displayed in a frame along the right side of window 600, and the composite image in progress is displayed in a large central frame in window 600…image compositing application 820 may be configured to render merged and/or modified images into a different window than input images”; Note: digital content is displayed/rendered to a window separate from where the content was edited/modified. Generative digital content was previously taught by Acha in the rejection of claim 16. The displaying occurs after obtaining (since it must be obtained in order to display it) but before communicating/displaying to the content-editing window (since the modified content is yet to be displayed in that window)). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Acha to incorporate the teachings of Jonsson to display the generative digital content to a separate window before the communicating for the benefit of highlighting the final output and allowing the user to view it before using it or making manual edits to it, separate from the machine learning model.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Acha in view of Meng.
Regarding claim 19, Acha teaches the one or more computer-readable storage media as described in claim 16. Acha further teaches presenting a control that is user selectable via the user interface (Paragraph 0057 – “the user interface module 202 generates presets that are displayed with an initial image. The user interface module 202 generates a preset as a selectable icon that, when selected, causes an output image to be generated that satisfies the description in the preset. In some embodiments, the user interface module 202 provides the same set of presets in response to a user selecting an edit button and/or a suggestions button”; Note: presets are controls that are user selectable). Acha does not teach specifying an amount of fidelity to be applied by the machine-learning model in generating the generative digital content and wherein the prompt includes the amount. However, Meng teaches presenting a control via the user interface to specify an amount of fidelity to be applied by the machine-learning model in generating the generative digital content and wherein the prompt includes the amount (Paragraph 3 on Page 5 – “We note that the quality of the guide may affect the overall quality of the synthesized image. For reasonable guides, we find that t0 ∈ [0.3,0.6] works well. However, if the guide is an image with only white pixels, then even the closest “realistic” samples from the model distribution can be quite far, and we must sacrifice faithfulness for better realism by choosing a large t0. In interactive settings (where user draws a sketch-based guide), we can initialize t0 ∈ [0.3,0.6], synthesize a candidate with SDEdit, and ask the user whether the sample should be more faithful or more realistic; from the responses, we can obtain a reasonable t0 via binary search”; Note: faithfulness is equivalent to fidelity, and t0 is equivalent to the prompt in this case. The user interface is implied since a user response is obtained). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Acha to incorporate the teachings of Meng to have a control for specifying an amount of fidelity to be applied by the machine-learning model for the benefit of creating an output image that better adheres to user preferences. It is also important to include the amount in the prompt to ensure that the machine-learning model knows to apply that specific amount when generating the output.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Seth et al. (US 20250342630 A1) teaches a method of image generation by a generative model based on user input and construction of a prompt. Wu et al. (US 20260045012 A1) teaches a method of image generation using a selected machine-learning model based on a user request and prompt.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE HAU MA whose telephone number is (571)272-2187. The examiner can normally be reached M-Th 7-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE HAU MA/ Examiner, Art Unit 2617
/KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights