Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Election/Restrictions
Applicant’s election without traverse of Group I in the reply filed on 5/26/26 is acknowledged.
Allowable Subject Matter
Claims 4, 6 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim 4 recites “receiving the natural language text input providing the instructions for modifying the digital image comprises receiving the natural language text input providing the instructions to remove a first object portrayed in the digital image, the first object holding a second object; modifying the digital image comprises removing the first object from the digital image while leaving the second object in the digital image based the editing region including the first object and excluding the second object; and receiving the user input for modifying the editing region within the digital image comprises receiving the user input for adding the second object into the editing region.” Where natural language text input specifically removes a first object holding a second object, leaving the second object, and input is received for adding the second object into the editing region. The prior art does not teach these limitations in combination with the other limitations.
Claim 6 recites “further comprising providing, to the large language model, one or more in-context examples having a format for indicating objects targeted by natural language text inputs, wherein determining, using the large language model, the object that is targeted by the natural language text input for modification comprises generating, using the large language model, a natural language text output that indicates the object in the format of the one or more in-context examples.” Where in-context examples are sent to the LLM so the LLM outputs the object in the example format. The prior art does not teach these limitations in combination with the other limitations.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5, 7-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu U.S. Patent/PG Publication 20220067992 in view of Liu U.S. Patent/PG Publication 20240338862.
Regarding claim 1 (independent):
A computer-implemented method comprising: (Xu [0070] Any suitable computing system or group of computing systems can be used for performing the operations described herein. For example, FIG. 9 depicts an example of computing device 900 that may be at least a portion of image editing system 100. The implementation of the computing device 900 could be used for one or more of the image editor 105 or the user system 135. In an embodiment, a single image editing system 100 having devices similar to those depicted in FIG. 9 (e.g., a processor, a memory, etc.) combines the one or more operations and data stores depicted as separate subsystems in FIG. 1. Further, FIG. 8 illustrates a cloud computing system 800 by which at least a portion of the image editing system 100 may be offered.).
receiving, from a client device, a digital image and natural language text input providing instructions for modifying the digital image (Xu [0059] To illustrate and only as a non-limiting example, while the natural language-based image editor 110 is executing, the user 520 uploads the source image 505, and then speaks the following natural language request 510: “Please remove the girl from the photo and increase the light.”)
generating, using a (Xu [0059] which passes the source image 505 and the natural language request 510 to the operation classifier model 115. The operation classifier model 115 evaluates the source image 505 and the natural language request 510 using trained neural networks to infer that the natural language request 510 describes two separate image editing operations. The operation classifier model 115 segments the natural language request 510 into two phrases: “Please remove the girl from the photo” and “increase the light.” Further, the operation classifier model 115 infers that the phrase “Please remove the girl from the photo” corresponds to the “inpaint_obj” image editing operation. The operation classifier model 115 also infers that the phrase “increase the light” corresponds to the “Brightness” image editing operation. [0060] The operation classifier model 115 outputs “inpaint_obj” and “Brightness” as multiple inferred image editing operations.)
the executable action code being compatible with an editing application (Xu [0058] FIG. 5 depicts an example of a process 500, in which a user 520 executes a natural language-based image editor 110 )(Xu [0060] The grounding model 120 receives the inferred image editing operations of “inpaint_obj” and “Brightness” as inputs (along with other inputs, such as the natural language request 510 and the source image 505, not shown). The grounding model 120 classifies the “inpaint_obj” image editing operation as a local image editing operation, and then grounds the “inpaint_obj” image editing operation to the pixels of the source image 505 that depicts a girl. The grounding model 120 generates an image mask that hides the background pixels but reveals the pixels depicting the girl in the source image 505 (or vice versa). Likewise, the grounding model 120 classifies the “Brightness” image editing operation as a global image editing operation, and then grounds the “Brightness” image editing operation to the entire source image 505. In this case, the entire source image 505 is the image mask.)
modifying the digital image by executing the executable action code via the editing application and providing the modified digital image for display via a graphical user interface of the client device (Xu [0061] The grounding model 120 outputs two masks to the operation modular network 125. Mask #1 is a local image mask for the “inpaint_obj” image editing operation. Mask #2 is a global image mask for the “Brightness” image editing operation. Each mask is received at the operation modular network 125, which is trained to perform the two inferred image editing operations. The operation modular network 125 includes a submodule for the “inpaint_obj” image editing operation and another separate and distinct submodule for the “Brightness” image editing operation. The “Brightness” submodule infers a parameter (e.g., a brightness value indicating a magnitude of applying brightness to the source image 505) for the “Brightness” image editing operation because the natural language request 510 does not specify any parameters. Each of the “inpaint_obj” and the “Brightness” image editing operations are executed sequentially (or in parallel) to generate the modified source image 515.)
Xu does not teach a LLM. In a related field of endeavor, Liu teaches:
generating, using a large language model, executable action code for modifying the digital image in accordance with the instructions of the natural language text input, (Liu [0137] In step S461, in each round of dialogue, obtaining the interaction context 410 (i.e., the current dialogue data) between the current user and the image processing system, and understanding the user's requirement expressed in the interaction context 410 by using the large language model 420. Thereby, a multi-round requirement understanding is achieved.)(Liu [0139] In step S463, executing the action flow by calling the cross-modal image generating model 430 and/or the cross-modal image editing model 440 to obtain a target image.).
the executable action code being compatible with an editing application (Liu [0144] the image processing system generates the target image 552 and the explanation data thereof “I have changed the background of the image of film A into a starry sky using an image editing tool, and the path of the new image is image/ccc.png.” using the method 300 of the embodiments of the present disclosure.)
modifying the digital image by executing the executable action code via the editing application and providing the modified digital image for display via a graphical user interface of the client device (Liu Fig. 5)(Liu [0147] According to embodiments of the present disclosure, the image processing system can carry out multiple rounds of interaction with the user without limiting the number of rounds, and continuously guide the user to express complete and precise real requirement during the interaction, and gradually approach the ideal result that satisfies the user's requirement in multiple steps, by which both the image generating task and the image editing task can be completed, and the universality of the AI image creation application is significantly improved.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use a LLM as taught by Liu. The rationale for doing so would have been a simple substitution of one known element for another to obtain predictable results where Xu uses a neural network and Liu uses a LLM, where both are using artificial intelligence to interpret natural language to edit an image. Therefore it would have been obvious to combine Liu with Xu to obtain the invention.
Regarding claim 2:
The computer-implemented method of claim 1, has all of its limitations taught by Xu in view of Liu. Xu further teaches wherein generating the executable action code for modifying the digital image comprises generating the executable action code for modifying an editing region from the digital image using one or more editing operations of the editing application (Xu [0060] The grounding model 120 classifies the “inpaint_obj” image editing operation as a local image editing operation, and then grounds the “inpaint_obj” image editing operation to the pixels of the source image 505 that depicts a girl. The grounding model 120 generates an image mask that hides the background pixels but reveals the pixels depicting the girl in the source image 505 (or vice versa). Likewise, the grounding model 120 classifies the “Brightness” image editing operation as a global image editing operation, and then grounds the “Brightness” image editing operation to the entire source image 505. In this case, the entire source image 505 is the image mask.).
Regarding claim 3:
The computer-implemented method of claim 2, has all of its limitations taught by Xu in view of Liu. Xu further teaches further comprising:
receiving, via the client device, user input for modifying the editing region within the digital image (Xu [0059] “Please remove the girl from the photo”).
generating, using the large language model, additional executable action code that incorporates the modified editing region and the one or more editing operations and modifying the modified digital image by executing the additional executable action code via the editing application (Xu Further, the operation classifier model 115 infers that the phrase “Please remove the girl from the photo” corresponds to the “inpaint_obj” image editing operation. The operation classifier model 115 also infers that the phrase “increase the light” corresponds to the “Brightness” image editing operation. [0060] The operation classifier model 115 outputs “inpaint_obj” and “Brightness” as multiple inferred image editing operations. The grounding model 120 receives the inferred image editing operations of “inpaint_obj” and “Brightness” as inputs (along with other inputs, such as the natural language request 510 and the source image 505, not shown). The grounding model 120 classifies the “inpaint_obj” image editing operation as a local image editing operation, and then grounds the “inpaint_obj” image editing operation to the pixels of the source image 505 that depicts a girl. The grounding model 120 generates an image mask that hides the background pixels but reveals the pixels depicting the girl in the source image 505 (or vice versa). Likewise, the grounding model 120 classifies the “Brightness” image editing operation as a global image editing operation, and then grounds the “Brightness” image editing operation to the entire source image 505. In this case, the entire source image 505 is the image mask.).
Regarding claim 5:
The computer-implemented method of claim 1, has all of its limitations taught by Xu in view of Liu. Xu further teaches further comprising:
determining, using the (Xu [0059] Further, the operation classifier model 115 infers that the phrase “Please remove the girl from the photo” corresponds to the “inpaint_obj” image editing operation.).
and determining, using a segmentation model, an editing region of the digital image that corresponds to the object, wherein generating the executable action code for modifying the digital image in accordance with the instructions of the natural language text input comprises generating the executable action code for modifying the editing region of the digital image (Xu Further, the operation classifier model 115 infers that the phrase “Please remove the girl from the photo” corresponds to the “inpaint_obj” image editing operation. The operation classifier model 115 also infers that the phrase “increase the light” corresponds to the “Brightness” image editing operation. [0060] The operation classifier model 115 outputs “inpaint_obj” and “Brightness” as multiple inferred image editing operations. The grounding model 120 receives the inferred image editing operations of “inpaint_obj” and “Brightness” as inputs (along with other inputs, such as the natural language request 510 and the source image 505, not shown). The grounding model 120 classifies the “inpaint_obj” image editing operation as a local image editing operation, and then grounds the “inpaint_obj” image editing operation to the pixels of the source image 505 that depicts a girl. The grounding model 120 generates an image mask that hides the background pixels but reveals the pixels depicting the girl in the source image 505 (or vice versa). Likewise, the grounding model 120 classifies the “Brightness” image editing operation as a global image editing operation, and then grounds the “Brightness” image editing operation to the entire source image 505. In this case, the entire source image 505 is the image mask.).
Xu does not teach a LLM. In a related field of endeavor, Liu teaches:
large language model, (Liu [0137] In step S461, in each round of dialogue, obtaining the interaction context 410 (i.e., the current dialogue data) between the current user and the image processing system, and understanding the user's requirement expressed in the interaction context 410 by using the large language model 420. Thereby, a multi-round requirement understanding is achieved.)(Liu [0139] In step S463, executing the action flow by calling the cross-modal image generating model 430 and/or the cross-modal image editing model 440 to obtain a target image.).
Therefore, it would have been obvious before the effective filing date of the claimed invention to use a LLM as taught by Liu. The rationale for doing so would have been a simple substitution of one known element for another to obtain predictable results where Xu uses a neural network and Liu uses a LLM, where both are using artificial intelligence to interpret natural language to edit an image. Therefore it would have been obvious to combine Liu with Xu to obtain the invention.
Regarding claim 7:
The computer-implemented method of claim 5, has all of its limitations taught by Xu in view of Liu. Xu further teaches wherein determining, using the segmentation model, the editing region of the digital image that corresponds to the object comprises generating, using the segmentation model, a set of vertices that outline the object within the digital image (Xu [0067] FIG. 7 depicts an example of various stages of process 600 as described with respect to FIG. 6. In example 705, the input refers to the source image inputted by the user. The natural language request is “crop out everyone except the pastor and the guy in the black short sleeve shirt.” The source image and the natural language request pass through the operation classifier model, which infers that the user's intended image editing operation is “inpaint_obj,” which is an operation that removes an object from an image. The inferred image editing operation then is passed through the grounding model, which generates the “predicted” image mask shown in example 705 of FIG. 7. The “gt mask” refers to the ground truth image mask. The operation modular network receives the predicted image mask, the source image, and the natural language request to generate the output image (under “gt output”) by executing the submodule specifically configured for performing the “inpaint_obj” image editing operation. The output image is modified according to the user's natural language request because the individuals other than the “guy in the black short sleeve shirt” have been removed.).
Regarding claim 8:
The computer-implemented method of claim 1, has all of its limitations taught by Xu in view of Liu. Xu further teaches further comprising:
determining, from the natural language text input using the large language model, one or more editing actions for modifying the digital image (Xu [0060] The grounding model 120 classifies the “inpaint_obj” image editing operation as a local image editing operation, and then grounds the “inpaint_obj” image editing operation to the pixels of the source image 505 that depicts a girl. The grounding model 120 generates an image mask that hides the background pixels but reveals the pixels depicting the girl in the source image 505 (or vice versa). Likewise, the grounding model 120 classifies the “Brightness” image editing operation as a global image editing operation, and then grounds the “Brightness” image editing operation to the entire source image 505. In this case, the entire source image 505 is the image mask.).
and determining one or more executable code examples for the editing application that correspond to the one or more editing actions, wherein generating the executable action code for modifying the digital image in accordance with the instructions of the natural language text input comprises generating the executable action code for modifying the digital image using one or more editing operations of the editing application that correspond to the one or more editing actions (Xu [0061] The grounding model 120 outputs two masks to the operation modular network 125. Mask #1 is a local image mask for the “inpaint_obj” image editing operation. Mask #2 is a global image mask for the “Brightness” image editing operation. Each mask is received at the operation modular network 125, which is trained to perform the two inferred image editing operations. The operation modular network 125 includes a submodule for the “inpaint_obj” image editing operation and another separate and distinct submodule for the “Brightness” image editing operation. The “Brightness” submodule infers a parameter (e.g., a brightness value indicating a magnitude of applying brightness to the source image 505) for the “Brightness” image editing operation because the natural language request 510 does not specify any parameters. Each of the “inpaint_obj” and the “Brightness” image editing operations are executed sequentially (or in parallel) to generate the modified source image 515.)(Xu [0039] In some implementations, the prediction layer 250 includes one or more fully connected layers that are trained to generate the final output 255. As only a non-limiting example, the prediction layer 250 is trained using a multi-label cross-entropy loss technique. Any other text categorization or text classification techniques may be used to infer the image editing operations from the natural language request 205 and/or the source image 210. Additionally, while the operation classifier model 115 is described above as being trained using supervised learning techniques, the present disclosure is not limited thereto. In some implementations, the operation classifier model 115 is trained using unsupervised learning techniques for text analytics, such as, for example, clustering or latent semantic indexing (LSI).).
Regarding claim 9:
The computer-implemented method of claim 8, has all of its limitations taught by Xu in view of Liu. Xu further teaches further comprising providing, to the large language model, one or more in-context examples having a format for indicating editing actions from natural language text inputs, wherein determining, from the natural language text input using the large language model, the one or more editing actions for modifying the digital image comprises generating, from the natural language text input using the large language model, a natural language text output that indicates the one or more editing actions in the format of the one or more in-context examples (Xu [0059] To illustrate and only as a non-limiting example, while the natural language-based image editor 110 is executing, the user 520 uploads the source image 505, and then speaks the following natural language request 510: “Please remove the girl from the photo and increase the light.” The source image 505 and the natural language request 510 is received as inputs to the natural language-based image editor 110, which passes the source image 505 and the natural language request 510 to the operation classifier model 115. The operation classifier model 115 evaluates the source image 505 and the natural language request 510 using trained neural networks to infer that the natural language request 510 describes two separate image editing operations. The operation classifier model 115 segments the natural language request 510 into two phrases: “Please remove the girl from the photo” and “increase the light.” Further, the operation classifier model 115 infers that the phrase “Please remove the girl from the photo” corresponds to the “inpaint_obj” image editing operation. The operation classifier model 115 also infers that the phrase “increase the light” corresponds to the “Brightness” image editing operation.).
Conclusion
For the prior art referenced and the prior art considered pertinent to Applicant’s disclosure but not relied upon, see PTO-892 “Notice of References Cited”.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON PRINGLE-PARKER whose telephone number is (571) 272-5690 and e-mail is jason.pringle-parker@uspto.gov. The examiner can normally be reached on 8:30am-5:00pm est Monday-Friday. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, King Poon can be reached on (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, seehttp://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON A PRINGLE-PARKER/
Primary Examiner, Art Unit 2617
Prosecution Insights