Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Regarding 35 USC § 102/103.
Applicant argues:
As discussed during the interview, however, neither reference teaches or suggests the above-mentioned limitation. For instance, while the cited references discuss interpreting natural language input into editing operations and then executing those editing operations to modify a digital image, they fail to teach or suggest generating and executing code for an editing application to modify an image. Rather, both references appear to use trained models to perform the edits. Indeed, the system described by Liu uses a large language model to understand and identify a requested change and then generate an action flow to implement the change, but Liu fails to describe using the large language model to generate executable action code. More specifically, Liu fails to teach or suggest generating executable action code that "includes one or more code segments written in a code language of the editing application to invoke one or more editing operations of the editing application." Xu likewise fails as Xu describes generating masks and operation classifications, not executable action code.
Examiner replies that:
Applicants arguments are not found persuasive. Xu explicitly recites that the image editing operation is performed by an image editor application (Xu [0017]). As discussed during the interview, Examiner agrees with the conceptual difference that Xu does not create code for an external application, however the claim as amended does not clearly distinguish the editing application as an entirely separate application.
Applicant argues:
Indeed, as described above, both Xu and Liu invoke models to modify a digital image. Xu, for example, uses a trained model to modify an image based on a generated mask and operation classification. Liu similarly uses an image generation model or an image editing model to perform edits based on an action flow. Neither reference, however, teaches or suggests modifying a digital image by executing executable action code via an editing application. Further, neither reference teaches or suggests executing code generated by a large language model.
Examiner replies that:
Applicants arguments are not found persuasive. Applicant argues “Neither reference, however, teaches or suggests modifying a digital image by executing executable action code via an editing application “. The editing application limitation is address above. With regard to executable action code, Xu is cited at [0059]-[0060] for teaching the code. Applicant has not provided any arguments or explanation as to why this is not executable action code.
Allowable Subject Matter
Claims 4, 6, 24, 26 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claims 4, 24 recites “receiving the natural language text input providing the instructions for modifying the digital image comprises receiving the natural language text input providing the instructions to remove a first object portrayed in the digital image, the first object holding a second object; modifying the digital image comprises removing the first object from the digital image while leaving the second object in the digital image based the editing region including the first object and excluding the second object; and receiving the user input for modifying the editing region within the digital image comprises receiving the user input for adding the second object into the editing region.” Where natural language text input specifically removes a first object holding a second object, leaving the second object, and input is received for adding the second object into the editing region. The prior art does not teach these limitations in combination with the other limitations.
Claims 6, 26 recites “further comprising providing, to the large language model, one or more in-context examples having a format for indicating objects targeted by natural language text inputs, wherein determining, using the large language model, the object that is targeted by the natural language text input for modification comprises generating, using the large language model, a natural language text output that indicates the object in the format of the one or more in-context examples.” Where in-context examples are sent to the LLM so the LLM outputs the object in the example format. The prior art does not teach these limitations in combination with the other limitations.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5, 7-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu U.S. Patent/PG Publication 20220067992 in view of Liu U.S. Patent/PG Publication 20240338862.
Regarding claim 1 (independent):
A computer-implemented method comprising: (Xu [0070] Any suitable computing system or group of computing systems can be used for performing the operations described herein. For example, FIG. 9 depicts an example of computing device 900 that may be at least a portion of image editing system 100. The implementation of the computing device 900 could be used for one or more of the image editor 105 or the user system 135. In an embodiment, a single image editing system 100 having devices similar to those depicted in FIG. 9 (e.g., a processor, a memory, etc.) combines the one or more operations and data stores depicted as separate subsystems in FIG. 1. Further, FIG. 8 illustrates a cloud computing system 800 by which at least a portion of the image editing system 100 may be offered.).
receiving, from a client device, a digital image and natural language text input providing instructions for modifying the digital image (Xu [0059] To illustrate and only as a non-limiting example, while the natural language-based image editor 110 is executing, the user 520 uploads the source image 505, and then speaks the following natural language request 510: “Please remove the girl from the photo and increase the light.”) via an editing application (Xu [0017] The artificial intelligence techniques described herein is be used by an image editor application to enable a user to perform image editing operations on a source image)
generating, using a and in response to receiving the natural language text input executable action code for modifying the digital image in accordance with the instructions of the natural language text input, (Xu [0059] which passes the source image 505 and the natural language request 510 to the operation classifier model 115. The operation classifier model 115 evaluates the source image 505 and the natural language request 510 using trained neural networks to infer that the natural language request 510 describes two separate image editing operations. The operation classifier model 115 segments the natural language request 510 into two phrases: “Please remove the girl from the photo” and “increase the light.” Further, the operation classifier model 115 infers that the phrase “Please remove the girl from the photo” corresponds to the “inpaint_obj” image editing operation. The operation classifier model 115 also infers that the phrase “increase the light” corresponds to the “Brightness” image editing operation. [0060] The operation classifier model 115 outputs “inpaint_obj” and “Brightness” as multiple inferred image editing operations.)
the executable action code being compatible with an editing application in that the executable action code includes one or more code segments written in a code language of the editing application to invoke one or more editing operations of the editing application; (Xu [0058] FIG. 5 depicts an example of a process 500, in which a user 520 executes a natural language-based image editor 110 )(Xu [0060] The grounding model 120 receives the inferred image editing operations of “inpaint_obj” and “Brightness” as inputs (along with other inputs, such as the natural language request 510 and the source image 505, not shown). The grounding model 120 classifies the “inpaint_obj” image editing operation as a local image editing operation, and then grounds the “inpaint_obj” image editing operation to the pixels of the source image 505 that depicts a girl. The grounding model 120 generates an image mask that hides the background pixels but reveals the pixels depicting the girl in the source image 505 (or vice versa). Likewise, the grounding model 120 classifies the “Brightness” image editing operation as a global image editing operation, and then grounds the “Brightness” image editing operation to the entire source image 505. In this case, the entire source image 505 is the image mask.)
modifying the digital image by executing the executable action code via the editing application and providing the modified digital image for display via a graphical user interface of the client device (Xu [0061] The grounding model 120 outputs two masks to the operation modular network 125. Mask #1 is a local image mask for the “inpaint_obj” image editing operation. Mask #2 is a global image mask for the “Brightness” image editing operation. Each mask is received at the operation modular network 125, which is trained to perform the two inferred image editing operations. The operation modular network 125 includes a submodule for the “inpaint_obj” image editing operation and another separate and distinct submodule for the “Brightness” image editing operation. The “Brightness” submodule infers a parameter (e.g., a brightness value indicating a magnitude of applying brightness to the source image 505) for the “Brightness” image editing operation because the natural language request 510 does not specify any parameters. Each of the “inpaint_obj” and the “Brightness” image editing operations are executed sequentially (or in parallel) to generate the modified source image 515.)
Xu does not teach a LLM. In a related field of endeavor, Liu teaches:
generating, using a large language model, executable action code for modifying the digital image in accordance with the instructions of the natural language text input, (Liu [0137] In step S461, in each round of dialogue, obtaining the interaction context 410 (i.e., the current dialogue data) between the current user and the image processing system, and understanding the user's requirement expressed in the interaction context 410 by using the large language model 420. Thereby, a multi-round requirement understanding is achieved.)(Liu [0139] In step S463, executing the action flow by calling the cross-modal image generating model 430 and/or the cross-modal image editing model 440 to obtain a target image.).
the executable action code being compatible with an editing application (Liu [0144] the image processing system generates the target image 552 and the explanation data thereof “I have changed the background of the image of film A into a starry sky using an image editing tool, and the path of the new image is image/ccc.png.” using the method 300 of the embodiments of the present disclosure.)
modifying the digital image by executing the executable action code via the editing application and providing the modified digital image for display via a graphical user interface of the client device (Liu Fig. 5)(Liu [0147] According to embodiments of the present disclosure, the image processing system can carry out multiple rounds of interaction with the user without limiting the number of rounds, and continuously guide the user to express complete and precise real requirement during the interaction, and gradually approach the ideal result that satisfies the user's requirement in multiple steps, by which both the image generating task and the image editing task can be completed, and the universality of the AI image creation application is significantly improved.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use a LLM as taught by Liu. The rationale for doing so would have been a simple substitution of one known element for another to obtain predictable results where Xu uses a neural network and Liu uses a LLM, where both are using artificial intelligence to interpret natural language to edit an image. Therefore it would have been obvious to combine Liu with Xu to obtain the invention.
Regarding claim 2:
The computer-implemented method of claim 1, has all of its limitations taught by Xu in view of Liu. Xu further teaches wherein generating the executable action code for modifying the digital image comprises generating the executable action code for modifying an editing region from the digital image using one or more editing operations of the editing application (Xu [0060] The grounding model 120 classifies the “inpaint_obj” image editing operation as a local image editing operation, and then grounds the “inpaint_obj” image editing operation to the pixels of the source image 505 that depicts a girl. The grounding model 120 generates an image mask that hides the background pixels but reveals the pixels depicting the girl in the source image 505 (or vice versa). Likewise, the grounding model 120 classifies the “Brightness” image editing operation as a global image editing operation, and then grounds the “Brightness” image editing operation to the entire source image 505. In this case, the entire source image 505 is the image mask.).
Regarding claim 3:
The computer-implemented method of claim 2, has all of its limitations taught by Xu in view of Liu. Xu further teaches further comprising:
receiving, via the client device, user input for modifying the editing region within the digital image (Xu [0059] “Please remove the girl from the photo”).
generating, using the large language model, additional executable action code that incorporates the modified editing region and the one or more editing operations and modifying the modified digital image by executing the additional executable action code via the editing application (Xu Further, the operation classifier model 115 infers that the phrase “Please remove the girl from the photo” corresponds to the “inpaint_obj” image editing operation. The operation classifier model 115 also infers that the phrase “increase the light” corresponds to the “Brightness” image editing operation. [0060] The operation classifier model 115 outputs “inpaint_obj” and “Brightness” as multiple inferred image editing operations. The grounding model 120 receives the inferred image editing operations of “inpaint_obj” and “Brightness” as inputs (along with other inputs, such as the natural language request 510 and the source image 505, not shown). The grounding model 120 classifies the “inpaint_obj” image editing operation as a local image editing operation, and then grounds the “inpaint_obj” image editing operation to the pixels of the source image 505 that depicts a girl. The grounding model 120 generates an image mask that hides the background pixels but reveals the pixels depicting the girl in the source image 505 (or vice versa). Likewise, the grounding model 120 classifies the “Brightness” image editing operation as a global image editing operation, and then grounds the “Brightness” image editing operation to the entire source image 505. In this case, the entire source image 505 is the image mask.).
Regarding claim 5:
The computer-implemented method of claim 1, has all of its limitations taught by Xu in view of Liu. Xu further teaches further comprising:
determining, using the (Xu [0059] Further, the operation classifier model 115 infers that the phrase “Please remove the girl from the photo” corresponds to the “inpaint_obj” image editing operation.).
and determining, using a segmentation model, an editing region of the digital image that corresponds to the object, wherein generating the executable action code for modifying the digital image in accordance with the instructions of the natural language text input comprises generating the executable action code for modifying the editing region of the digital image (Xu Further, the operation classifier model 115 infers that the phrase “Please remove the girl from the photo” corresponds to the “inpaint_obj” image editing operation. The operation classifier model 115 also infers that the phrase “increase the light” corresponds to the “Brightness” image editing operation. [0060] The operation classifier model 115 outputs “inpaint_obj” and “Brightness” as multiple inferred image editing operations. The grounding model 120 receives the inferred image editing operations of “inpaint_obj” and “Brightness” as inputs (along with other inputs, such as the natural language request 510 and the source image 505, not shown). The grounding model 120 classifies the “inpaint_obj” image editing operation as a local image editing operation, and then grounds the “inpaint_obj” image editing operation to the pixels of the source image 505 that depicts a girl. The grounding model 120 generates an image mask that hides the background pixels but reveals the pixels depicting the girl in the source image 505 (or vice versa). Likewise, the grounding model 120 classifies the “Brightness” image editing operation as a global image editing operation, and then grounds the “Brightness” image editing operation to the entire source image 505. In this case, the entire source image 505 is the image mask.).
Xu does not teach a LLM. In a related field of endeavor, Liu teaches:
large language model, (Liu [0137] In step S461, in each round of dialogue, obtaining the interaction context 410 (i.e., the current dialogue data) between the current user and the image processing system, and understanding the user's requirement expressed in the interaction context 410 by using the large language model 420. Thereby, a multi-round requirement understanding is achieved.)(Liu [0139] In step S463, executing the action flow by calling the cross-modal image generating model 430 and/or the cross-modal image editing model 440 to obtain a target image.).
Therefore, it would have been obvious before the effective filing date of the claimed invention to use a LLM as taught by Liu. The rationale for doing so would have been a simple substitution of one known element for another to obtain predictable results where Xu uses a neural network and Liu uses a LLM, where both are using artificial intelligence to interpret natural language to edit an image. Therefore it would have been obvious to combine Liu with Xu to obtain the invention.
Regarding claim 7:
The computer-implemented method of claim 5, has all of its limitations taught by Xu in view of Liu. Xu further teaches wherein determining, using the segmentation model, the editing region of the digital image that corresponds to the object comprises generating, using the segmentation model, a set of vertices that outline the object within the digital image (Xu [0067] FIG. 7 depicts an example of various stages of process 600 as described with respect to FIG. 6. In example 705, the input refers to the source image inputted by the user. The natural language request is “crop out everyone except the pastor and the guy in the black short sleeve shirt.” The source image and the natural language request pass through the operation classifier model, which infers that the user's intended image editing operation is “inpaint_obj,” which is an operation that removes an object from an image. The inferred image editing operation then is passed through the grounding model, which generates the “predicted” image mask shown in example 705 of FIG. 7. The “gt mask” refers to the ground truth image mask. The operation modular network receives the predicted image mask, the source image, and the natural language request to generate the output image (under “gt output”) by executing the submodule specifically configured for performing the “inpaint_obj” image editing operation. The output image is modified according to the user's natural language request because the individuals other than the “guy in the black short sleeve shirt” have been removed.).
Regarding claim 8:
The computer-implemented method of claim 1, has all of its limitations taught by Xu in view of Liu. Xu further teaches further comprising:
determining, from the natural language text input using the large language model, one or more editing actions for modifying the digital image (Xu [0060] The grounding model 120 classifies the “inpaint_obj” image editing operation as a local image editing operation, and then grounds the “inpaint_obj” image editing operation to the pixels of the source image 505 that depicts a girl. The grounding model 120 generates an image mask that hides the background pixels but reveals the pixels depicting the girl in the source image 505 (or vice versa). Likewise, the grounding model 120 classifies the “Brightness” image editing operation as a global image editing operation, and then grounds the “Brightness” image editing operation to the entire source image 505. In this case, the entire source image 505 is the image mask.).
and determining one or more executable code examples for the editing application that correspond to the one or more editing actions, wherein generating the executable action code for modifying the digital image in accordance with the instructions of the natural language text input comprises generating the executable action code for modifying the digital image using one or more editing operations of the editing application that correspond to the one or more editing actions (Xu [0061] The grounding model 120 outputs two masks to the operation modular network 125. Mask #1 is a local image mask for the “inpaint_obj” image editing operation. Mask #2 is a global image mask for the “Brightness” image editing operation. Each mask is received at the operation modular network 125, which is trained to perform the two inferred image editing operations. The operation modular network 125 includes a submodule for the “inpaint_obj” image editing operation and another separate and distinct submodule for the “Brightness” image editing operation. The “Brightness” submodule infers a parameter (e.g., a brightness value indicating a magnitude of applying brightness to the source image 505) for the “Brightness” image editing operation because the natural language request 510 does not specify any parameters. Each of the “inpaint_obj” and the “Brightness” image editing operations are executed sequentially (or in parallel) to generate the modified source image 515.)(Xu [0039] In some implementations, the prediction layer 250 includes one or more fully connected layers that are trained to generate the final output 255. As only a non-limiting example, the prediction layer 250 is trained using a multi-label cross-entropy loss technique. Any other text categorization or text classification techniques may be used to infer the image editing operations from the natural language request 205 and/or the source image 210. Additionally, while the operation classifier model 115 is described above as being trained using supervised learning techniques, the present disclosure is not limited thereto. In some implementations, the operation classifier model 115 is trained using unsupervised learning techniques for text analytics, such as, for example, clustering or latent semantic indexing (LSI).).
Regarding claim 9:
The computer-implemented method of claim 8, has all of its limitations taught by Xu in view of Liu. Xu further teaches further comprising providing, to the large language model, one or more in-context examples having a format for indicating editing actions from natural language text inputs, wherein determining, from the natural language text input using the large language model, the one or more editing actions for modifying the digital image comprises generating, from the natural language text input using the large language model, a natural language text output that indicates the one or more editing actions in the format of the one or more in-context examples (Xu [0059] To illustrate and only as a non-limiting example, while the natural language-based image editor 110 is executing, the user 520 uploads the source image 505, and then speaks the following natural language request 510: “Please remove the girl from the photo and increase the light.” The source image 505 and the natural language request 510 is received as inputs to the natural language-based image editor 110, which passes the source image 505 and the natural language request 510 to the operation classifier model 115. The operation classifier model 115 evaluates the source image 505 and the natural language request 510 using trained neural networks to infer that the natural language request 510 describes two separate image editing operations. The operation classifier model 115 segments the natural language request 510 into two phrases: “Please remove the girl from the photo” and “increase the light.” Further, the operation classifier model 115 infers that the phrase “Please remove the girl from the photo” corresponds to the “inpaint_obj” image editing operation. The operation classifier model 115 also infers that the phrase “increase the light” corresponds to the “Brightness” image editing operation.).
Regarding claim 21:
The claim is a parallel version of claim 1. As such it is rejected under the same teachings.
Regarding claim 22:
The claim is a parallel version of claim 2. As such it is rejected under the same teachings.
Regarding claim 23:
The claim is a parallel version of claim 3. As such it is rejected under the same teachings.
Regarding claim 25:
The claim is a parallel version of claim 5. As such it is rejected under the same teachings.
Regarding claim 27:
The claim is a parallel version of claim 7. As such it is rejected under the same teachings.
Regarding claim 28:
The claim is a parallel version of claim 8. As such it is rejected under the same teachings.
Regarding claim 29:
The claim is a parallel version of claim 9. As such it is rejected under the same teachings.
Regarding claim 30:
The claim is a parallel version of claim 1. As such it is rejected under the same teachings.
Regarding claim 31:
The claim is a parallel version of claim 2. As such it is rejected under the same teachings.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON PRINGLE-PARKER whose telephone number is (571) 272-5690 and e-mail is jason.pringle-parker@uspto.gov. The examiner can normally be reached on 8:30am-5:00pm est Monday-Friday. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, King Poon can be reached on (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, seehttp://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON A PRINGLE-PARKER/
Primary Examiner, Art Unit 2617