Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-6, 8-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Shi et al, (“LightningDrag: Lightning Fast and Accurate Drag-based Image Editing Emerging from Videos”, National University of Singapore, ByteDance Inc., Published on May 22, 2024, PP 1-13).
In regards to claim 1, Shi et al discloses a method of implementing drag-based
image editing, (see at least: Page 1, right-hand-column, “presenting LIGHTNINGDRAG, a rapid approach enabling high quality drag-based image editing …”, and section 4, “methodology”), comprising:
generating feature maps based on latent representations of an image by a first sub-model of a machine learning model, wherein the first sub-model is configured to preserve an identity of the image, (see at least: Fig. 3, where the appearance encoder corresponds to the first sub-model of a machine learning model. Further, on Page 5, left-hand-column, first paragraph, where the appearance encoder takes the reference latents
Z
0
s
r
c
as input, and extracts the reference feature maps from the self-attention layers, which are subsequently used to guide the self-attention process in the denoising backbone, [i.e., generating feature maps, “extracts the reference feature maps from the self-attention layers”, based on latent representations of an image, “reference latents
Z
0
s
r
c
“, by a first sub-model of a machine learning model, “appearance encoder”, wherein the first sub-model is configured to preserve an identity of the image, “the reference-only approach has demonstrated efficacy in preserving fine-grained details of the reference image”);
generating embeddings corresponding to at least one pair of points by a second
sub-model of the machine learning model, wherein each pair of points comprises a handle point and a target point, wherein the handle point identifies an area of the image, and wherein the target point indicates a target location to which the area is to be relocated, (see at least: page 4, sections 4.1-4.2, and Fig. 3, “Point Embedding Network to encode the (handle, target) points pairs”, and Page 5, section 4.2.3, eq. (4), Given the user-specified handle and target points, we first convert them into a handle and a target point map that is of the same resolution of the input image … and once obtaining the handle and target point maps, we encode them into embedding via a point embedding network, which is composed of 12 layers of convolution and SILU activation. This network outputs embedding at four different resolutions, corresponding to the four different resolutions of SD UNet activation maps, [i.e., generating embeddings corresponding to at least one pair of points by a second sub-model of the machine learning model, “point embedding network encodes the (handle, target) points pairs”, wherein each pair of points comprises a handle point and a target point, “(handle, target) points”, wherein the handle point identifies an area of the image, and wherein the target point indicates a target location to which the area is to be relocated, “see Fig. 1, last row (animal), “user input”, where the handle point (red) identifies an area of animal’s nose, and the target point (blue) identifies the location to which the nose area is to be moved by using lightingDrag method”]); and
injecting the feature maps and the embeddings into a third sub-model of the machine learning model to guide a process of generating a target image by the third sub-model, wherein the target image depicts the area of the image relocated at the target location, (see at least: Page 2, section 1, right-hand-column, lines 9-14, these embeddings are then injected into self-attention module of the backbone diffusion model, (i.e., a third sub-model), …to guide the generation process, and Page 5, the appearance encoder extracts the reference feature maps from the self-attention layers, which are subsequently used to guide the self-attention process in the denoising backbone, [i.e., injecting the feature maps and the embeddings into a third sub-model of the machine learning model, “subsequently using reference feature maps to guide the self-attention process in the denoising backbone, and injecting the embeddings into self-attention module of the backbone diffusion model …to guide the generation process”]. See also, Fig.2, Page 4, section 4.2.1, We utilize the Stable Diffusion Inpainting U-Net as our backbone, which takes concatenation of the following as input: noise latents
Z
t
, a binary mask M, and masked latents M ⊙
Z
s
r
c
,
(see Fig. 2, “concatenating masked input and mask and noise as input, implicitly for identifying the area of the input image, and output the target image showing the masked area moved to the target location”]).
In regards to claim 2, Shi et al further discloses inputting clean latent representations of the image into the first sub-model, (see at least: page 5, Fig. 3, input the reference image, “clean latent representations of the image”, into the appearance encoder model, “first sub-model”); and extracting the feature maps by the first sub-model once during the process of generating the target image, (Page 5, left-hand-column, first paragraph, where the appearance encoder takes the reference latents
Z
0
s
r
c
as input, and extracts the reference feature maps from the self-attention layers, extracting the feature maps by the first sub-model once during the process of generating the target image, [i.e., extracting the feature maps by the first sub-model, “extracts the reference feature maps by the appearance encoder”, once during the process of generating the target image, “during the process of generating the output image of Fig. 3”]).
In regards to claim 3, Shi et al further discloses converting the handle point into a handle map and converting the target point into a target map; and encoding the handle map and the target map into the embeddings by the second sub-model, (see at least: Page 5, section 4.2.3, we first convert the user-specified handle and target points into a handle and a target point map, “i.e., converting the handle point into a handle map and converting the target point into a target map”, … and once obtaining the handle and target point maps, we encode them into embedding via point embedding network, “i.e., encoding the handle map and the target map into the embeddings by the second sub-model”).
In regards to claim 4, Shi et al further discloses generating the target image by
the third sub-model based on noised latent representations of the image, masked latent representations of the image and a binary mask that indicate a region of the image to remain unedited during the process of generating the target image, (Fig.2, Page 4, section 4.2.1, using the Stable Diffusion Inpainting U-Net as our backbone, “i.e., third sub-model , which takes concatenation of the following as input: noise latents
Z
t
, a binary mask M ⊙
Z
s
r
c
, “i.e., based on noised latent representations of the image, masked latent representations of the image and a binary mask that indicate a region of the image”; and from page 4, section 4.2, an image inpainting backbone to enforce unmasked region remains identical, “i.e., to remain unedited during the process of generating the target image”).
In regards to claim 5, Shi et al further discloses extracting image features from the image using a fourth sub-model of the machine learning model, (see at least: Page 4, section 4.2.1, extracting the image feature of the source image using IP-Adapter, “fourth sub-model of the machine learning model”); and injecting the image features into the third sub-model to guide the process of generating the target image by the third sub-model, (see at least: Fig. 3, the images features extracted by the IP-Adapter, “i.e., fourth sub-model”, are input into the inpainting backbone, “third sub-model”, to implicitly guide the process of generating the output image shown in Fig. 3).
In regards to claim 6, Shi et al further discloses employing a time-dependent classifier-free guidance (CFG) mechanism to strengthen effects of the embeddings corresponding to the at least one pair of points during the process of generating the target image, (see at least: page 4, section 4.3.2, implementing the following point-following classifier-free guidance (PF-CFG) to strengthen the effects of given (handle, target) points pairs, (eq. (5)) ….we similarly find that a dynamic time-dependent CFG scale can help strike an appropriate balance between the accuracy of point-following and image quality of the results).
In regards to claim 8, Shi et al further discloses wherein training the machine learning model on the pairs of training data comprises training the first sub-model and the second sub-model on the pairs of training data while keeping the third-sub model frozen, (see at least: Fig. 3, training the appearance encoder and the point encoding while keeping the inpainting backbone model frozen).
In regards to claim 9, Shi et al further discloses presenting a recommendation to increase a quantity of pairs of handle and target points in response to determining that an editing result does not satisfy a threshold, (see at least: Fig. 6, section 4.4.1, When the region specified by handle points fail to move to the target locations, “i.e., in response to determining that an editing result does not satisfy a threshold”, augmenting the drag instruction with additional pairs of handle and target points has proven effective in improving results, [i.e., presenting a recommendation, “augmenting the drag instruction with additional pairs of handle and target points”, to increase a quantity of pairs of handle and target points, “has proven effective in improving results”]).
In regards to claim 10, Shi et al further discloses presenting a recommendation to employ sequential dragging for editing the image in response to determining that an editing result does not satisfy a threshold, (see at least: Page 7, section 4.4.2, and Fig. 7, In cases when single dragging operation cannot attain the desired outcome, a simple workaround is to break the operation down into a sequence of shorter dragging trajectories).
In regards to claim 11, Shi et al further discloses wherein the identity of the image comprises information of identifying objects in the image, (see at least: Page 3, section 4.2, image identity (e.g., human face, texture, etc.) should be preserved after dragging, [i.e., the human face, texture represent the information of identifying objects in the image]).
Regarding claim 12, claim 12 recites substantially similar limitations as set forth in claim 1. As such, claim 12 is rejected for at least similar rational.
The Examiner further acknowledged the following additional limitation(s): “a system of implementing drag-based image editing”. However, Shi et al discloses the “system of implementing drag-based image editing”, (see at least: Page 1, right-hand-column, “presenting LIGHTNINGDRAG, a rapid approach enabling high quality drag-based image editing …”. Further, fig. 3, shows the pipeline system of LightingDrag”).
Regarding claim 13, claim 13 recites substantially similar limitations as set forth in claim 2. As such, claim 13 is rejected for at least similar rational.
Regarding claim 14, claim 14 recites substantially similar limitations as set forth in claim 3. As such, claim 14 is rejected for at least similar rational.
Regarding claim 15, claim 15 recites substantially similar limitations as set forth in claim 4. As such, claim 15 is rejected for at least similar rational.
Regarding claim 16, claim 16 recites substantially similar limitations as set forth in claim 5. As such, claim 16 is rejected for at least similar rational.
Regarding claim 17, claim 17 recites substantially similar limitations as set forth in claim 1. As such, claim 17 is rejected for at least similar rational.
The Examiner further acknowledged the following additional limitation(s): “a non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations”. However, Shi et al discloses the “non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations”, (see at least: Page 4, right-hand-column, first paragraph, “tracking algorithm”).
Regarding claim 18, claim 18 recites substantially similar limitations as set forth in claim 2. As such, claim 18 is rejected for at least similar rational.
Regarding claim 19, claim 19 recites substantially similar limitations as set forth in claim 3. As such, claim 19 is rejected for at least similar rational.
Regarding claim 20, claim 20 recites substantially similar limitations as set forth in claim 4. As such, claim 20 is rejected for at least similar rational.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Shi et al, (“LightningDrag: Lightning Fast and Accurate Drag-based Image Editing Emerging from Videos”, National University of Singapore, ByteDance Inc., Published on May 22, 2024, PP 1-13).
Shi et al further discloses generating pairs of training data using videos, wherein each pair of training data comprises a reference frame and a target frame, wherein each reference frame comprises at least one reference handle point for identifying an area of the reference frame, (see at least: as shown in Fig. 2, “i.e., the paired video frames comprise reference frame and target frame, and from Page 2, left-hand-column, second paragraph, the training data is constructed from paired video frames, comprising source image and target image, “i.e., generating pairs of training data using videos, wherein each pair of training data comprises a reference frame and a target frame”, … and employing CoTracker2 [22] to identify the handle points’ corresponding target points in the second frame”, [i.e., each reference frame comprises at least one reference handle point for identifying an area of the reference frame]), and
training the machine learning model on the pairs of training data, (see at least: Page 2, left-hand-column, first paragraph, train our model on largescale paired video frames, “i.e., training the machine learning model on the pairs of training data”)
Shi et al does not expressly disclose wherein each target frame comprises at least one ground-truth target point indicating a location where the area is relocated.
However, Shi discloses providing a quantitative assessment of the method on DragBench [55], comprising 205 samples with pre-define drag points and masks, (see at least: page 7, section 5.2), which the pre-define drag points, technically represent the ground-truth target point indicating a location where the masks, (i.e., areas), are relocated.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMARA ABDI whose telephone number is (571)272-0273. The examiner can normally be reached 9:00am-5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at (571) 272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMARA ABDI/Primary Examiner, Art Unit 2668 06/25/2026
Prosecution Insights