DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Compact Prosecution
With respect to Claim Interpretation, the Examiner has provided some notes regarding “[BRI on the record]” throughout the Office Action, so that the record is clear about the scope of the claimed invention, and the record is also clear about the basis for the Examiner’s analyses. A clear record of the claim interpretation could expedite the examination by creating the condition to allow the examination to focus on Applicant’s inventive concept and its comparison with related prior art.
If there are disagreements, Applicant may present an alternative interpretation based on MPEP 2111. The Examiner will adopt Applicant’s interpretation on the record, if Applicant’s interpretation is reasonable and/or arguments are persuasive.
Applicant may amend claims relying on the Examiner’s claim interpretation provided on the record.
With respect to Claims 5-6, 8-12, and 17-18, the Examiner relies on Mirzaei et al. (“Watch Your Steps: Local Image and Scene Editing by Text Instructions”), which was published on 8/17/2023, which is earlier than the effective filing date (9/5/2023) of the instant application. Mirzaei et al. and the instant application share six authors/inventors. However, Mirzaei et al. has one additional author Jonathan Kelly, who is not listed as an inventor of the instant application.
MPEP 2153.01(a) states, “If, however, the application names fewer joint inventors than a publication (e.g., the application names as joint inventors A and B, and the publication names as authors A, B and C), it would not be readily apparent from the publication that it is an inventor-originated disclosure and the publication would be treated as prior art under AIA 35 U.S.C. 102(a)(1) unless there is evidence of record that an exception under AIA 35 U.S.C. 102(b)(1) applies.”
MPEP 2153.01(a) provides guidance on the procedure to invalidate Mirzaei et al. as a prior art reference.
Claims 5-6, 8-12, and 17-18 will be allowed depending on updated search and consideration, if Mirzaei et al is invalidated by Applicant as prior art following the procedure disclosed in MPEP 2153.01(a).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 2, 9, and 14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 2 recites “corresponds to an improved InstructPix2Pix (IP2P).”
Claim 9 recites “comprises an improved InstructNeRF2NeRF (IN2N) and an improved InstructPix2Pix (IP2P).”
Claim 14 recites “corresponds to an improved InstructPix2Pix (IP2P).”
These claims recite “improved.” MPEP 2173.05(b) recites “Terms of degree are not necessarily indefinite. ‘Claim language employing terms of degree has long been found definite where it provided enough certainty to one of skill in the art when read in the context of the invention.’” However, there is not enough certainty to one of skill in the art when read in the context of the invention to understand what constitutes the “improvement,” especially when the scopes of “InstructPix2Pix” and “InstructNeRF2NeRF” are also unclear.
The Examiner finds it difficult to determine the scope of “InstructPix2Pix” and “InstructNeRF2NeRF.” For example, with respect to “InstructPix2Pix,” which of the following interpretation should the Examiner adopt?
Instruction (Instruct) and Picture (Pix) as input and Picture (Pix) as output.
The specifically trained conditional diffusion model as taught by Brooks et al. (“InstructPix2Pix: Learning to Follow Image Editing Instructions”) as specified Fig. 2.
Is the scope of InstructPix2Pix frozen at Brooks et al.? Is InstructPix2Pix an evolving technique?
Due to these competing interpretations, the scope of “InstructPix2Pix” is indefinite. For similar reasons, the scope of “InstructNeRF2NeRF” is also indefinite.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-4, 7, 13, 15-16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Couairon et al. ("Diffedit: Diffusion-based semantic image editing with mask guidance.") in view of Jung et al. (US 20220097522 A1).
Regarding Claim 1, Couairon teaches A method for editing a local area of a target image using a diffusion model (
Couairon Abstract: “In contrast, our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited, by contrasting predictions of a diffusion model conditioned on different text prompts.”
The local area is mapped to “regions of the input image that need to be edited.”
The target image is mapped to the “input image.”
), the method comprising:
receivingWithin Couairon fig. 2 X0) (
“The task of semantic image editing consists in modifying an input image in accordance with a
textual transformation query. For instance, given an image of a bowl of fruits and the query ‘fruits’ [Wingdings font/0xE0] ‘pears’, the aim is to produce a novel image where the fruits have been changed into pears,
while keeping the bowl and the background as similar as possible to the input image. The text query
can also be a more elaborate description like ‘A basket of fruits’.” Couairon 1 Introduction.
The input image is mapped to Couairon fig. 2 X0; or the “an image of a bowl of fruits” in the introduction.);
receivingWithin Couairon fig. 2: “Query Q: Zebra”) to edit the input image (X0) (Couairon fig. 2 shows the input image X0 is edited according to the text of Query Q to produce output image Couairon fig. 2 Y0);
generating a relevance map (Within Couairon fig. 2 “Mask M”) based on the diffusion model and the text instruction (Within Couairon fig. 2: “Query Q: Zebra”) (
[BRI on the record] With respect to “relevance map,” the Examiner is reading the limitation to mean: a mask map/image to indicate region(s) of an image to edit. This interpretation is made in light of the specification.
[0063] Throughout the disclosure, the discrepancy between the first noise prediction and the second noise prediction may be referred to as or may be represented by a ‘relevance map.’ In some embodiments, ‘binarizing’ the ‘relevance map’ gives the mask of the region that may be edited. Throughout the disclosure, ‘binarizing’ means converting a real-valued number (e.g., 0.2, 0.7, 1.8) of the relevance map to a binary relevance mask (zero (0) or one (1)). In some embodiments, if the real-valued number is above a threshold (e.g., one (1)), then the real-valued number is converted to one (1). If the real-valued number is equal to or below the threshold, then the real-valued number is converted to zero (0).
Spec. ¶ 63.
[Mapping Analysis]
Couairon fig. 2 shows that the query Q, mapped to text instruction, is used to generate Mask M.
“When the denoising an image, a text-conditioned diffusion model will yield different noise estimates given different text conditionings. We can consider where the estimates are different, which gives information about what image regions are concerned by the change in conditioning text. For instance, in Figure 2, the noise estimates conditioned to the query zebra and reference text horse1 are different on the body of the animal, where they will tend to decode different colors and textures depending on the conditioning. For the background, on the other hand, there is little change in the noise estimates. The difference between the noise estimates can thus be used to infer a mask that identifies what parts on the image need to be changed to match the query.” Couairon Step 1: Computing editing mask.
The diffusion model is mapped to the text-conditioned diffusion model.
The relevance map is mapped to the Mask M, and the mapping is consistent with specification ¶ 63.
The Mask M is based on the text instruction, comprising “Estimate noise conditionally to query Q,” as shown in Couairon fig. 2 and according to explanation provided in Couairon Step 1: Computing editing mask.);
generating a rendered image (Within Couairon fig. 2: Y0) by performing a relevance guided image editing method (method according to Couairon fig. 2) on the input image (Within Couairon fig. 2: X0), based on the generated relevance map (Within Couairon fig. 2: “Mask M”) (
Couairon fig. 2 Step 3:
“The task of semantic image editing consists in modifying an input image in accordance with a
textual transformation query. For instance, given an image of a bowl of fruits and the query ‘fruits’ [Wingdings font/0xE0] ‘pears’, the aim is to produce a novel image where the fruits have been changed into pears,
while keeping the bowl and the background as similar as possible to the input image. The text query
can also be a more elaborate description like ‘A basket of fruits’.” Couairon 1 Introduction. “The difference between the noise estimates can thus be used to infer a mask that identifies what parts on the image need to be changed to match the query.” Couairon Step 1: Computing editing mask.
); and
providingWithin Couairon fig. 2: Y0), wherein the relevance guided image editing method comprises:
generating a second noisy image (e.g., Xt within Couairon fig. 2 Step 2) of the input image by adding second noise to the input image by using the diffusion model (
“Denoising diffusion probabilistic models (Ho et al., 2020) is a class of generative models that are trained to invert a diffusion process. For a number of timesteps T, the diffusion process gradually adds noise to the input data, until the resulting distribution is (almost) Gaussian.” Couairon 3.1 Background: Diffusion Models, DDIM and Encoding. Also see equation (1).
The second noise is mapped to noise added at a step.);
generating a third noisy image (Within Couairon fig. 2 Step 3: Y(t+1) or Yt) of the input, which comes from an output of a previous step (Within Couairon fig. 2 Step 3: DDIM Step (Zebra)) of a denoising step of the diffusion model (
Couairon fig. 2 Step 3: shows the process of generating Yt and Y(t+1) is similarly generated:
Yt and Y(t+1) are computed according to Couairon 3.1 Background: Diffusion Models, DDIM and Encoding: Equation (2):
“The mask-guided DDIM update can be written as ŷ= MYt + (1-M) Xt, where Yt is computed from Yt-dt with Eq. 2, and Xt is the corresponding DDIM encoded latent.” Couairon 3.2 Step 3.);
receiving, from a code of the diffusion model, an output image of the third noisy image (Couairon fig. 2), which is obtained via the denoising step (Couairon fig. 2 Step 3: DDIM Step (Zebra)) of the diffusion model (
[BRI on the record]
With respect to “a code of the diffusion model,” the Examiner is reading the limitation to mean program based on the diffusion model. This interpretation is in light of the specification:
[0031] It will be apparent that systems and/or methods, described herein, may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and/or methods are not limited to the implementations.
Spec. ¶ 31.
[Mapping Analyses]
Yt are computed according to equation (2) based on Y(t+1):
“The mask-guided DDIM update can be written as ŷ= MYt + (1-M) Xt, where Yt is computed from Yt-dt with Eq. 2, and Xt is the corresponding DDIM encoded latent.” Couairon 3.2 Step 3; Couairon 3.1 Background: Diffusion Models, DDIM and Encoding: Equation (2)(3):
The abovementioned process is run on a program for a GPU-based computing device. Couairon 4.1 Experiment Setup.); and
generating the rendered image (Within Couairon fig. 2: Y0 ) by the code (program based on the diffusion model) based on the relevance map (“MASK M”), the second noisy image (Xt ) of the input image, the output image (Yt) received from the diffusion model (
Couairon Fig. 2 Step 3: shows how the relevance map, the second noisy image, the output image are used to generate the rendered image:
“The mask-guided DDIM update can be written as ŷ= MYt + (1-M) Xt, where Yt is computed from Yt-dt with Eq. 2, and Xt is the corresponding DDIM encoded latent.” Couairon 3.2 Step 3.).
Couairon is an academic paper. Although Couairon suggests the following features, Couairon does not explicitly disclose:
receiving, from a user of an electronic device, input e.g., input image;
receiving, from the user, input e.g., text instruction; or
providing, to the user, an output.
Jung teaches:
receiving, from a user of an electronic device, input (“Then, the controller 870 displays an image selected by a user among the images included in the list.” Jung ¶ 486; see Fig. 1.);
receiving, from the user, input (“Here, the processor 1800 may obtain intent information corresponding to the user input by using at least one of a Speech to Text (STT) engine for converting a voice or audio input into a text string and a natural language processing (NLP) engine for obtaining intent information of a natural language.” Jung ¶ 99.); and
providing, to the user, an output (“Then, the controller 870 displays an image selected by a user among the images included in the list.” Jung ¶ 486. “In this case, the output unit 1500 may include a display module or unit for outputting visual information, a speaker for outputting auditory information, a haptic module for outputting tactile information, and the like.” Jung ¶ 93.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Jung’s user interface with Couairon. One of ordinary skill in the art would be motivated to allow effective and/or convenient communication between a user and computer.
Regarding Claim 3, Couairon in view of Jung teaches The method of claim 1, wherein the receiving the input image comprises at least one of:
capturing the input image by the user (“the input unit 1200 may include a camera for inputting an image (or video) signal” Jung ¶ 85.), loading the image by the user from a storage of the electronic device (“. . . the controller 870 may generate a plurality of synthesized images from one of the plurality of images and display a list of the plurality of synthesized images. Then, the controller 870 displays an image selected by a user among the images included in the list.” Jung ¶ 486. “the memory 1700 may store input data acquired from the input unit 1200” Jung ¶ 94. “The computer-readable medium may include all types of recording devices each storing data readable by a computer system. Examples of such computer-readable media may include hard disk drive (HDD), solid state disk (SSD), silicon disk drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage element and the like.” Jung ¶ 499.), or receiving the input image from an outside of the electronic device.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Jung’s data acquiring method with Couairon. One of ordinary skill in the art would be motivated to provide a range of options to generate input data, e.g., image data. It makes the process more convenient and/or flexible for a user depending on the situation.
Regarding Claim 4, Couairon in view of Jung teaches The method of claim 1, wherein the text instruction corresponds to an instruction generated by a speech-to-text operation (“Here, the processor 1800 may obtain intent information corresponding to the user input by using at least one of a Speech to Text (STT) engine for converting a voice or audio input into a text string and a natural language processing (NLP) engine for obtaining intent information of a natural language.” Jung ¶ 99.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Jung’s user interface with Couairon. One of ordinary skill in the art would be motivated to allow effective and/or convenient communication between a user and computer.
Regarding Claim 7, Couairon in view of Jung teaches The method of claim 1, the generating the rendered image based on the relevance map, the second noisy image of the input image, the output image received from the diffusion model (refers the identical limitation in Claim 1, and the same analyses apply here), comprises:
generating a first set of pixels by multiplying pixels of the second noisy image of the input image with unmasked pixels, wherein the unmasked pixels correspond to pixels of (1−the relevance map) (
“The mask-guided DDIM update can be written as ŷ= MYt + (1-M) Xt, where Yt is computed from Yt-dt with Eq. 2, and Xt is the corresponding DDIM encoded latent.” Couairon 3.2 Step 3.
Here, (1-M) Xt = represents multiplying pixels of the second noisy image of the input image with unmasked pixels
Xt [Wingdings font/0xE0] second noisy image
M is Mask M as shown in fig. 2 [Wingdings font/0xE0] relevance map
(1-M) [Wingdings font/0xE0] (1−the relevance map) [Wingdings font/0xE0] unmasked pixels);
generating a second set of pixels by multiplying pixels of the third noisy image of the input image with masked pixels, wherein the masked pixels correspond to pixels of the relevance map (
“The mask-guided DDIM update can be written as ŷ= MYt + (1-M) Xt, where Yt is computed from Yt-dt with Eq. 2, and Xt is the corresponding DDIM encoded latent.” Couairon 3.2 Step 3.
Here, MYt represents multiplying pixels of the third noisy image of the input image with masked pixels
Yt [Wingdings font/0xE0] third noisy image
M is Mask M as shown in fig. 2 [Wingdings font/0xE0] relevance map
); and
generating the rendered image by adding the first set of pixels to the second set of pixels (
“The mask-guided DDIM update can be written as ŷ= MYt + (1-M) Xt, where Yt is computed from Yt-dt with Eq. 2, and Xt is the corresponding DDIM encoded latent.” Couairon 3.2 Step 3.
Ŷ [Wingdings font/0xE0] the generated rendered image
Couairon Fig. 2 shows the result.).
Claims 13, 15-16, 19 are substantially similar to Claims 1, 3-4, 7. The rejections analyses based on Couairon in view of Jung for Claims 1, 3-4, 7 are applied to Claims 13, 15-16, 19. In addition, Claim 13 recites “An electronic device for editing a local area of a target image using a diffusion model, the electronic device comprising: at least one processor comprising processing circuitry; and memory coupled to the at least one processor, the memory configured to store one or more instructions which, when executed by the at least one processor individually or collectively, cause the electronic device to: . . .” ( Couairon 4.1; Jung Fig. 1. “An AI device 1000 may be configured as a fixed (or stationary) device or a movable (or mobile) device such as a TV, a projector, a mobile phone, a smartphone, a desktop computer, a notebook computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a tablet PC, a wearable device, and a set-top box (STB), a DMB receiver, a radio, a washing machine, a refrigerator, a desktop computer, digital signage, a robot, a vehicle, and the like.” Jung ¶ 80. “The processor 1800 may control at least some of the components of the AI device 1000 to run an application program stored in the memory 1700. Further, the processor 1800 may operate two or more components included in the AI device 1000 in combination to execute the application program.” Jung ¶ 102.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Jung’s computer with Couairon. One of ordinary skill in the art would be motivated to automate the computation process with the use of computer, which could increase processing speed and accuracy.
Claims 2 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Couairon in view of Jung as applied to Claim 1 or 13, in further view of Brooks et al. (“InstructPix2Pix: Learning to Follow Image Editing Instructions”).
Regarding Claim 2, Couairon in view of Jung teaches The method of claim 1.
Couairon in view of Jung does not explicitly disclose wherein the diffusion model corresponds to an improved InstructPix2Pix (IP2P).
Brooks teaches wherein the diffusion model corresponds to an improved InstructPix2Pix (IP2P) (“Our conditional diffusion model, InstructPix2Pix, is trained on our generated data, and generalizes to real images and user-written instructions at inference time.” Brooks Abstract.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Brooks’ InstructPix2Pix with Couairon in view of Jung. One of ordinary skill in the art would be motivated to process the data faster and/or produce better results. Brooks recites, “. . . our model edits images quickly, in a matter of seconds. We show compelling editing results for a diverse collection of input images and written instructions.” Brooks Abstract.
Claim 14 is substantially similar to Claim 2. The rejections analyses based on Couairon in view of Jung and Brooks for Claim 2 is applied to Claim 14.
Claims 5-6 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Couairon in view of Jung as applied to Claim 1, in further view of Mirzaei et al. (“Watch Your Steps: Local Image and Scene Editing by Text Instructions”).
Regarding Claim 5, Couairon in view of Jung teaches The method of claim 1.
Couairon in view of Jung does not explicitly disclose
wherein the diffusion model comprises a first Unet and a second Unet, and
wherein the generating the relevance map based on the diffusion model and the text instruction, comprises:
generating a first noisy image of the input image by adding first noise to the input image by using the diffusion model;
inputting the input image and the first noisy image to the first Unet without any text instruction and inputting the input image and the first noisy image to the second Unet with a text instruction;
obtaining a difference between a first output of the first Unet and a second output of the second Unet; and
normalizing the obtained difference.
Mirzaei teaches
wherein the diffusion model comprises a first Unet and a second Unet (Mirzaei Fig. 1 shows a model with two “IP2P Unets.” “We then use IP2P’s noise prediction Unet, ϵθ, to get two different predictions: . . ..” Mirzaei 4.1 Relevance map calculation. “Denoising diffusion models have enabled high-quality image generation and editing. We present a method to localize the desired edit region implicit in a text instruction. We leverage InstructPix2Pix (IP2P) and identify the discrepancy between IP2P predictions with and without the instruction.” Mirzaei Abstract.), and
wherein the generating the relevance map based on the diffusion model and the text instruction (Mirzaei Fig. 1 shows that the relevance map is generated based on the diffusional model (IP2P) and text instruction: “Make the owl a falcon”), comprises:
generating a first noisy image of the input image by adding first noise to the input image by using the diffusion model (
“We then use IP2P’s noise prediction Unet, ϵθ, to get two different predictions: . . ..” Mirzaei 4.1 Relevance map calculation. “Denoising diffusion models have enabled high-quality image generation and editing. We present a method to localize the desired edit region implicit in a text instruction. We leverage InstructPix2Pix (IP2P) and identify the discrepancy between IP2P predictions with and without the instruction. This discrepancy is referred to as the relevance map. The relevance map conveys the importance of changing each pixel to achieve the edits, and is used to to guide the modifications. This guidance ensures that the irrelevant pixels remain unchanged.” Mirzaei Abstract.);
inputting the input image (Within Mirzaei Fig. 1: Original Image) and the first noisy image (Noisy Image) to the first Unet (left IP2P Unet) without any text instruction (“” in the figure indicting without text instruction) and inputting the input image (Original Image) and the first noisy image (Noisy Image) to the second Unet (right IP2P Unet) with a text instruction (“Make the owl a falcon”);
obtaining a difference (Within Mirzaei Fig. 1: “Normalized difference”) between a first output of the first Unet and a second output of the second Unet (Within Mirzaei Fig. 1); and
normalizing the obtained difference (Within Mirzaei Fig. 1: “Normalized difference”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Mirzaei’ relevance map generation with Couairon in view of Jung. One of ordinary skill in the art would be motivated to relax the requirement on text instructions, as compared to Couairon. “Our method achieves state-of-the-art performance on both image and NeRF editing tasks.” Mirzaei Abstract.
Regarding Claim 6, Couairon in view of Jung and Mirzaei teaches The method of claim 5, wherein the first Unet and the second Unet are the same InstructPix2Pix (IP2P) Unet (Mirzaei Fig. 1 shows two IP2P Unets. “We then use IP2P’s noise prediction Unet, ϵθ, to get two different predictions: . . ..” Mirzaei 4.1 Relevance map calculation. “Denoising diffusion models have enabled high-quality image generation and editing. We present a method to localize the desired edit region implicit in a text instruction. We leverage InstructPix2Pix (IP2P) and identify the discrepancy between IP2P predictions with and without the instruction.” Mirzaei Abstract.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Mirzaei’ relevance map generation with Couairon in view of Jung. One of ordinary skill in the art would be motivated to relax the requirement on text instructions, as compared to Couairon. “Our method achieves state-of-the-art performance on both image and NeRF editing tasks.” Mirzaei Abstract.
Claims 17-18 is substantially similar to Claims 5-6. The rejections analyses based on Couairon in view of Jung and Mirzaei for Claims 5-6 are applied to Claim 17-18.
Claims 8-12 are rejected under 35 U.S.C. 103 as being unpatentable over Mirzaei et al. (“Watch Your Steps: Local Image and Scene Editing by Text Instructions”) in view of Jung et al. (US 20220097522 A1).
Mirzaei et al. (“Watch Your Steps: Local Image and Scene Editing by Text Instructions”), which was published on 8/17/2023, which is earlier than the effective filing date (9/5/2023) of the instant application. Mirzaei et al. and the instant application share six authors/inventors. However, Mirzaei et al. has one additional author Jonathan Kelly, who is not listed as an inventor of the instant application.
Mirzaei et al. (“Watch Your Steps: Local Image and Scene Editing by Text Instructions”) and the instant application discloses the same invention.
The disclosure in the provisional application 63/536,671 appears to be a subset of Mirzaei et al.. For example, all the figures in 63/536,671 are included in Mirzaei et al. and Mirzaei et al. provides more.
Jung et al. (US 20220097522 A1) has been introduced to address claimed features related to user interactions, which Mirzaei et al. implicitly teaches.
Regarding Claim 8, Mirzaei teaches A method for editing a local area of a target scene using a diffusion model comprising a Neural Radiance Field (NeRF) (
“Watch Your Steps: Local Image and Scene Editing by Text Instructions” Mirzaei Title. Mirzaei Figs. 1-3. “Our method achieves state-of-the-art performance on both image and NeRF editing tasks”
Mirzaei section 3:
PNG
media_image1.png
444
748
media_image1.png
Greyscale
), the method comprising:
receiving
Mirzaei section 4.3:
PNG
media_image2.png
188
726
media_image2.png
Greyscale
The plurality of images are mapped to the multiview capture.
NeRF is mapped to NeRF.);
receiving, from the user of the electronic device, a text instruction to edit the input scene (
Mirzaei section 4.3: the quote as provided above.
The text instruction is mapped to the text prompt.);
generating a plurality of relevance maps respectively corresponding to the plurality of images (
Mirzaei section 4.3:
PNG
media_image3.png
212
730
media_image3.png
Greyscale
The relevance maps is mapped to relevance maps.
The plurality of images is mapped to the training views/ the multiview capture.), and
generating a relevance field by fitting the NeRF to the plurality of relevance maps (
Mirzaei section 1. Introduction:
PNG
media_image4.png
214
734
media_image4.png
Greyscale
Here, the relevance field is trained/generated based on relevance maps.
What has been trained is NePF, and training is a form fitting parameters for NePF),
);
generate an edited scene by performing a relevance guided scene editing method, based on the input scene, the text instruction, and the generated relevance field (Mirzaei Fig. 3); and
providingMirzaei Fig. 3, showing outputting “Edited Image.”),
wherein the relevance guided scene editing method (Mirzaei Fig. 3) comprises:
generating an edited image (Mirzaei Fig. 3 Edited Image) and an updated relevance map (Mirzaei Fig. 3 Relevance Map) corresponding to the edited image (Mirzaei Fig. 3) by performing a relevance guided image editing method (Mirzaei Fig. 3 “Relevance-guided Image Editor”) on an original image of the plurality of images (Mirzaei Fig. 3 shows original images of a bear from different viewpoints) and a rendered image obtained from the fitted NeRF, based on the text instruction (Mirzaei Fig. 3 Text Input “Turn the bear into a panda”) and a relevance map obtained from the relevance field (
Mirzaei Fig. 3:
Mirzaei section 1. Introduction:
PNG
media_image4.png
214
734
media_image4.png
Greyscale
Here, the relevance field is trained/generated based on relevance maps. What has been trained is NePF, and training is a form fitting parameters for NePF), and
updating the NeRF and the relevance field with the generated edited image and the updated relevance map (Mirzaei Fig. 3 that includes: “The relevance guided image editing module (§ 4.2) returns an edited image and an updated relevance, which are used to update the corresponding training views for the NeRF and the relevance field, respectively.”), and
wherein the relevance guided image editing method (Mirzaei Fig. 1) comprises:
generating a second noisy image (Mirzaei Fig. 2:
PNG
media_image5.png
144
140
media_image5.png
Greyscale
) of the original image (Mirzaei Fig. 2: “Input image”) by adding second noise to the original image by using the diffusion model (Mirzaei Fig. 2);
generating a third noisy image (Zt that is similar to and used to generate
PNG
media_image6.png
136
140
media_image6.png
Greyscale
) of the original image, which comes from an output of a previous step of a denoising step (
PNG
media_image7.png
150
94
media_image7.png
Greyscale
) of the diffusion model ( “Denoising diffusion models have enabled high-quality image generation and editing.” Mirzaei Abstract. Mirzaei Fig. 2 Denoise Step);
receiving, from a code of the diffusion model (program based on a diffusion model), an output image (
PNG
media_image6.png
136
140
media_image6.png
Greyscale
) of the third noisy image (Zt), which is obtained via the denoising step (
PNG
media_image7.png
150
94
media_image7.png
Greyscale
) of the diffusion model for the third noisy image (“Denoising diffusion models have enabled high-quality image generation and editing.” Mirzaei Abstract. Mirzaei Fig. 2 Denoise Step); and
generating the edited image (Mirzaei Fig. 2: “Edited Image”) by the code (program based on a diffusion model) based on the relevance map (fig. 2 relevance map), the second noisy image (Mirzaei Fig. 2:
PNG
media_image5.png
144
140
media_image5.png
Greyscale
of the first row) of the original image, the output image (
PNG
media_image6.png
136
140
media_image6.png
Greyscale
of the second row) from the code (Mirzaei Fig. 2).
Mirzaei is an academic paper. Although Mirzaei suggests the following features, Mirzaei does not explicitly disclose:
receiving, from a user of an electronic device, input e.g., input scene;
receiving, from the user, input e.g., text instruction; or
providing, to the user of the electronic device, an output.
Jung teaches:
receiving, from a user of an electronic device, input (“Then, the controller 870 displays an image selected by a user among the images included in the list.” Jung ¶ 486; Fig. 1.);
receiving, from the user, input (“Here, the processor 1800 may obtain intent information corresponding to the user input by using at least one of a Speech to Text (STT) engine for converting a voice or audio input into a text string and a natural language processing (NLP) engine for obtaining intent information of a natural language.” Jung ¶ 99.); and
providing, to the user of the electronic device, an output (“Then, the controller 870 displays an image selected by a user among the images included in the list.” Jung ¶ 486. “In this case, the output unit 1500 may include a display module or unit for outputting visual information, a speaker for outputting auditory information, a haptic module for outputting tactile information, and the like.” Jung ¶ 93.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Jung’s user interface with Mirzaei. One of ordinary skill in the art would be motivated to allow effective and/or convenient communication between a user and computer.
Regarding Claim 9, Mirzaei in view of Jung teaches The method of claim 8, wherein the diffusion model comprises an improved InstructNeRF2NeRF (IN2N) and an improved InstructPix2Pix (IP2P) (
Mirzaei:
PNG
media_image8.png
282
736
media_image8.png
Greyscale
).
Regarding Claim 10, Mirzaei in view of Jung teaches The method of claim 8, wherein the text instruction corresponds to an instruction generated by a speech-to-text operation (“Here, the processor 1800 may obtain intent information corresponding to the user input by using at least one of a Speech to Text (STT) engine for converting a voice or audio input into a text string and a natural language processing (NLP) engine for obtaining intent information of a natural language.” Jung ¶ 99.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Jung’s user interface with Mirzaei. One of ordinary skill in the art would be motivated to allow effective and/or convenient communication between a user and computer.
Regarding Claim 11, Mirzaei in view of Jung teaches The method of claim 8,
wherein the diffusion model comprises a first Unet and a second Unet (Mirzaei Fig. 1 shows two IP2P Unets. “We then use IP2P’s noise prediction Unet, ϵθ, to get two different predictions: . . ..” Mirzaei 4.1 Relevance map calculation. “Denoising diffusion models have enabled high-quality image generation and editing. We present a method to localize the desired edit region implicit in a text instruction. We leverage InstructPix2Pix (IP2P) and identify the discrepancy between IP2P predictions with and without the instruction.” Mirzaei Abstract.),
wherein the generating the plurality of relevance maps respectively corresponding to the plurality of images, comprises:
generating a first noisy image of each of the plurality of images by adding first noise to the edited image by using the diffusion model (
“We then use IP2P’s noise prediction Unet, ϵθ, to get two different predictions: . . ..” Mirzaei 4.1 Relevance map calculation. “Denoising diffusion models have enabled high-quality image generation and editing. We present a method to localize the desired edit region implicit in a text instruction. We leverage InstructPix2Pix (IP2P) and identify the discrepancy between IP2P predictions with and without the instruction. This discrepancy is referred to as the relevance map. The relevance map conveys the importance of changing each pixel to achieve the edits, and is used to to guide the modifications. This guidance ensures that the irrelevant pixels remain unchanged.” Mirzaei Abstract.);
inputting the each of the plurality of images (Within Mirzaei Fig. 1: Original Image) and the first noisy image to the first Unet (left IP2P Unet) without any text instruction (“” in the figure indicting without text instruction) and inputting the each of the plurality of images (Original Image) and the first noisy image (Noisy Image) to the second Unet (right IP2P Unet) with a text instruction (“Make the owl a falcon”);
obtaining a difference (Within Mirzaei Fig. 1: “Normalized difference”) between a first output of the first Unet and a second output of the second Unet (Within Mirzaei Fig. 1); and
normalizing the obtained difference (Within Mirzaei Fig. 1: “Normalized difference”).
Regarding Claim 12, Mirzaei in view of Jung teaches The method of claim 11, wherein the first Unet and the second Unet are the same InstructPixel2Pixel (IP2P) Unet (Mirzaei Fig. 1 shows two IP2P Unets. “We then use IP2P’s noise prediction Unet, ϵθ, to get two different predictions: . . ..” Mirzaei 4.1 Relevance map calculation. “Denoising diffusion models have enabled high-quality image generation and editing. We present a method to localize the desired edit region implicit in a text instruction. We leverage InstructPix2Pix (IP2P) and identify the discrepancy between IP2P predictions with and without the instruction.” Mirzaei Abstract.).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Levin et al. (“Differential Diffusion: Giving Each Pixel Its Strength”): Figure 1.
Similar to the claimed invention because of the use of relevance map (change map).
However, the disclosed change map’s creation process is different from the claimed invention.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHENGXI LIU whose telephone number is (571)270-7509. The examiner can normally be reached M-F 9 AM - 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZHENGXI LIU/Primary Examiner, Art Unit 2611
Prosecution Insights