Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) was filed on 08/05/2024. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
Examiner makes note of the interview with Keith Vick reg. no. 70055 held on July/21/2026, no agreement was made.
Applicant first argues the 112(f) interpretation concerning “a masking module configured to…” in claim 10 includes the Fourier based decoders and encoders, by mentioning that it does not fall within the 112(f) requirements according to Dyfan v. Target Corp., 28 F.4 1360 (mentions that code and application are not generic placeholders when considered in full. See MPEP 2181). Examiner agrees and removes the 112(f) interpretation (paragraphs 32 and 81-87 show support for system processor).
Applicant secondly argues the newly amended independent claim limitations by mentioning that prior cited art Zheng does not teach the “fast Fourier convolution” applied instead of “Fourier based”. Examiner agrees and presents previously cited prior art Roman to cure the deficiencies of Zheng. Roman shows clear improvement under the KSR rationales with the use of a Fast Fourier convolution instead of just being Fourier based, it is simply provides higher perceptual quality and better performance by improving efficiency, including less training data and less computations (See page 2 column 1 paragraph 4. Roman), one of ordinary skill in the art would implement the use of fast Fourier convolutions in the modulation decoders. Therefore all of the claims 1-20 stay rejected.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Zheng et. al., hereafter Zheng (US Pub. No. 20230360180 A1) in view of Roman et. al., hereafter Roman (Suvorov, Roman, et al. "Resolution-robust Large Mask Inpainting with Fourier Convolutions." arXiv preprint arXiv:2109.07161 (2021).) .
As per claim 1, Zheng teaches “a method of performing image object removal, the method being executed by at least one processor and comprising:
obtaining an input image comprising an object;” (See fig. 2 and paragraph 23 “[0023] In this manner, the image inpainting system can generate and utilize inpainted digital images for a variety of applications. To illustrate, in some implementations the image inpainting system receives, via a client device, a digital image with a replacement region (e.g., a region portraying a distracting object). In response, the image inpainting system generates an inpainted digital image that replaces the distracting object with replacement pixels.” Zheng)
“obtaining a mask corresponding the object in the input image;” (See fig. 2 and paragraph 45 “[0045] For example, the replacement region 204 can include an area, portion, mask, or hole within a digital image to replace, cover, or fill with replacement pixels.” Zheng)
“generating a masked image based on the input image and the mask, wherein the masked image comprises a missing area corresponding to the object;” (See fig. 2 and paragraph 45 “In some embodiments, the image inpainting system 102 identifies the replacement region 204 based on user selection of pixels to move, remove, cover, or replace from a digital image. To illustrate, a client device can select a distracting or undesired object or region of a digital image. The image inpainting system 102 can delete or remove the distracting or undesired object or region and generate replacement pixels. In some case, the image inpainting system 102 identifies the replacement region 204 by generating a digital image mask via a segmentation model (e.g., a segmentation neural network identifying an object to move or remove).” See also paragraph 48 and fig. 2.)
“providing the masked image to a Fourier-based encoder; and” (See fig. 2 and paragraphs 43, 46, 30 and 56 “0056] In one or more embodiments, the image inpainting system 102 utilizes a plurality of Fourier convolutional encoder layer to generate an image encoding (e.g., the encoded feature vectors, the style code 312, and/or the global feature code 316).” Zheng)
“generating a composed image by providing an output of the encoder to a Fourier-based cascaded decoder comprising a… global modulation decoder cascaded with a… spatial modulation decoder, wherein the object is not included in the composed image.” (See figs. 2-3 and paragraphs 49 “[0049] Accordingly, the cascaded modulation inpainting neural network 116 includes a generative inpainting neural network that utilizes a decoder having one or more cascaded modulation decoder layers. Indeed, as illustrated in FIG. 2, the cascaded modulation inpainting neural network 116 includes a plurality of cascaded modulation decoder layers 210-216. For example, a cascaded modulation decoder layer includes at least two connected (e.g., cascaded) modulations blocks for modulating an input signal in generating an inpainted digital image. To illustrate, a cascaded modulation decoder layer can include a first global modulation block and a second global modulation block. Similarly, a cascaded modulation decoder layer can include a first global modulation block (that analyzes global features and utilizes a global, spatially-invariant approach) and a second spatial modulation block (that analyzes local features utilizing a spatially-varying approach). Additional detail regarding modulation blocks will be provided below (e.g., in relation to FIGS. 3, 4).” See also fig. 3 and paragraphs 58 and 52 “[0052] As illustrated, the cascaded modulation inpainting neural network 302 includes an encoder 304 and a decoder 306…” See also paragraph 50 “0050] As shown, the image inpainting system 102 utilizes the cascaded modulation inpainting neural network 116 (and the cascaded modulation decoder layers 210-216) to generate the inpainted digital image 208. Specifically, the cascaded modulation inpainting neural network 116 generates the inpainted digital image 208 by generating replacement pixels for the replacement region 204. As illustrated, the replacement region 204 is now filled with replacement pixels that portray a photorealistic scene in place of the replacement region 204.” Zheng ), however Zheng does not teach “a fast Fourier convolution” applied to the global modulation and spatial modulation.
Roman teaches “fast Fourier convolution…” (See abstract and section 2.1 Global context within early layers pages 2-3 along with fig. 2 on page 3. Page 2 column 1 paragraph 4 shows “We propose an inpainting network based on recently developed fast Fourier convolutions (FFCs) [4]. FFCs al low for a receptive field that covers an entire image even in the early layers of the network. We show that this property of FFCs improves both perceptual quality and parameter efficiency of the network. Interestingly, the inductive bias of FFC allows the network to generalize to high resolutions that are never seen during training (Figure 5, Figure 6). This finding brings significant practical benefits, as less training data and computations are needed.” Roman )
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Zheng with the teachings of Roman to use a fast Fourier convolution for the global modulation decoder and the spatial modulation decoder. The modification would have been motivated by the desire to have better quality and performance, therefore it is an improvement, as suggested by Roman (Page 2 column 1 paragraph 4 shows “We propose an inpainting network based on recently developed fast Fourier convolutions (FFCs) [4]. FFCs al low for a receptive field that covers an entire image even in the early layers of the network. We show that this property of FFCs improves both perceptual quality and parameter efficiency of the network. Interestingly, the inductive bias of FFC allows the network to generalize to high resolutions that are never seen during training (Figure 5, Figure 6). This finding brings significant practical benefits, as less training data and computations are needed.” Roman)
Claim 10 is rejected under the same analysis as claim 1. (The network is based on features as previously seen and also seen on paragraphs 2, 52-63 and 70-85. Zheng)
Claim 20 is rejected under the same analysis as claim 1. (The network is based on features as previously seen and also seen on paragraphs 2, 52-63 and 70-85. Zheng)
As per claim 2, Zheng in view of Roman already teaches “The method of claim 1, wherein each of the encoder, the global modulation decoder, and the spatial modulation decoder comprises at least one… convolution layer.” (Zheng teaches the use of a fast Fourier convolution with the encoder on paragraph 30. In addition, it mentions the use of convolution layers on the global modulation and spatial modulation on paragraphs 19, 21, 71-72 and 75-78. Zheng) , however Zheng does not teach “fast Fourier convolution”.
Roman teaches “fast Fourier convolution” (See abstract and section 2.1 Global context within early layers pages 2-3 along with fig. 2 on page 3. Page 2 column 1 paragraph 4 shows “We propose an inpainting network based on recently developed fast Fourier convolutions (FFCs) [4]. FFCs al low for a receptive field that covers an entire image even in the early layers of the network. We show that this property of FFCs improves both perceptual quality and parameter efficiency of the network. Interestingly, the inductive bias of FFC allows the network to generalize to high resolutions that are never seen during training (Figure 5, Figure 6). This finding brings significant practical benefits, as less training data and computations are needed.” Roman ) The same motivations used in claim 1 also apply.
Claim 11 is rejected under the same analysis as claim 2.
Claim 20 is rejected under the same analysis as claim 2.
As per claim 3, Zheng in view of Roman already teaches “the method of claim 2, wherein the at least one fast Fourier convolution layer is used to generate repeating patterns in the missing area of the input image.” (See page 6 fig. 4, the fast Fourier is used to generate repeating patterns such as those presented in the structures “Figure 4: The side-by-side comparison of various inpainting systems on 512×512 images. Repetitive structures, such as windows and chain-link fences are known to be hard to inpaint. FFCs allow to generate these types of structures significantly better.” See also fig. 1, repetitive patterns were infilled. See also fig. 2. Roman) (See also fig. 2, fig. 5 and fig. 6 in Zheng, it shows that a repeated pattern was infilled and it utilizes FFCs as seen on paragraph 60. Zheng )
Claim 12 is rejected under the same analysis as claim 3.
As per claim 4, Zheng in view of Roman teaches “The method of claim 1, wherein the cascaded decoder corresponds to a generator of a generative adversarial network.” (See paragraphs 47-49 “[0048] Along these lines, a generative adversarial neural network refers to a neural network having a specific architecture or a specific purpose such as a generative inpainting neural network… ” and “[0049] Accordingly, the cascaded modulation inpainting neural network 116 includes a generative inpainting neural network that utilizes a decoder having one or more cascaded modulation decoder layers… ” Zheng)
Claim 13 is rejected under the same analysis as claim 4.
As per claim 5, Zheng in view of Roman teaches “The method of claim 1, wherein the output of the encoder comprises a global style vector which is concatenated with a noisy style vector to generate a global vector,” (See paragraphs 70 “Thus, for example, the global feature map 412 includes a feature vector reflecting global features generated from a previous global modulation block of a cascaded decoder layer. The global feature map 412 can also include a feature vector corresponding to the encoded feature vectors generated by the encoder (e.g., at a first decoder layer the image inpainting system 102 can utilize an encoded feature vector, style code, global feature code, constant, noise vector, or other feature vector as input).” See fig. 3 and paragraph 54 “[0054] As illustrated, in one or more embodiments, the image inpainting system 102 applies a neural network layer (e.g., a fully connected layer) to the final encoded feature vector to generate a style code 312 (e.g., a style vector). In addition, the image inpainting system 102 generates the global feature code by combining the style code 312 with a random style code 314. In particular, the image inpainting system 102 generates the random style code 314 by utilizing a neural network layer (e.g., a multi-layer perceptron) to process an input noise vector. The neural network layer maps the input noise vector to a random style code 314. The image inpainting system 102 combines (e.g., concatenates, adds, or multiplies) the random style code 314 with the style code 312 to generate the global feature code 316…” Examiner interprets “random style code 314” as a noisy style vector, “style code 312” as the global style vector and “global feature code 316” as the global vector. See also paragraphs 55-60. Zheng)
“wherein the global style vector and the global vector are provided to the global modulation decoder, and” (See fig. 3, units 314, 312 are concatenated into 316 which is inputted into the decoder by the global modulation output (faint arrow). See also paragraphs 61-64 Zheng )
“wherein the global style vector and an output of the global modulation decoder are provided to the spatial modulation decoder.” (See fig. 3, the global style vector (312) and the outputs of the global modulation (faint arrow) are provided as input for the spatial modulation in the decoder (bold arrow). See also paragraphs 61-64. Zheng )
Claim 14 is rejected under the same analysis as claim 5.
As per claim 6, Zheng in view of Roman teaches “The method of claim 5, wherein the cascaded decoder comprises a plurality of global modulation blocks and a plurality of spatial modulation blocks, and wherein each global modulation block from among the plurality of global modulation blocks is bridged to a corresponding spatial modulation block from among the plurality of spatial modulation blocks.” (See fig. 3 and fig. 4, it shows that each block global modulation block is connected to a spatial modulation block, therefore it covers the broadest reasonable interpretation (BRI) of being “bridged” and see also paragraphs 18-22 “[0021] In one or more implementations, the image inpainting system cascades information from the global modulation block to the spatial modulation block at each cascaded modulation layer. For example, the image inpainting system generates a spatial tensor for the spatial modulation block from the intermediate global feature of the global modulation block… ”. Therefore, by cascading information, they are interpreted as bridged. See also paragraphs 70, 74, 76 and 81. Zheng )
Claim 15 is rejected under the same analysis as claim 6.
As per claim 7, Zheng in view of Roman teaches “The method of claim 6, further comprising: generating global output features based on the global vector using the plurality of global modulation blocks; providing the global output features to the plurality of spatial modulation blocks;” (See paragraphs 70-72 “[0071] As shown, the first global modulation operation 404 includes a modulation layer 404a, an upsampling layer 404b, a convolutional layer 404c, and a normalization layer 404d. In particular, the image inpainting system 102 utilizes the modulation layer 404a to perform a global modulation of the initial global feature map 412 based on a global feature code 414 (e.g., the global feature code 316). Specifically, the image inpainting system 102 applies a neural network layer (i.e., a fully connected layer) to the global feature code 414 to generate a global feature vector 416.”, “[0072]… As shown, the first global modulation operation 404 generates a global intermediate feature 418. In particular, in one or more embodiments, the image inpainting system 102 generates the global intermediate feature 418 by combining (e.g., concatenating) the output of the first global modulation operation 404 with an encoded feature vector 420 (e.g., from a convolutional layer of the encoder having a matching scale/resolution).” See also paragraph 76 “[0076] As illustrated in FIG. 4, the image inpainting system 102 utilizes the spatial modulation operation 410 to generate a new local feature map 428. Indeed, the spatial modulation operation 410 modulates the local intermediate feature 426 based on the global intermediate feature 418. Specifically, the image inpainting system 102 generates a spatial tensor 430 from the global intermediate feature 418. ” Zheng) “and correcting distortions in the global output features and injecting spatial details into the global output features based on the global style vector using the plurality of spatial modulation blocks.” (See paragraph 76 “For example, the image inpainting system 102 applies a convolutional affine parameter network to generate the spatial tensor 430. In particular, the image inpainting system 102 applies a convolutional affine parameter network to generate an intermediate spatial tensor. The image inpainting system 102 combines the intermediate spatial tensor with the global feature vector 416 to generate the spatial tensor 430. The image inpainting system 102 utilizes the spatial tensor 430 to modulate the local intermediate feature 426 (utilizing the spatial modulation layer 410a) and generated a modulated tensor.” Then see in paragraphs 77-79 where it shows that the spatial modulation contains the normalization layer 410c. The purpose of this layer is to avoid artifacts, such as water droplets which is interpreted as a type of distortion, therefore it implicitly teaches that the spatial modulation corrects distortions such as water droplets “0078] Although illustrated as a normalization layer 410c, in one or more embodiments, the image inpainting system 102 applies a demodulation layer. For example, the image inpainting system 102 applies a modulation-convolution-demodulation pipeline (e.g., general normalization rather than instance normalization). This approach can avoid potential artifacts (e.g., water droplet artifacts) caused by instance normalization. ” See also paragraphs 83-92 shows that spatial details are injected and distortions corrected. See also paragraph 16, the purpose is to correct distorted details of features “[0016]…For example, in each decoder layer, the image inpainting system starts with global code modulation that captures the global-range image structures followed by a spatially adaptive modulation that refines the global predictions. Accordingly, in one or more implementations, the image inpainting system provides a mechanism to correct distorted local details, making the inpainted pixels coherent with the rest of the image globally and locally.”. See also paragraphs 26, 29, 60, 82-83 and 111. Therefore it falls within a BRI of correcting distorted features. Zheng)
Claim 16 is rejected under the same analysis as claim 7.
As per claim 8, Zheng in view of Roman already teaches “The method of claim 6, wherein each global modulation block and each spatial modulation block comprises at least one… convolution layer” (See paragraphs 19 and 21, 71-72 and 75-78. Zheng), however Zheng does not teach “fast Fourier convolution”.
Roman teaches “fast Fourier convolution” (See abstract and section 2.1 Global context within early layers pages 2-3 along with fig. 2 on page 3. Page 2 column 1 paragraph 4 shows “We propose an inpainting network based on recently developed fast Fourier convolutions (FFCs) [4]. FFCs al low for a receptive field that covers an entire image even in the early layers of the network. We show that this property of FFCs improves both perceptual quality and parameter efficiency of the network. Interestingly, the inductive bias of FFC allows the network to generalize to high resolutions that are never seen during training (Figure 5, Figure 6). This finding brings significant practical benefits, as less training data and computations are needed.” Roman) The same reasoning of claim 1 also apply.
Claim 17 is rejected under the same analysis as claim 8.
As per claim 9, Zheng in view of Roman already teaches “The method of claim 8, wherein the method further comprises: performing a fast Fourier transform operation using the at least one fast Fourier convolution layer;”, however only Roman teaches “performing a convolution operation on an output of the fast Fourier transform operation using the at least one fast Fourier convolution layer; and performing an inverse fast Fourier transform on an output of the convolution operation using the at least one fast Fourier convolution layer.” (See fig. 2 and pages 2-3 section 2.1 Global context within early layer, subsection Fast Fourier convolution (FFC) “FFC splits channels into two parallel branches: i) local branch uses conventional convolutions, and ii) global branch uses real FFT to account for global context. Real FFT can be applied only to real valued signals, and inverse real FFT ensures that the output is real valued. Real FFT uses only half of the spectrum compared to the FFT. Specifically, FFC makes following steps:…” Steps a), b) and c) show an FFT (fast Fourier transform) being applied to an output and also the use of an inverse FFT to the output to make sure it is real. Fig. 2 also shows a convolution operation with the FFT and also an inverse to the output. See also pages 2-4 section 2 Method. Zheng)
Claim 18 is rejected under the same analysis as claim 9.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN J MENDEZ MUNIZ whose telephone number is (703)756-5672. The examiner can normally be reached M-F, 8AM - 5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at (571) 272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN JOHN MENDEZ MUNIZ/Examiner, Art Unit 2675
/VU LE/Supervisory Patent Examiner, Art Unit 2668
