DETAILED ACTION
Election/Restrictions
Applicant's election with traverse of species II in the reply filed on 3/2/2026 is acknowledged. The traversal is on the ground(s) that ”Text-to-image generation and diffusion-based generation are not mutually exclusive … A search of one species would necessarily encompass the prior art relevant to the other”. This is not found persuasive because election requirement is not based on exclusivity, but is based on distinctiveness. Also it is not obvious to an ordinary skill in the art searching for one is similar to searching for the other, and the following office action shows the contrary.
The requirement is still deemed proper and is therefore made FINAL.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s 1, 4-15, 17, 18 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Jain (“Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand”, Jan 2023)
Regarding claim 1, Jain teaches a method comprising:
obtaining a composite image ( second column from left in fig. 3(a)) and a mask( M in fig. 3(a); Page 210, left column, section 3, M), wherein the composite image includes a high-resolution region (Iorg, in Fig. 3(a); Page 210, right column, section 3., Iorg, ) and a low-resolution region ( black portion in second column from left in fig. 3(a); Page 210, right column, section 3., Ihole) ;
identifying, by an upsampling network, the low-resolution region of the composite image based on the mask region ( black portion in second column from left in fig. 3(a)); and
generating, using the upsampling network, an upsampled composite image based on the composite image and the mask, wherein the upsampled composite image comprises higher frequency details in the low-resolution region than the composite image ( Page 210, left column, section 3.1, During the upsampling process in the generator, global texture features, both in the non-hole regions and in the generated hole regions, can be extracted by fast fourier convolutional layers and integrated appropriately to refine textures).
Regarding claim 4, Jain teaches the method of claim 1, wherein generating the upsampled image comprises:
downsampling the composite image to obtain a downsampled composite image( Encoder in Fig. 3; Page 212, Section 4.2, The encoder ξ downscales the input to a spatial size ); and
upsampling the downsampled composite image to obtain the upsampled composite image( Xskip in Fig. 3; Page 211, right column, FaF-Syn takes in both the encoded skip connected features Xskip, and the features Xskip upsampled from the previous level in the generator).
Regarding claim 5, Jain teaches the method of claim 1, wherein obtaining the composite image comprises:
obtaining a low-resolution image( M in fig. 3(a)) and a high-resolution image(Iorg, in Fig. 3(a)); and
combining the low-resolution image and the high-resolution image to obtain the composite image, wherein the high-resolution region is based on the high-resolution image and the low-resolution region is based on the low-resolution image (second column from left in fig. 3(a)).
Regarding claim 6, Jain teaches the method of claim 1, further comprising: performing a Fast Fourier Convolution (FFC) at a skip connection of the upsampling network(page 215, section 4.4, directly connecting the FFC layers with the skipped features Xskip from the encoder).
Regarding claim 7, Jain teaches the method of claim 5, wherein combining the low-resolution image and the high-resolution image comprises: inserting content of the low-resolution image into the high-resolution image to obtain the composite image(second column from left in fig. 3(a)).
Regarding claim 8, Jain teaches a method comprising:
obtaining training data comprising a composite image including a low-resolution region from a low-resolution image and a high-resolution region from a high-resolution image, a mask indicating the low-resolution region, and a ground-truth composite image( see rejections for claim 1); and
training an upsampling network to generate an upsampled composite image using the training data by upsampling the low-resolution region of the composite image based on the mask( Page 212, section 3.3, For the loss of the generator, similar to LaMa, we use a high receptive field perceptual loss (HRFPL) [26] which computes the ℓ2 distance between Icomp and Iorg, after mapping these images onto higher level features) .
Regarding claim 9, Jain teaches the method of claim 8, wherein training the upsampling network comprises:
obtaining a pretrained upsampling network( Page 212, section 3.3, The feature extractor is based on dilated ResNet-50 [35, 36] and is pre trained for ADE20K [50, 51] semantic segmentation); and
appending a downsampling layer to the pretrained upsampling network to initialize the upsampling network( Encoder in Fig. 3; Page 212, Section 4.2, The encoder ξ downscales the input to a spatial size ).
Regarding claim 10, Jain teaches the method of claim 9, further comprising: appending a Fast Fourier Convolution (FFC) layer to the pretrained upsampling network to obtain the upsampling network( page 215, section 4.4, directly connecting the FFC layers with the skipped features Xskip from the encoder).
Regarding claim 11, Jain teaches the method of claim 8, wherein: the upsampling network is trained as a generative adversarial network (GAN) ( Generator and Discriminator in Fig. 3).
Regarding claim 12, Jain teaches the method of claim 8, wherein obtaining the training data comprises: downsampling the high-resolution image to obtain the low-resolution image( Encoder in Fig. 3; Page 212, Section 4.2, The encoder ξ downscales the input to a spatial size ).
Regarding claim 13, Jain teaches the method of claim 8, wherein: the mask indicating the region of the high-resolution image is created using a mask generation model( Free-Form Mask Generator in Fig. 3) .
Claims 14, 17, 18 recite the apparatus comprising processor for the method in claims 1, 6, 11. Since Jain’s method is inherently carried out in a computer apparatus, those claims are also rejected.
Regarding claim 15, Jain teaches the apparatus of claim 14, wherein: the upsampling network comprises a U-net architecture (Encoder and Generation in fig. 3 ).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 is also at least rejected under 35 U.S.C. 103 as being unpatentable over LIN ( US 20210342984) in view of SERESHT (US 20220335572).
Regarding claim 1, LIN teaches a method comprising:
obtaining a composite image, wherein the composite image includes a high-resolution region and a low-resolution region( 402, 404 in Fig. 4);
generating, using the upsampling network, an upsampled composite image based on the composite image, wherein the upsampled composite image comprises higher frequency details in the low-resolution region than the composite image(408, 410 in Fig. 4).
LIN does not expressly teach
obtaining a mask;
identifying, by an upsampling network, the low-resolution region of the composite image based on the mask and
generating, using the upsampling network, an upsampled composite image based on the mask
However SERESHT teaches
obtaining a mask ( [0030], feature masks);
identifying, by an upsampling network, the low-resolution region of the composite image based on the mask( [0030], a low-resolution counterpart x′ is an input to the generator 110 … The discriminator 120 outputs the realism of image patches d.sub.i,j. The upsampled images {circumflex over (x)} are fed into the resulting semantic feature mask) and
generating, using the upsampling network, an upsampled composite image based on the mask([0033], create upsampled images {circumflex over (x)} based on three signals: pixel-space comparison to high-resolution training images x via Mean Absolute Error, feedback from the discriminator 120 which predicts the realism of image patches d.sub.i,j, and quality of the resulting semantic feature masks).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of LIN and SERESHT, by substituting the upsampling network in LIN with the one taught by SERESHT which obtains a mask, with motivation to “ generates the image realism prediction values“ ( SERESHT, Abstract ).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANGENG SUN whose telephone number is (571)272-3712. The examiner can normally be reached 8am to 5pm, EST, M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Randolph Vincent can be reached at 571 272 8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JIANGENG SUN
Examiner
Art Unit 2661
/Jiangeng Sun/Examiner, Art Unit 2671
Prosecution Insights