DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Response to Amendment
In response to the reply filed on 04/13/2026, applicant’s amendments have overcome all objections. The amendment to the specification is entered. Claims 1, 3, 11, 13, 18, and 20 have been amended, claims 2, 12, and 19 are cancelled. Claims 1, 3-11, 13-18, and 20 are pending and under examination.
Response to Arguments
Applicant’s argument filed 04/13/2026 have been fully considered but they are not persuasive.
Applicant argues regarding claim 3, Zhang et al. (US 20240005574 A1) (hereinafter Zhang) describe detection of object. However, Zhang does not disclose pre-processing an object, particularly when the object has already been separated the background of the image which is was captured. See Applicant’s response, pp. 8.
Examiner’s replies, Zhang discloses “As illustrated in FIG. 4, the object detection machine learning model 408 detects several objects for the digital image 416. In some instances, the detection-masking neural network 400 identifies all objects within the bounding boxes. For example, the bounding boxes comprise the approximate boundary area indicating the detected object. An approximate boundary refers to an indication of an area including an object that is larger and/or less accurate than an object mask. In one or more embodiments, an approximate boundary includes at least a portion of a detected object and portions of the digital image 416 not comprising the detected object. An approximate boundary includes any shape, such as a square, rectangle, circle, oval, or other outline surrounding an object. In one or more embodiments, an approximate boundary comprises a bounding box.” in para. [0058];
Zhang discloses the detection of an object and moreover, Zhang discloses approximate boundary that includes outline surrounding an object, which corresponds to one or more preprocessing techniques to the object.
Regarding the remaining arguments: Applicant argues with respect to the amended claim language, which is fully address in the prior art rejection set forth below.
Conclusion: for the aforementioned reasons, the rejections are maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-9, 11, 13-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Fang et al. (“A Comprehensive Pipeline for Complex Text-to-Image Synthesis”, JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY, vol. 35, no. 3. 2020-05-01, page 522-537) in view of Zhang et al. (US 20240005574 A1).
Regarding claim 1, Fang et al. discloses a method of generating imagery, comprising: receiving, with one or more processors, a captured image of an object and a captured background; (see Abstract, “we retrieve the required foreground objects from the foreground object dataset segmented from Microsoft COCO dataset, and retrieve an appropriate background scene image from the background image dataset extracted from the Internet.” and Fig.1, “Finally, we perform post-processing to obtain the blended final synthesis result.”)
receiving text or verbal input specifying scenery to be generated for the object; (see Introduction, “we firstly apply natural language processing tools to parse the input text and extract the names of necessary foreground objects, foreground objects’ attributes, mutual positional relationships, and background scene information.” and Fig.1, “Input Text”)
separating, with the one or more processors, a foreground of the captured image from the captured background, the foreground including the object; (see 4.1 Foreground Objects Retrieval, “We use these masks to separate target objects from other parts of source images to form the foreground object dataset.” and Fig.1, “Foreground Retrieval”)
applying one or more pre-processing techniques to the object; (see 5.2 Scale Adjustment, “We set the sizes of foreground objects according to two factors, that is, intrinsic scaling factor and perspective factor. We first resize every foreground object’s bounding box, make them have the same height and then adjust their scales according to the two factors.” and 5.3 Iterative Optimization with MCMC, “In every movement, we randomly select a foreground object and change its position for a small distance from its current location along one of the surrounding eight directions, and the direction is denoted by d. The moving distance is uniformly sampled from 0 to 30 pixels. The scale of the foreground object is modified subsequently.)
generating, with the one or more processors (see 5 Image Synthesis Using Constrained MCMC, “After selecting satisfactory foreground objects and background scene image, we put the foreground objects in the right place of the background image with a proper size to ensure that all the scene items comply with the constraints required in the input text.” and Fig.1, “Then we optimize the sizes and positions of the foreground objects on the background scene by the constrained MCMC method. Finally, we perform post-processing to obtain the blended final synthesis result.”)
However, Fang et al. fail to disclose
applying one or more post-processing techniques to the generated scenery, including removing the depicted object from the generated imagery, retouching the scenery of the imagery in an area corresponding to the object, and re-inserting the object from the captured image into the imagery.
However, Zhang et al. teach, in the context of imagery generation, generating, with the one or more processors executing an artificial intelligence (AI) model, (see para. [0018], “This disclosure describes one or more embodiments of an object-aware texture transfer system that utilizes a sequence of methods and/or machine learning models to transfer global style features from a source digital image to a target digital image.”; para. [0048], “the object-aware texture transfer system 106 utilizes, as the object detection machine learning model, one of the machine learning models or neural networks described in U.S. patent application Ser. No. 17/158,527, entitled “Segmenting Objects In Digital Images Utilizing A Multi-Object Segmentation Model Framework,” filed on Jan. 26, 2021; or U.S. patent application Ser. No. 16/388,115, entitled “Robust Training of Large-Scale Object Detectors with Noisy Data,” filed on Apr. 8, 2019; or U.S. patent application Ser. No. 16/518,880, entitled “Utilizing Multiple Object Segmentation Models To Automatically Select User-Requested Objects In Images,” filed on Jul. 22, 2019; or U.S. patent application Ser. No. 16/817,418, entitled “Utilizing A Large-Scale Object Detector To Automatically Select Objects In Digital Images,” filed on Mar. 20, 2020; or Ren, et al., Faster r-cnn: Towards real-time object detection with region proposal networks, NIPS, 2015; or Redmon, et al., You Only Look Once: Unified, Real-Time Object Detection, CVPR 2016, the contents of each of the foregoing applications and papers are hereby incorporated by reference in their entirety.”; para. [0107], “To further illustrate, FIG. 9 shows experimental results of an object-aware texture transfer system 106 generating a modified digital image 908 in accordance with embodiments of the present disclosure. Specifically, FIG. 9 shows results of an object-aware texture transfer system 106 transferring global style features from a source digital image 902 to a target digital image 904 while maintaining an object style of an object (i.e., the white car) portrayed within the target digital image 904. As further shown in FIG. 9 , modified digital image 906 is the result of transferring global style features between the source digital image 902 and the target digital image 904 without maintaining the object style of the portrayed object. Indeed, as shown in FIG. 9 , the modified digital image 908 exhibits a significantly more realistic portrayal of the object (i.e., the car) within the image after transference of the global style of the source digital image 802 thereto.” Examiner’s note: modified digital images are results of transferring global style from the source digital image and one of machine learning models or neural networks is used to generate the scenery.)
applying one or more post-processing techniques to the generated scenery, including removing the depicted object from the generated imagery, retouching the scenery of the imagery in an area corresponding to the object, and re-inserting the object from the captured image into the imagery. (see para. [0085], “As discussed above, in some embodiments, the object-aware texture transfer system 106 reinserts one or more extracted objects into a modified digital image after texture transference and harmonizes a background region of the modified digital image proximate to the reinserted objects. For example, FIG. 7 illustrates the object-aware texture transfer system 106 generating a harmonized digital image utilizing a harmonization neural network 706 having a dual-branched neural network architecture. Indeed, as shown in FIG. 7, the object-aware texture transfer system 106 provides a modified digital image 702 with a reinserted object (i.e., the person portrayed in the foreground) and a segmentation mask 704 (e.g., an object mask generated as described above in relation to FIGS. 4-5) corresponding to the reinserted object to a harmonization neural network 706.”)
PNG
media_image1.png
610
1062
media_image1.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include an artificial intelligence (AI) model, AI-generated scenery, and applying one or more post-processing techniques to the generated scenery, including removing the depicted object from the generated imagery, retouching the scenery of the imagery in an area corresponding to the object, and re-inserting the object from the captured image into the imagery in the method disclosed by Fang et al. according to the teaching of Zhang et al. in order to overcome shortcomings of generated digital imagery regarding accuracy, efficiency, and flexibility. (see Background of Zhang et al.)
Regarding claim 3, Fang et al. in view of Zhang et al. disclose all the limitations of claim 1, and Zang et al. further disclose wherein applying one or more pre-processing techniques to the object comprises applying a border stroke to an outline of the object. (see para. [0058] of Zhang et al., “As illustrated in FIG. 4, the object detection machine learning model 408 detects several objects for the digital image 416. In some instances, the detection-masking neural network 400 identifies all objects within the bounding boxes. For example, the bounding boxes comprise the approximate boundary area indicating the detected object. An approximate boundary refers to an indication of an area including an object that is larger and/or less accurate than an object mask. In one or more embodiments, an approximate boundary includes at least a portion of a detected object and portions of the digital image 416 not comprising the detected object. An approximate boundary includes any shape, such as a square, rectangle, circle, oval, or other outline surrounding an object. In one or more embodiments, an approximate boundary comprises a bounding box.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include wherein applying one or more pre-processing techniques to the object comprises applying a border stroke to an outline of the object in the method disclosed by Fang et al. according to the teaching of Zhang et al. in order to overcome shortcomings of generated digital imagery regarding accuracy, efficiency, and flexibility. (see Background of Zhang et al.)
Regarding claim 4, Fang et al. in view of Zhang et al. disclose all the limitations of claim 3, and Zang et al. further disclose wherein applying the border stroke comprises automatically detecting, with the one or more processors, an edge of the object and applying the border stroke to the detected edge in response. (see para. [0059] of Zhang et al., “Upon detecting the objects in the digital image 416, the detection-masking neural network 400 generates object masks for the detected objects. Generally, instead of utilizing coarse bounding boxes during object localization, the detection-masking neural network 400 generates segmentations masks that better define the boundaries of the object. The following paragraphs provide additional detail with respect to generating object masks for detected objects in accordance with one or more embodiments. In particular, FIG. 4 illustrates the object-aware texture transfer system 106 utilizing the object segmentation machine learning model 410 to generate segmented objects in accordance with some embodiments.”)
PNG
media_image2.png
646
1042
media_image2.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include wherein applying the border stroke comprises automatically detecting, with the one or more processors, an edge of the object and applying the border stroke to the detected edge in response in the method disclosed by Fang et al. according to the teaching of Zhang et al. in order to overcome shortcomings of generated digital imagery regarding accuracy, efficiency, and flexibility. (see Background of Zhang et al.)
Regarding claim 5, Fang et al. in view of Zhang et al. disclose all the limitations of claim 1, and Zang et al. further disclose wherein removing the object from the imagery comprises applying a repair mask to the generated imagery (see para. [0085] of Zhang et al., “Indeed, as shown in FIG. 7, the object-aware texture transfer system 106 provides a modified digital image 702 with a reinserted object (i.e., the person portrayed in the foreground) and a segmentation mask 704 (e.g., an object mask generated as described above in relation to FIGS. 4-5) corresponding to the reinserted object to a harmonization neural network 706.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include wherein removing the object from the imagery comprises applying a repair mask to the generated imagery in the method disclosed by Fang et al. according to the teaching of Zhang et al. in order to overcome shortcomings of generated digital imagery regarding accuracy, efficiency, and flexibility. (see Background of Zhang et al.)
Regarding claim 6, Fang et al. in view of Zhang et al. disclose all the limitations of claim 1, and Zang et al. further disclose wherein retouching the scenery of the imagery comprises blurring or feathering the imagery (see para. [0044] of Zhang et al., “Further, in some embodiments, the object-aware texture transfer system 106 utilizes inpainting 314 to fill holes corresponding the objects extracted by segmentation 306. For instance, as shown in FIG. 3, the object-aware texture transfer system 106 utilizes inpainting 314 to fill a hole in the first intermediate target digital image 308 to generate a second intermediate target digital image 316” and para. [0071] of Zhang et al., “In one or more implementations, the object-aware texture transfer system 106 utilizes a content aware fill machine learning model 516 in the form of a deep inpainting model to generate the content (and optionally fill) the hole corresponding to the removed object. For example, the object-aware texture transfer system 106 utilizes a deep inpainting model trained to fill holes.” and para [0071] of Zhang et al., “In one or more implementations, the object-aware texture transfer system 106 utilizes a content aware fill machine learning model 516 in the form of a deep inpainting model to generate the content (and optionally fill) the hole corresponding to the removed object.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include wherein retouching the scenery of the imagery comprises blurring or feathering the imagery in the method disclosed by Fang et al. according to the teaching of Zhang et al. in order to overcome shortcomings of generated digital imagery regarding accuracy, efficiency, and flexibility. (see Background of Zhang et al.)
Regarding claim 7 and claim 8, Fang et al. in view of Zhang et al. disclose the method of claim 1, further comprising upscaling the object prior to generating the imagery and the method of claim 7, further comprising downsizing the object upon placement in the generated imagery. (see 5.2 Scale Adjustment of Fang et al., “We set the sizes of foreground objects according to two factors, that is, intrinsic scaling factor and perspective factor. We first resize every foreground object’s bounding box, make them have the same height, and then adjust their scales according to the two factors.”, also see Fig. 7 of Fang et al, “Scale adjustment by (a) intrinsic scaling factor and (b) perspective factor.”)
PNG
media_image3.png
287
622
media_image3.png
Greyscale
Regarding claim 9, Fang et al. in view of Zhang et al. disclose all the limitations of claim 1, and Zang et al. further disclose further comprising: applying a mask to the captured image, the mask defining a shape of the object; and augmenting the mask (see para. [0047] of Zhang et al., “Thus, the scene-based image editing system 106 dilates (e.g., expands) the object mask of an object to avoid associated artifacts when removing the object. Dilating objects masks, however, presents the risk of removing portions of other objects portrayed in the digital image. For instance, where a first object to be removed overlaps, touches, or is proximate to a second object, a dilated mask for the first object will often extend into the space occupied by the second object. Thus, when removing the first object using the dilated object mask, significant portions of the second object are often removed and the resulting hole is filled in (generally improperly), causing undesirable effects in the resulting image. Accordingly, the scene-based image editing system 106 utilizes smart dilation to avoid significantly extending the object mask of an object to be removed into areas of the digital image occupied by other objects.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include further comprising: applying a mask to the captured image, the mask defining a shape of the object; and augmenting the mask in the method disclosed by Fang et al. according to the teaching of Zhang et al. in order to overcome shortcomings of generated digital imagery regarding accuracy, efficiency, and flexibility. (see Background of Zhang et al.)
Regarding claim 11, the system claim 11 is similar in scope to claim 1 and is rejected under the same rationale.
Regarding claim 13, the system claim 13 is similar in scope to claim 3 and is rejected under the same rationale.
Regarding claim 14, the system claim 14 is similar in scope to claim 4 and is rejected under the same rationale.
Regarding claim 15, the system claim 15 is similar in scope to claim 5 and is rejected under the same rationale.
Regarding claim 16, the system claim 16 is similar in scope to claim 7 and is rejected under the same rationale.
Regarding claim 17, the system claim 17 is similar in scope to claim 8 and is rejected under the same rationale.
Regarding claim 18, the non-transitory computer-readable medium claim is similar cope to the claim 1 and is rejected under the same rationale.
Regarding claim 20, Fang et al. disclose all the limitation of claim 18, but does not disclose wherein removing the object from the imagery comprises applying a repair mask to the generated imagery. However, Zhang et al. teach, in the context of imagery generation, wherein removing the object from the imagery comprises applying a repair mask to the generated imagery (see para. [0085] of Zhang et al., “Indeed, as shown in FIG. 7, the object-aware texture transfer system 106 provides a modified digital image 702 with a reinserted object (i.e., the person portrayed in the foreground) and a segmentation mask 704 (e.g., an object mask generated as described above in relation to FIGS. 4-5) corresponding to the reinserted object to a harmonization neural network 706.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include wherein removing the object from the imagery comprises applying a repair mask to the generated imagery in the method disclosed by Fang et al. according to the teaching of Zhang et al. in order to overcome shortcomings of generated digital imagery regarding accuracy, efficiency, and flexibility. (see Background of Zhang et al.)
Allowable Subject Matter
Claim 10 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement of reasons for allowance:
Prior art does not teach “the method of claim 1, wherein the one or more post-processing techniques comprises detecting whether the object in the generated imagery appears to be floating, the detecting comprising: generating a depth map for the object from the generated imagery; generating an object mask from the depth map; generating a convex hull of the object mask; calculating an integral of the mask while vertically displacing the object mask downward; computing a surface region beneath the object by subtracting the integral from the convex hull mask; computing a depth for the object mask and a depth of the surface region; and computing depth displacement based on the depth for the object mask and the depth of the surface region; and determining whether a normalized value for the computed depth displacement falls within a predetermined range.”, in combination with limitations recited in claim 1.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hyorim Park whose telephone number is (571)272-3859. The examiner can normally be reached Monday - Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Hyorim Park/Examiner, Art Unit 2615
/ALICIA M HARRINGTON/Supervisory Patent Examiner, Art Unit 2615
Prosecution Insights