Prosecution Insights
Last updated: November 04, 2026
Application No. 17/723,879

SEGMENTED DIFFERENTIABLE OPTIMIZATION WITH MULTIPLE GENERATORS

Non-Final OA §103
Filed
Apr 19, 2022
Priority
Sep 17, 2021 — provisional 63/245,628 +1 more
Examiner
HE, WEIMING
Art Unit
2611
Tech Center
2600 — Communications
Assignee
Czech Technical University In Prague
OA Round
7 (Non-Final)
47%
Grant Probability
Moderate
7-8
OA Rounds
0m
Est. Remaining
59%
With Interview

Examiner Intelligence

Grants 47% of resolved cases
47%
Career Allowance Rate
199 granted / 426 resolved
-15.3% vs TC avg
Moderate +12% lift
Without
With
+12.1%
Interview Lift
resolved cases with interview
Typical timeline
3y 4m
Avg Prosecution
23 currently pending
Career history
465
Total Applications
across all art units

Statute-Specific Performance

§101
8.0%
-32.0% vs TC avg
§103
61.6%
+21.6% vs TC avg
§102
10.9%
-29.1% vs TC avg
§112
15.2%
-24.8% vs TC avg
Black line = Tech Center average estimate • Based on career data from 426 resolved cases

Office Action

§103
Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Continued Examination Under 37 CFR 1.114 A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 6/17/26 has been entered. Response to Amendment The amendment filed on 6/17/26 has been entered and made of record. Claims 1, 8 and 15 are amended. Claims 5, 12 and 19 are cancelled. Claims 1-4, 6-11, 13-18 and 20 are pending. Response to Arguments Applicant’s arguments with respect to claims 1, 8 and 15 have been fully considered but they are not persuasive. Applicant argues that the cited references, either alone or in combination, do not teach or suggest, at least, "generating, by the plurality of separate generator models using the plurality of latent codes, a plurality of generated images, wherein each generator model generates a separate image using a latent code for producing an image having a generated segment that matches the corresponding segment of the input image, and wherein other generated segments of the generated image are not required to match corresponding segments of the input image," as recited in amended claim 1 (p. 9 of Remarks). Applicant states that specification discloses “Specifically in the face editing scenario this results in more accurate projection, with superior identity preservation and the ability to cleanly spatially disentangle partial edits. As discussed, by segmenting the object being generated, the search is less constrained. Accordingly, the most accurate projection for a specific segment of the image may result in an overall image that does not closely resemble the input image. For example, while the generated eyes segment may closely match the input image, the rest of the generated face may not resemble the input at all. This allows for the most accurate segment to be used rather than having to attempt to identify the projection that matches the entirety of the image" in [0027]; "The resulting images are then provided to segmentation layer 304, which stitches (e.g., composites) the appropriate segments from the images to create generated image 306" in [0041]. In this example, the face segment (i.e. eye or nose etc.) can be edited and then stitched to obtain a final image. Examiner notices that Ling teaches segmentation of an image, edit segmentation mask and learn editing vector in latent space in Fig 1A & 1B. Here, one or more vehicle components can be edited (i.e. enlarge a front wheel) by learning an editing vector in a lateen space of a generative adversarial network, and then stitched to obtain a final image (generated image). Here, many editable segments are not required to match corresponding segments of the input image because each editable segment can be edited within a latent space. Ling is silent on a plurality of separate generator models. Qian further discloses “In the present invention, the real image is decomposed into a plurality of sub-modes by the mode decomposer, the MIMO-GAN module is used to generate the complete sub-modal image of the real image, and finally the output image is obtained by the mode synthesizer. In the present invention, the real image is decomposed into multiple sub-patterns through a mode decomposer, and multiple generators and multiple discriminators in the MIMO-GAN module are used to fight the game to capture the complete modal information of the real image… The MIMO-GAN module is composed of a generator set composed of multiple generators” at p. 2-3; see also Fig 2 below: PNG media_image1.png 456 635 media_image1.png Greyscale . Claim Rejections - 35 USC § 103 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claims 1-4, 6-11, 13-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ling et al. (US 2022/0383570) in view of Qian et al. (CN 111260652A) and Cherian et al. (US 2022/0309672 A1) further in view of Lee et al. (KR102225753 B1), Arora et al. (Seam Reconstruct: Dynamic Scene Stitching with Large Exposure Difference, 2009 Second International Conference on the Applications of Digital Information and Web Technologies, Aug. 04-06, 2009) and Ravi et al. (US 2020/0125956 A1). As to Claim 1, Ling teaches A computer-implemented method comprising: receiving an input image and a segmentation mask, the segmentation mask dividing an object depicted in the input image into a plurality of segments (Ling discloses part segmentation masks, for example, a mask for a headlight of a car in [0022]; “Further, to train the GAN 102 and to perform segmentation on a new image, an input or original image may be embedded into the latent space 110 of the GAN 112“ in [0025]; “As such, annotated images, x, may be embedded from a dataset labeled with semantic segmentations into the latent space 110, and the semantic segmentation branch of the generator of the GAN 112 may be trained using, e.g., standard supervised learning objectives (e.g., cross entropy loss)” in [0026]. Here, the segmentation mask can be used to decompose a car in the image into a plurality of segments. Qian also discloses “The mode decomposer is used to receive the real image and decompose the real image into multiple real image sub-patterns” at p. 2); determining, by a plurality of separate generator models corresponding to the plurality of segments, a plurality of latent codes, wherein each separate generator model determines a latent code from its separate latent space, and wherein each of the plurality of latent codes corresponding to a separate generator are independently optimized for a specific segment of the plurality of segments based on contents of a corresponding segment of the input image; generating, by the plurality of separate generator models using the plurality of latent codes, a plurality of generated images, wherein each generator model generates a separate image using a latent code for producing an image having a generated segment that matches the corresponding segment of the input image, and wherein other generated segments of the image are not required to match corresponding segments of the input image; and compositing the generated segment from each of the plurality of generated images into an output image corresponding to the input image (Ling discloses “When training the GAN 102, the generator of the GAN may map latent codes z E Z, drawn from a multivariate normal distribution, into realistic images… The GAN 112 may thus use the joint distribution p(x, y) to perform high-precision semantic image editing of real and synthesized images. As such, the GAN 112 may model p(x, y) by adding an additional segmentation branch to the image generator” in [0024]; synthesizes segmentation outputs of the GAN 112 in [0029]. Ling is silent on a plurality of separate generator models. Qian further discloses “In the present invention, the real image is decomposed into a plurality of sub-modes by the mode decomposer, the MIMO-GAN module is used to generate the complete sub-modal image of the real image, and finally the output image is obtained by the mode synthesizer. In the present invention, the real image is decomposed into multiple sub-patterns through a mode decomposer, and multiple generators and multiple discriminators in the MIMO-GAN module are used to fight the game to capture the complete modal information of the real image… The MIMO-GAN module is composed of a generator set composed of multiple generators and a judge set composed of multiple judges; the MIMO-GAN module accepts the sub-modes and input signals of the real image from the mode resolver , Through the adversarial game of the discriminator, multiple sub-mode outputs are generated. The mode synthesizer is used to receive the m sub-modes output by the generator of the MIMO-GAN module, merge these sub-modes to generate a complete image, and obtain the final output result. An image generation method based on MIMO-GAN, using an image generation system based on MIMO-GAN, includes the following steps: Step S1: input the real image into the mode decomposer for mode decomposition, and obtain n sub-patterns of the real image; Step S2: Input the input signal into m generators respectively to obtain m generated initial sub-patterns; Step S3: Fix the parameters of all generators in the MIMO-GAN module and read in the sub-patterns of the n real images in step S1 and the initial sub-patterns generated in step S2, train n discriminators in turn to obtain MIMO -The parameters of all discriminators in the GAN module” at p. 2-3; see also Fig 2 below: PNG media_image1.png 456 635 media_image1.png Greyscale . Ling also teaches optimizing the latent code for a specific segment, such as, “The image editing may be achieved by using segmentation mask modifications (e.g., provided by a user, or otherwise) to optimize the latent code to be consistent with the updated segmentation, thus effectively changing the original, e.g., RGB image. To improve efficiency of the system, and to not require optimizations for each edit on each image, editing vectors may be learned in latent space that realize the edits and that can be directly applied on other images with or without additional optimizations” in [0006]; “At a high level, the process 100 may include embedding an image into a GANs 112 latent space 110, and condition latent code optimization may be executed according to an edit(s) to the segmentation mask. As a result, the corresponding image generated by the jointly modeled GAN may also be modified to match the edits to segmentation mask. To memorialize or amortize the optimization for a given edit, one or more editing vectors 122 may be learned in the latent space 110 that realize the edits. This framework allow for learning any number of editing vectors corresponding to any number of different edit types (e.g., enlarge wheels, shrink headlights, remove trunk of sedan, add a smile to a person, change gaze direction of a person, add frown to image depicting a painting or other out-of-domain image type, etc.)” in [0022]; “For example, with respect to FIG. 1B, the semantic segmentations 114A may be modified and the latent code z may be optimized for consistency with the new segmentation 114B within the editing region (e.g., using a first loss function 134A), and with the new image 116B appearance outside of the editing region” in [0033]. See also Above “Response to Arguments”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Ling with the teaching of Qian so as to turn the traditional GAN composed of a single generator and a single discriminator into one a GAN with a multiple input and output structure composed of multiple generators and multiple judges, and apply the invention not only to simple single-mode images, but also to complex multi-mode images, and even Cross-mode signals, such as image signals, language signals, and text signals (Qian, p. 5). In response to the argument that Qian fails to describe how the ''plurality of latent codes" are selected for a particular GAN or that they are selected with any relation to the received input image, examiner notices that it is inherent that each latent code is extracted from the corresponding segment. For example, Cherian discloses “Some embodiments include the InSeGAN designed with a generator module that, instead of taking a single noise vector as input (as in standard GANs), the generator module takes noise vectors, and each noise vector of d-dimensions from a standard normal distribution, and generates a multiple-instance depth image as output, as noted above” in [0013]; “At least one benefit of taking the multiple random noise vectors (equal to the number of instances) as input to produce instance-specific feature maps, such that when training the model in a generative-adversarial setting, the InSeGAN framework implicitly forces the generator to capture the distribution of single object instances” in [0014]; “The instance encoder produces latent vectors for each 3D object instance. The latent vectors are iteratively used to produce corresponding 3D transformation matrices by the pose encoder, to produce single instance depth images by the generator consisting of only one instance of the 3D object in each depth image” in [0021]; see also [0016, 0018] and Fig 1 & 4. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Ling and Qian with the teaching of Cherian so as to take the multiple random noise vectors [latent code] as input to produce instance-specific feature maps, and explicitly force the generator to capture the distribution of single object instances when training the model in a generative-adversarial setting (Cherian, [0014]). Ling, Qian and Cherian don’t directly teach stitching loss and seam visibility. The combination of Lee, Arora and further teaches following limitations: wherein during compositing the segmentation mask is dynamically updated to change boundaries of one or more segments by backpropagating gradients resulting from a stitching loss calculated by a stitching layer, wherein the stitching loss evaluates a visibility of stitching at the boundaries of the one or more segments (Ling discloses “For example, with respect to FIG. 1B, the semantic segmentations 114A may be modified and the latent code z may be optimized for consistency with the new segmentation 114B within the editing region (e.g., using a first loss function 134A), and with the new image 116B appearance outside of the editing region (e.g., using a second loss function 134B). To do this, in embodiments, corresponding gradients may be backpropagated 132 through the shared generator of the GAN 112…” in [0033]; “The processor of claim 2, wherein at least one latent code optimization iteration of the one or more latent code optimization iterations includes backpropagating one or more gradients through the GAN” in claim 3. Lee further discloses “The stitching error segmentation model may be learned considering stitching error classification loss, bounding box regression loss, and mask prediction loss” in [0017]; “By providing a deep learning-based quality evaluation method and device for a stitched panoramic image according to an embodiment of the present invention, the quality of the stitched image can be evaluated based on stitching distortion segmentation, and there is an advantage in that real-time quality evaluation is possible” in [0023]; quality evaluation unit in [0087-0089]. Here, Lee’s quality evaluation unit may have to consider seam visibility. For example, Arora discloses “The Seam Evaluation Approach is a network based algorithm[9] which avoids “cutting through” objects where a seam would look unnatural. For this approach firstly we implemented Graph Cut technique[3] which is based on the concept of maxflow theorem. In this approach the difference of pixel value at the overlapping pixels was used to find an optimal cut so that the seam visibility is least. This cost function is given below: PNG media_image2.png 414 644 media_image2.png Greyscale Please notes that Ling’s backpropagating gradients resulting from a loss, for example, Ravi discloses “backpropagating a gradient of a combined loss function” in claim 2.) It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the invention of Ling, Qian and Cherian with the teaching of Lee so as to provide a deep learning-based real-time quality evaluation method for a stitched image based on stitching distortion segmentation (Lee, [0023]). The motivation of combining the invention of Arora is to propose a more efficient cost function with accounts for the edges present in the input images (Arora, p. 576), while the combination of Ravi is to explain backpropagating a gradient of a combined loss function. As to Claim 2, Ling in view of Qian, Cherian, Lee, Arora and Ravi teaches The computer-implemented method of claim 1, further comprising: receiving a request to edit a first portion of the input image; determining a segment of the input image corresponding to the first portion of the input image; generating, by a separate generator model corresponding to the segment of the input image, an edited image by exploring a latent space associated with the separate generator model; and generating an edited output image by compositing the edited image with the input image (Ling discloses “For example, the segmentation mask 114A generated by the semantic segmentation branch of the GAN 112 may by edited by, e.g., a user 120, and/or may be edited using an automated editing process to generate the edited segmentation mask 114B… The edit(s) may include adjustments to one or more features of one or more objects represented in the segmentation mask 114A and/or the generated image 116A… The edits may be made using interactive digital painting or labeling tools, in examples, to manually modify the segmentation according to a desired edit” in [0027]; “For example, in FIG. 2A, a vehicle in original image 202A may have its shape changed in image 202B, its wheels enlarged in image 202C, and/or its front light shrunk in image 202D. As another example, in FIG. 2B, a person in original image 204A may have a frown in image 204B, a change in gaze direction to the left in image 204C, and/or a smile in image 204D” in [0028]. Qian further discloses “The MIMO-GAN module consists of two parts, one is the generator set, and the other is the judge set. The generator set consists of m generator networks, which are respectively denoted as generator G1, generator G2,..., generator Gm, which can generate m sub-patterns. These m sub-patterns are composed of m generators. The input signal z is transformed, and the distribution of these signals is Pz…” at p. 5-6, see also Fig 2-3.) As to Claim 3, Ling in view of Qian, Cherian, Lee, Arora and Ravi teaches The computer-implemented method of claim 1, wherein the plurality of separate generator models are clones of a single generator model such that the plurality of separate generator models have a same parameters and weight values (Ling discloses “the shared generator of the GAN 112” in [0033]. Qian also discloses “In step S2, m can be set according to actual needs, the minimum can be set to 1, and the maximum does not exceed 1.2n” at p. 3.) As to Claim 4, Ling in view of Qian, Cherian, Lee, Arora and Ravi teaches The computer-implemented method of claim 1, wherein the plurality of separate generator models includes two or more different generator models having different parameters or weight values (Qian discloses “The generator set consists of m generator networks, which are respectively denoted as generator G1, generator G2,..., generator Gm, which can generate m sub-patterns. These m sub-patterns are composed of m generators. The input signal z is transformed, and the distribution of these signals is Pz…” at p. 5-6, see also Fig 2-3.) As to Claim 6, Ling in view of Qian, Cherian, Lee, Arora and Ravi teaches The computer-implemented method of claim 1, wherein receiving an input image and a segmentation mask further comprises: processing the input image using a semantic segmentation model to generate the segmentation mask (Ling discloses “For example, with respect to FIG. 1B, the semantic segmentations 114A may be modified and the latent code z may be optimized for consistency with the new segmentation 114B within the editing region” in [0033].) As to Claim 7, Ling in view of Qian, Cherian, Lee, Arora and Ravi teaches The computer-implemented method of claim 1, wherein receiving an input image and a segmentation mask further comprises: receiving an input identifying at least one segment of the segmentation mask via a user interface, wherein the input includes painting the at least one segment on a representation of the input image (Ling discloses “For example, the segmentation mask 114A generated by the semantic segmentation branch of the GAN 112 may by edited by, e.g., a user 120… The edit(s) may include adjustments to one or more features of one or more objects represented in the segmentation mask 114A and/or the generated image 116A…. The edits may be made using interactive digital painting or labeling tools, in examples, to manually modify the segmentation according to a desired edit” in [0027].) Claim 8 recites similar limitations as claim 1 but in a computer readable medium form. Therefore, the same rationale used for claim 1 is applied. Claim 9 is rejected based upon similar rationale as Claim 2. Claim 10 is rejected based upon similar rationale as Claim 3. Claim 11 is rejected based upon similar rationale as Claim 4. Claim 13 is rejected based upon similar rationale as Claim 6. Claim 14 is rejected based upon similar rationale as Claim 7. Claim 15 recites similar limitations as claim 1 but in a system form. Therefore, the same rationale used for claim 1 is applied. Claim 16 is rejected based upon similar rationale as Claim 2. Claim 17 is rejected based upon similar rationale as Claim 3. Claim 18 is rejected based upon similar rationale as Claim 4. Claim 20 is rejected based upon similar rationale as Claim 6. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEIMING HE whose telephone number is (571)270-1221. The examiner can normally be reached on Monday-Friday, 8:30am-5:00pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached on 571-272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /WEIMING HE/ Primary Examiner, Art Unit 2611
Read full office action

Prosecution Timeline

Show 19 earlier events
Aug 27, 2025
Non-Final Rejection mailed — §103
Nov 24, 2025
Applicant Interview (Telephonic)
Nov 24, 2025
Examiner Interview Summary
Nov 24, 2025
Response Filed
Dec 17, 2025
Final Rejection mailed — §103
Jun 17, 2026
Request for Continued Examination
Jun 22, 2026
Response after Non-Final Action
Aug 20, 2026
Non-Final Rejection mailed — §103 (current)

Precedent Cases

Applications granted by this same examiner with similar technology

Patent 12771928
METHOD AND SYSTEM FOR DISPLAYING AND MANAGING A SITUATION IN THE ENVIRONMENT OF AN AIRCRAFT
4y 7m to grant Granted Oct 20, 2026
Patent 12762795
Computer Vision Systems and Methods for Supplying Missing Point Data in Point Clouds Derived from Stereoscopic Image Pairs
4y 7m to grant Granted Oct 13, 2026
Patent 12725332
SELECTING FASHION ITEMS FOR AN AVATAR
2y 4m to grant Granted Sep 01, 2026
Patent 12720007
IMAGE DISPLAYING METHOD, INFORMATION PROCESSING APPARATUS, AND NON-TRANSITORY COMPUTER-READABLE STORAGE MEDIUM STORING PROGRAM
3y 3m to grant Granted Aug 25, 2026
Patent 12703209
VEHICLE MONITORING SYSTEM
1y 6m to grant Granted Aug 11, 2026
Study what changed to get past this examiner. Based on 5 most recent grants.

Strategy Recommendation AI-generated — please review before filing

Get a prosecution strategy drawn from examiner precedents, rejection analysis, and claim mapping.
Typically takes 5-10 seconds — AI-generated, attorney review required before filing

Prosecution Projections

7-8
Expected OA Rounds
47%
Grant Probability
59%
With Interview (+12.1%)
3y 4m (~0m remaining)
Median Time to Grant
High
PTA Risk
Based on 426 resolved cases by this examiner. Grant probability derived from career allowance rate.

Sign in with your work email

Enter your email to receive a magic link. No password needed.

Personal email addresses (Gmail, Yahoo, etc.) are not accepted.

Free tier: 3 strategy analyses per month