DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Prior arts cited in this office action:
Cheng et al. (LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation, Feb. 2023, hereinafter “Cheng”)
Zhang et al. (US 20230081171 A1, hereinafter “Zhang”)
Feng et al. (US 20240311960 A1, hereinafter “Feng”)
Zhao et al. (CN 113721764 A, hereinafter “Zhao”)
Karmakar et al. (US 20220415012 A1, hereinafter “Karmakar”)
Response to Arguments
Applicant arguments/remarks filed on 10/23/2025 have been fully considered and are moot in view of the new ground of rejections set forth below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 7-9, 11-20 are rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation, Feb. 2023, hereinafter “Cheng”) in view of Zhang et al. (US 20230081171 A1, hereinafter “Zhang”) and in view of Feng et al. (US 20240311960 A1, hereinafter “Feng”).
Regarding claim 1:
Cheng teaches a method comprising:
receiving, from a user device, an image generation request comprising a prompt (Cheng section 3.4, where Cheng teaches The null embedding is also used for classifier-free guidance (CFG)[9]. CFG was proposed for improving the sample quality of conditional DMs. During CFG, the DM predicts denoised target of both positive condition (e.g., text prompt) and negative condition (e.g., an empty string for text prompt);
responsive to receiving the image generation request, generating, by a processor- implemented automated image generator and based on the prompt, a first image having a first aspect ratio (Cheng section 3.4, where Cheng teaches The null embedding is also used for classifier-free guidance (CFG)[9]. CFG was proposed for improving the sample quality of conditional DMs. During CFG, the DM predicts denoised target of both positive condition (e.g., text prompt) and negative condition (e.g., an empty string for text prompt);
determining one or more candidate regions of interest in the first image (Cheng section 4.6, where Cheng discloses perform object recognition by perform alignment of a desired object with objects in the image to determine which one matches the best);
determining a prompt alignment indicator for each of the one or more candidate regions of interest in the first image, the prompt alignment indicator indicating a level of alignment between each candidate region of interest and the prompt image (Cheng sections 4.2 and 4.6, where Cheng discloses determine YOLO score uses a pretrained YOLOv4 model to detect the objects in the generated images. This metric reflects both the generation quality as well as the align-ment fidelity to the reference layout.;
determining a target region of interest among the one or more candidate regions of interest in the first image, the target region of interest being determined automatically based on a comparison among one or more prompt alignment indicators for the one or more region of interest (Cheng section 4.6, where Cheng teaches We use YOLO score [21] and SceneFID [21] to evaluate the recognizability of the object in the generated images. YOLO score uses a pretrained YOLOv4 [1] model to detect the objects in the generated images. This metric reflects both the generation quality as well as the alignment fidelity to the reference layout. SceneFID is the FID score computed on the cropped objects, which measures the distribution difference between real and generated objects. Better quality images should generate easier-to-recognize objects and therefore have higher YOLO score and lower SceneFID);
processing the first image to obtain a second image having a second aspect ratio that is different from the first aspect ratio (Cheng section 4.6, where Cheng teaches, we use YOLO score [21] and SceneFID [21] to evaluate the recognizability of the object in the generated images. YOLO score uses a pretrained YOLOv4 [1] model to detect the objects in the generated images. This metric reflects both the generation quality as well as the alignment fidelity to the reference layout. SceneFID is the FID score computed on the cropped objects, which measures the distribution difference between real and generated objects. Better quality images should generate easier-to-recognize objects and therefore have higher YOLO score and lower SceneFID).
Cheng fails to explicitly teach where causing presentation of the second image on the user device
However, Cheng teaches The objects generated by LayoutDiffuse align with layout better (higher YOLO score) and are more realistic (lower SceneFID) (Cheng section 4.6, table 4). Zhang in the same line of endeavor teaches Text-to-image synthesis can be configured as a conditional generation task. It is desirable that generated images be realistic and well aligned with a given textual description. In some embodiments, the mutual information may be based on a contrastive loss between: (a) an image and an associated textual description, (b) a known image and a predicted image for a same associated textual description, and (c) portions of an image and corresponding portions of an associated textual description. To achieve this, the mutual information between the corresponding pairs may be optimized, where the pairs include: (1) an image and a sentence, (2) a generated image and a real image, both corresponding to the same textual description, and (3) image regions and words. Directly maximizing mutual information may be challenging; however, a lower bound of the mutual information may be maximized by optimizing contrastive (i.e., InfoNCE) losses. FIG. 3 is a diagram illustrating an example discriminator 300 for a text-to-image synthesis model, in accordance with example embodiments. Real image 302A (e.g., first real image 115 of FIG. 1) and generated image 302B (e.g., first generated image 130 of FIG. 1, such as, for example, generated image 270 of FIG. 2 generated by generator 200) may correspond to a same textual description (e.g., caption 210 of FIG. 2, or caption 320). Image encodings for real image 302A and generated image 302B may be passed through one or more down-sampling blocks 304A until the spatial dimension for the image encodings is reduced to 16×16. As indicated by legend 335, the one or more down-sampling blocks 304A may include one or more convolutional/MLP neural networks (Zhang [0058], [0080]-[0081], figs 2 and 3).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the application to receive a prompt such as a text, to generate an image or video according to the text and determine a region of interest accordingly or optimizing the first image which can correspond to a second image with different aspect ratio than the first image generated and present the optimized image to a user device, in order for the best and the most relevant image maybe presented to the user.
Cheng in view of Zhang fails to explicitly teach cropping the first image at the target region of interest to obtain a second image, the second image having a second aspect ratio that is different from the first aspect.
However, Feng teaches in any event, in response to receiving an extended image 704, the image aspect ratio adjuster 68 identifies visual features of the extended image 704. The image aspect ratio adjuster 68 then applies the visual features to the ROI machine learning model 720 to identify the ROI 706 in the extended image. In response to identifying the ROI 706, the image aspect ratio adjuster 68 selects a portion of the extended image for cropping the extended image around the identified ROI using the aspect ratio of the display area. For example, the image aspect ratio adjuster 68 may generate a first box around the ROI. The image aspect ratio adjuster 68 may also generate a second box having the same aspect ratio as the display area for cropping the extended image. The image aspect ratio adjuster 68 may adjust the position of the second box so that the first box fits within the second box. In some implementations, the second box must fit within the boundaries of the extended image (Feng [0075]-[0082], figs. 5-9).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to select a cropped region of interest from a first image to obtain a second image having a second aspect ratio that is different from the first aspect, as taught by Feng, in other to properly display the cropped image such that relevant information can fit, displayed and be viewed in the desired display.
Regarding claim 2:
Cheng in view of Zhang and in view of Feng teaches wherein the prompt is a text prompt (Cheng section 3, where Cheng teaches LayoutDiffuse adapts a foundational DM, either text-conditioned or unconditioned, for layout-to-image generation. We leverage the latent diffusion model (LDM) [31] as our backbone, which has demonstrated the capability to generate high-quality images with less computational cost. We propose two components, layout attention (Sec. 3.4) and task-adaptive prompts (Sec. 3.5)).
Regarding claim 3:
Cheng in view of Zhang and in view of Feng teaches wherein the processor-implemented automated image generator comprises a text-to-image machine learning model (Cheng section 3, where Cheng teaches Task-adaptive prompts are cues for the model to recognize if the generation task has been changed to layout-to-image from the pretraining task (i.e., unconditional image generation or text-to-image generation).
Regarding claim 4:
Cheng in view of Zhang and in view of Feng teaches wherein the text-to-image machine learning model is a diffusion model (Cheng section 3, where Cheng teaches LayoutDiffuse adapts a foundational DM, either text-conditioned or unconditioned, for layout-to-image generation. We leverage the latent diffusion model (LDM) [31] as our backbone, which has demonstrated the capability to generate high-quality images with less computational cost. We propose two components, layout attention (Sec. 3.4) and task-adaptive prompts (Sec. 3.5)).
Regarding claim 5:
Cheng in view of Zhang and in view of Feng teaches further comprising, prior to the cropping of the first image to obtain the second image, upsampling the first image by applying a uniform scaling factor to a width and a height of the first image (Cheng section 2, where Cheng teaches these methods encode the layout as an image that is downsampled and upsampled jointly with the data. StyleGAN [15] architecture is used in pSp [30] for mask-to-image; Feng [0075]-[0082], figs. 5-9).
Regarding claim 7:
Cheng in view of Zhang and in view of Feng teaches wherein teach candidate region of interest of the one or more candidate regions of interest is generated as a bounding box with respect to the first image (Cheng section 4.6, where Cheng teaches, We use YOLO score [21] and SceneFID [21] to evaluate the recognizability of the object in the generated images. YOLO score uses a pretrained YOLOv4 [1] model to detect the objects in the generated images. This metric reflects both the generation quality as well as the alignment fidelity to the reference layout. SceneFID is the FID score computed on the cropped objects, which measures the distribution difference between real and generated objects. Better quality images should generate easier-to-recognize objects and therefore have higher YOLO score and lower SceneFID; Feng [0075]-[0082], figs. 5-9).
Regarding claim 8:
Cheng in view of Zhang and in view of Feng teaches further comprising:
encoding, by a text encoder, the prompt to obtain an embedding of the prompt;
encoding, by an image encoder, each candidate region of interest of the one or more candidate regions of interest to obtain an embedding of the candidate region of interest; and
automatically comparing the embedding of each candidate region of interest with the embedding of the prompt to obtain the prompt alignment indicator for the candidate region of interest (Cheng section 3).
Regarding claim 9:
Cheng in view of Zhang and in view of Feng teaches wherein the prompt alignment indicator is an alignment score (Cheng section 4.6, where Cheng teaches, We use YOLO score [21] and SceneFID [21] to evaluate the recognizability of the object in the generated images. YOLO score uses a pretrained YOLOv4 [1] model to detect the objects in the generated images. This metric reflects both the generation quality as well as the alignment fidelity to the reference layout. SceneFID is the FID score computed on the cropped objects, which measures the distribution difference between real and generated objects. Better quality images should generate easier-to-recognize objects and therefore have higher YOLO score and lower SceneFID; Feng [0075]-[0082], figs. 5-9).
Regarding claim 11:
Cheng in view of Zhang and in view of Feng teaches further comprising: prior to the cropping, automatically adjusting the cropping region such that the cropping region has the second aspect ratio (Cheng sections 4.2 and 4.6, where Cheng teaches SceneFID computes the FID on crops of all objects and is tailored to the layout-to-image generation task. We follow [21] to train a ResNet-101 on generated object crops and test on validation crops. SceneFID is the FID score computed on the cropped objects, which measures the distribution difference between real and generated objects. Better quality images should generate easier-to-recognize objects and therefore have higher YOLO score and lower SceneFID; Feng [0075]-[0082], figs. 5-9).
Regarding claim 12:
Cheng in view of Zhang and in view of Feng teaches wherein the receiving the image generation request comprises:
causing presentation of an input text box in a user interface provided by an interaction client executing on the user device; and
receiving user input comprising the prompt via the input text box in the user interface, wherein the causing presentation of the second image on the user device comprises causing presentation of the second image in the user interface provided by the interaction client (Cheng sections 3.5 and 4.3, Feng [0075]-[0082], figs. 5-9).
Regarding claim 16:
Cheng in view of Zhang and in view of Feng teaches wherein the processor-implemented automated image generator comprises a text-to-image machine learning model, the text-to-image machine learning model being trained using a training data set comprising a plurality of training images, each training image having a corresponding text description forming part of the training data set (Cheng sections 3.5 and 4.3, Feng [0075]-[0082], figs. 5-9).
Regarding claim 17:
Cheng in view of Zhang and in view of Feng teaches wherein, for each training image, the corresponding text description comprises a first text description and a second text description, the first text description being different from the second text description, and the second text description being a caption generated using a processor-implemented automated caption generator (Cheng sections 3.5 and 4.3, Feng [0075]-[0082], figs. 5-9).
Regarding claim 18:
Cheng in view of Zhang and in view of Feng teaches wherein the processor-implemented automated caption generator comprises an image-to-text machine learning model (Cheng sections 3.5 and 4.3, Feng [0075]-[0082], figs. 5-9).
Regarding claim 13-15:
Cheng in view of Zang and in view of Feng fails to explicitly teaches the specific aspect ratio as claimed by the applicant.
However, Cheng teaches generating a plurality of bounding boxes. Each of the bounding box correspond to a particular object. The object can be of any size and so are the bounding boxes so that the object can be properly cropped (Cheng figs.1 and 4; And see also Feng [0041], [0080]-[0082]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to cropped or generate the image with aspect ratio desired such as squared, 1:1, 4:4, rectangle, 4:3, 16: 9, etc, since generating images or videos with these king of aspect ratios is well-known technique in the art that requires no undue burden to realize and can be made so suit any particular screen or display or whichever is easier to process for a particular application
Regarding claim 19 and 20:
Claims 19 and 20 contain similar limitation as claim 1 and are therefore rejected the same ground as claim 1 (See also Zhang [0010], [0015]).
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation, Feb. 2023, hereinafter “Cheng”) in view of Zhang et al. (US 20230081171 A1, hereinafter “Zhang”) and in view of Feng et al. (US 20240311960 A1, hereinafter “Feng”) and in view of Karmakar et al. (US 20220415012 A1, hereinafter “Karmakar”).
Regarding claim 21:
Cheng in view of Zhang and in view of Feng fails to teach wherein the prompt alignment indicator for the candidate region of interest comprises a cosine similarity between the embedding of the candidate region of interest and the embedding of the prompt.
However, Karmakar teaches a system may also employ an image matching techniques such as template matching techniques that loop over an input image at multiple scales for selecting a region with the largest correlation coefficient and using the region as a matched region. Additionally, or alternatively, the system may employ pre-trained image embeddings to identify the presence of a particular product in an image region based on higher cosine similarity between the product image embedding and a convolved part of the image.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to use cosine similarity to determine how close the two regions, since it is a well-known technique that is used to compare image regions and provide predictable result.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Cheng et al. (LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation, Feb. 2023, hereinafter “Cheng”) in view of Zhang et al. (US 20230081171 A1, hereinafter “Zhang”) and in view of Feng et al. (US 20240311960 A1, hereinafter “Feng”) and in view of Zhao et al. (CN 113721764 A, hereinafter “Zhao”).
Regarding claim 22:
Cheng in view of Zhang and in view of Feng fails to teach wherein the prompt is randomly generated in response to a user selection of an interactive element in a user interface.
However, Zhao teaches a human-computer interaction system based on IMU and control and evaluation method wherein in response to the target selection time, whether the target is hit, the target position of clicking, moving distance of the related data, after hitting the cursor will be automatically recentered, a new target will be randomly selected and prompt as the active target; orderly executing k targets as a task to return (Zhao [0027]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to be able to generate a random prompt based on the user selection or clicking of a target, in order to move to the next step of processing and continuing the interaction with the user (Zhao [0027]-[0028]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEDNEL CADEAU whose telephone number is (571)270-7843. The examiner can normally be reached Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chieh Fan can be reached at 571-272-3042. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEDNEL CADEAU/Primary Examiner, Art Unit 2632 January 2, 2026