Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/01/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
The information disclosure statement (IDS) submitted on 7/16/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
Applicant is reminded of the proper content of an abstract of the disclosure.
A patent abstract is a concise statement of the technical disclosure of the patent and should include that which is new in the art to which the invention pertains. The abstract should not refer to purported merits or speculative applications of the invention and should not compare the invention with the prior art.
If the patent is of a basic nature, the entire technical disclosure may be new in the art, and the abstract should be directed to the entire disclosure. If the patent is in the nature of an improvement in an old apparatus, process, product, or composition, the abstract should include the technical disclosure of the improvement. The abstract should also mention by way of example any preferred modifications or alternatives.
Where applicable, the abstract should include the following: (1) if a machine or apparatus, its organization and operation; (2) if an article, its method of making; (3) if a chemical compound, its identity and use; (4) if a mixture, its ingredients; (5) if a process, the steps.
Extensive mechanical and design details of an apparatus should not be included in the abstract. The abstract should be in narrative form and generally limited to a single paragraph within the range of 50 to 150 words in length.
See MPEP § 608.01(b) for guidelines for the preparation of patent abstracts.
The abstract of the disclosure does not commence on a separate sheet in accordance with 37 CFR 1.52(b)(4) and 1.72(b). A new abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text.
Claim Objections
Claim 3 is objected to because of the following informalities: “wherein generate” should read “wherein generating”. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Liu (WO 2021067126 A1), and further in view of Weber (US 12277193 B1).
Regarding claim 1, Liu teaches a computer-implemented method comprising:
receiving input text (par. 0063: “In at least one embodiment, a class-conditional image synthesis model can be used when input data are single class labels. Text-to-image models can be used when input data are text.);
generating, by using a text-to-image model and conditioned on both the input text and image and text pairs selected from a multi-modal knowledge base, an output image (par. 0063: “Conditional image synthesis models can be trained with or without input-output training pairs.”), wherein the generating comprises, at each of multiple time steps:
processing (i) an intermediate representation of the output image for the time step (par. 0114: “An activation function of an intermediate layer of nodes is referred to herein as an inner product kernel. These functions can include, for example, identity functions, step functions, sigmoidal functions, ramp functions, and so on. Activation functions can also be linear or non-linear.”) and (ii) the input text using an encoder of the text-to-image model to generate a first feature map for the time step (par. 0069: “In at least one embodiment, encoder 406 and generator 410 form a variational auto-encoder in which an encoder network attempts to capture a style of an image, while a generator combines an encoded style and a segmentation map information via SPADE to reconstruct an original image.”);
selecting one or more neighbor image and text pairs from the multi-modal knowledge base based on their similarities to the input text (par. 0071: “In at least one embodiment, image encoder 406 can consist of a series of convolutional layers followed by two linear layers that output a mean vector μ and a variance vector σ of output distribution. In at least one embodiment, architecture of generator 410 can consist of a series of SPADE residual blocks with nearest neighbor up-sampling.”);
for each of the one or more neighbor images and text pairs, processing (i) the image in the neighbor image and text pair and (ii) the text in the neighbor image and text pair using the encoder of the text-to-image model to generate a second feature map for the neighbor image and text pair (par. 0071: “In at least one embodiment, image encoder 406 can consist of a series of convolutional layers followed by two linear layers that output a mean vector μ and a variance vector σ of output distribution.”); and
applying an attention mechanism over the one or more second feature maps using one or more queries derived from the first feature map for the time step to generate an attended feature map (par. 0069: “In at least one embodiment, encoder 406 and generator 410 form a variational auto-encoder in which an encoder network attempts to capture a style of an image, while a generator combines an encoded style and a segmentation map information via SPADE to reconstruct an original image.”).
Liu fails to teach generating an updated intermediate representation of the output image for the time step based on using a noise term to de-noise the intermediate representation of the output image, comprising processing the attended feature map for the time step using a decoder of the text-to-image model to generate the noise term.
Weber teaches generating an updated intermediate representation of the output image for the time step based on using a noise term to de-noise the intermediate representation of the output image, comprising processing the attended feature map for the time step using a decoder of the text-to-image model to generate the noise term (col. 32, lines 26-40: “One approach to using a boundary map such as boundary map 2500 to reduce noise in an image such as image 24B is shown in FIG. 26 and is an embodiment of process 2300. In various embodiments, process 2600 is performed by denoiser 334. One example way to implement denoiser 334 is as a set of Python scripts incorporating the open-source libraries NumPy and SciPy. The process begins at 2602 when an image to be denoised (e.g., image 2450) and a boundary map (e.g., boundary map 2500) are received. At 2604, for each field in the boundary map, a regularized value (or set of values, where data for multiple spectral bands is present at each pixel) is determined using pixels in the received image that are located within the field. An image comprising the constrained values is generated as output at 2606.”).
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to include a denoising step in the process of generating an intermediate representation of an output image, as this is a well-known process in the art that serves only and directly to further increase the quality of the generated image. Anyone familiar in the art seeking to make use of the method of Liu would obviously think to include a denoising process in the method.
Regarding claim 2, Liu and Weber teach the method of claim 1. Liu further teaches wherein:
the input text specifies a particular object class (par. 0063: “In at least one embodiment, a class-conditional image synthesis model can be used when input data are single class labels.”); and
the output image depicts an object belonging to the particular object class (par. 0063, as above).
Regarding claim 3, Liu and Weber teach the method of claim 1. Weber further teaches wherein generating the first feature map for the time step further comprises processing time step data defining the time step using the encoder of the text-to-image model (col. 56, lines 6-9: “Feeding the input stacks into the change detector described above produces a batch of corresponding change maps 3708 with a change map at each corresponding time step.”).
Regarding claim 4, Liu and Weber teach the method of claim 1. Liu further teaches wherein selecting the one or more neighbor image and text pairs from the multi-modal knowledge base based on their similarities to the input text comprises, for each image and text pair:
determining a corresponding similarity of the image and text pair to the input text based on (i) a text-to-text similarity between the input text and the text in the image and text pair, (ii) a text-to-image similarity between the input text and the image in the image and text pair, or both (i) and (ii) (par. 0112: “Each filter attempts to learn something different from a structure of an instance, such as a sentence structure for textual data. In a convolutional layer, an activation function can be a rectified linear unit and a pooling type set as max pooling. Results can then be concatenated into a single dimensional vector, and a last layer is fully connected onto a two-dimensional output.”).
Regarding claim 7, Liu and Weber teach the method of claim 1. Weber further teaches wherein selecting the one or more neighbor image and text pairs from the multi-modal knowledge base comprises using search space pruning and quantization techniques (col. 15, lines 47-52: “An xz file comprises an array of N samples. The samples are x and y coordinates as 16-bit unsigned integers quantized to 30 UTM pixel indices, followed by b bands of pixel location (DN) values. The array of N samples is compressed using the Lempel-Ziv-Markov chain compression technique.”).
Regarding claim 8, Liu and Weber teach the method of claim 1. Weber further teaches wherein applying the attention mechanism over the one or more second feature maps comprises:
using the one or more second feature maps to generate one or more keys (col. 33, lines 65-67: “Storage service 2822 persists data to backend storage objects. It provides a mechanism to store arbitrary data and later retrieve it using key-value pair semantics.”); and
applying the attention mechanism over the one or more second feature maps generated from the one or more neighbor image and text pairs using the one or more queries and the one or more keys (col. 34, lines 5-8: “An example interaction with storage service 2822 would be composed of a PUT API call to set a blob (chunk of bytes) to a key, then retrieve that data using a GET API call referencing the same key.”).
Regarding claim 9, Liu and Weber teach the method of claim 1. Liu further teaches wherein the text-to-image image is a text-to-image diffusion model and each time step corresponds to a reverse diffusion time step (par. 0063, as above in claim 1 rejection).
Regarding claim 10, Liu and Weber teach the method of claim 1. Weber further teaches wherein the text-to-image diffusion model comprises a cascade of a low resolution diffusion model and a high resolution diffusion model, the high resolution diffusion model configured to generate a high resolution image as the output image conditioned on a low resolution image generated by the lower resolution diffusion model (col. 31, lines 43-55: “FIG. 22B depicts a boundary map (created in accordance with process 1100) for a portion of land, rasterized to 30 m resolution. FIG. 22A depicts a corresponding MODIS observation of the same portion of land that is represented by boundary map 2230. The MODIS observation (240 m resolution) has been upsampled so that both image 2200 and boundary map 2230 are the same size (e.g., with 2048×2048 pixels representing 30 m×30 m of ground in both cases). FIG. 22C illustrates a result of providing image 2200 and boundary map 2230 as input to process 2300. In particular, regularizing the pixel brightness values across the pixels bounded by a given field (e.g., using the median value) has resulted in a resolution-enhanced version of image 2200.”).
Regarding claim 11, Liu and Weber teach the method of claim 9. Liu further teaches wherein generating the output image comprises using a classifier-free guidance (par. 0058: “In at least one embodiment, an image synthesis network can then use a semantic layout as a guide for generating a final image.”).
Regarding claim 12, Liu and Weber teach the method of claim 9. Weber further teaches wherein generating the output image comprises using an interleaved guidance schedule of text-enhanced noise predictions and neighbor-enhanced noise predictions (col. 48, lines 39-49: “In a middle stage, what is input is the version of the image that was generated by the previous stage, along with noise. The task of the generator model is then to use those inputs to “fix” the details in the input image, where in the case of the input image being the background image plus inserted patches, the “fixing” includes making the combined image look real. That is, given an input image at a certain level of coarseness, the stage of the generator will include details to generate a finer version of the input image (e.g., to generate what the model predicts, according to its training, to be a higher resolution version of the input image).”).
Regarding claim 13, Liu and Weber teach the method of claim 9. Liu further teaches training the text-to-image diffusion model on an image and text dataset to determine trained values of parameters of the text-to-image diffusion model based on optimizing a time re-weighted square error loss (par. 0068: “In at least one embodiment, a generator can be trained with a same multi-scale discriminator and loss function used in pix2pixHD, for example, except that a least squared loss term can be replaced with a hinge loss term.”).
Regarding claim 14, Liu and Weber teach the method of claim 13. Weber further teaches wherein the training comprises training the text-to-image diffusion model to make unconditional noise predictions by randomly dropping out the input text (col. 48, lines 39-49: “In a middle stage, what is input is the version of the image that was generated by the previous stage, along with noise. The task of the generator model is then to use those inputs to “fix” the details in the input image, where in the case of the input image being the background image plus inserted patches, the “fixing” includes making the combined image look real. That is, given an input image at a certain level of coarseness, the stage of the generator will include details to generate a finer version of the input image (e.g., to generate what the model predicts, according to its training, to be a higher resolution version of the input image).”).
Regarding claim 15, Liu and Weber teach the method of claim 1. Liu further teaches a system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of the respective method of claim 1 (par. 0141: “FIG. 16 is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SOC) or some combination thereof 1600 formed with a processor that may include execution units to execute an instruction, according to at least one embodiment.”).
Regarding claim 16, Liu and Weber teach the method of claim 1. Liu further teaches a computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method of claim 1 (par. 0141, as above in claim 15 rejection).
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Liu (WO 2021067126 A1) and Weber (US 12277193 B1) as applied to claim 4 above, and further in view of Boue (US 20240020283 A1).
Regarding claim 5, Liu and Weber teach the method of claim 4, but fail to teach wherein the text-to-text similarity comprises a BM25 similarity.
Boue teaches a text-to-text similarity comprising a BM25 similarity (par. 0048: “The similarity score may be generated in various ways, including but not limited to, using a cosine similarity, Okapi BM25, Jaccard similarity index, MinHash, or other similar techniques as appreciated by those skilled in the art to quantitatively measure the similarity between two multi-dimensional representations (e.g., vectors).”).
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to make use of a BM25 similarity in comparing a text pair, as BM25 is well-known in the art as an effective means of generating a similarity score when comparing text. Boue is in the same field of endeavor of machine learning as Liu and Weber, making it easy for one familiar in the art to have been aware of the existence and utility of each.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Liu (WO 2021067126 A1) and Weber (US 12277193 B1) as applied to claim 4 above, and further in view of Smith (US 20230316475 A1).
Regarding claim 6, Liu and Weber teach the method of claim 4, but fail to teach wherein the text-to-image similarity comprises a CLIP similarity.
Smith teaches a text-to-image similarity comprising a CLIP similarity (par. 0021: “In certain examples, an input text CLIP code is compared against a set of CLIP codes in the latent code library to select a CLIP code that has a greatest similarity to the input text CLIP code. In certain examples, the image modification system determines a CLIP loss based on the input text CLIP code and a blended image CLIP code determined from a blended image output by the image modification model.”).
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to make use of a CLIP similarity in comparing text to an image, as CLIP is well-known in the art as an effective means of determining similarity between an image code and a text code. Smith is in the same field of endeavor of machine learning as Liu and Weber, making it easy for one familiar in the art to have been aware of the existence and utility of each.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN A BARHAM whose telephone number is (571)272-4338. The examiner can normally be reached Mon-Fri, 8:30am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN ALLEN BARHAM/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights