DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is in response to an action filed on December 18th, 2023. Claims 1-20 are pending in the current application.
Information Disclosure Statement
The information disclosure statements filed on September 24th , 2024 and March 27th, 2025 have been considered.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1 Under Step 1 of the Subject Matter Eligibility Test of Products and
Processes, the claim is directed towards a machine which is one of the four statutory categories.
Next, under a Step 2A Prong 1 Analysis, the claim recites the following limitations, which are interpreted to be, under the broadest reasonable interpretation, abstract ideas.
generate a first intermediate tensor based on processing the reference latent tensor and the first latent tensor (mental process)
generate a second latent tensor… based on the first latent tensor and at least in part on the first intermediate tensor. (mental process)
Therefore, we have to examine the claim under Step 2A prong 2, which considers the additional elements within the claim. The claim’s additional elements are:
one or more memories comprising processor-executable instructions;
and one or more processors configured to execute the processor-executable instructions
access a reference latent tensor generated based on a reference input to a diffusion machine learning model;
access a first latent tensor generated during a first iteration of processing data
using a denoising backbone of the diffusion machine learning model
using an auxiliary machine learning model
during a second iteration of processing data using the denoising backbone
The limitations, “one or more memories comprising processor-executable instructions”, “one or more processors configured to execute the processor-executable instructions”, “using a denoising backbone of the diffusion machine learning model”, “using an auxiliary machine learning model”, and “during a second iteration of processing data using the denoising backbone” are interpreted to be, under the broadest reasonable interpretation, mere instructions to apply a judicial exception, as it instructs to use one or more memories and processors as tools to perform the abstract idea, and instructs on how and when to use the denoising backbone, diffusion machine learning model, and auxiliary machine learning model. (See MPEP 2106.05(f)) To “access a reference latent tensor generated based on a reference input to a diffusion machine learning model”, and to “access a first latent tensor generated during a first iteration of processing data” are limitations that are considered to be insignificant extra-solution activity. (See MPEP 2106.05(g)) Therefore, these additional elements do not integrate the abstract idea into a practical application. The claim is directed to an abstract idea.
Under a Step 2B analysis, the claim’s additional elements do not amount to significantly
more than the judicial exception as explained above in Step 2A prong 2. Additionally, to “access a reference latent tensor generated based on a reference input to a diffusion machine learning model”, and to “access a first latent tensor generated during a first iteration of processing data” are limitations that are considered to be well-understood, routine, and conventional, as it is considered to be data gathering. (See MPEP 2106.05(d)(ii)) Therefore, the claim is ineligible.
Regarding claim 11 Under Step 1 of the Subject Matter Eligibility Test of Products and Processes, the claim is directed towards a machine which is one of the four statutory categories.
Next, under a Step 2A Prong 1 Analysis, the claim recites the following limitations, which are interpreted to be, under the broadest reasonable interpretation, abstract ideas.
Generating a first intermediate tensor based on processing the reference latent tensor and the first latent tensor (mental process)
generating a second latent tensor… based on the first latent tensor and at least in part on the first intermediate tensor. (mental process)
Therefore, we have to examine the claim under Step 2A prong 2, which considers the additional elements within the claim. The claim’s additional elements are:
accessing a reference latent tensor generated based on a reference input to a diffusion machine learning model;
accessing a first latent tensor generated during a first iteration of processing data
using a denoising backbone of the diffusion machine learning model
using an auxiliary machine learning model
during a second iteration of processing data using the denoising backbone
The limitations, “using a denoising backbone of the diffusion machine learning model”, “using an auxiliary machine learning model”, and “during a second iteration of processing data using the denoising backbone” are interpreted to be, under the broadest reasonable interpretation, mere instructions to apply a judicial exception, as it instructs on how and when to use the denoising backbone, diffusion machine learning model, and auxiliary machine learning model. (See MPEP 2106.05(f)) To “access a reference latent tensor generated based on a reference input to a diffusion machine learning model”, and to “access a first latent tensor generated during a first iteration of processing data” are limitations that are considered to be insignificant extra-solution activity. (See MPEP 2106.05(g)) Therefore, these additional elements do not integrate the abstract idea into a practical application. The claim is directed to an abstract idea.
Under a Step 2B analysis, the claim’s additional elements do not amount to significantly
more than the judicial exception as explained above in Step 2A prong 2. Additionally, to “access a reference latent tensor generated based on a reference input to a diffusion machine learning model”, and to “access a first latent tensor generated during a first iteration of processing data” are limitations that are considered to be well-understood, routine, and conventional, as it is considered to be data gathering. (See MPEP 2106.05(d)(ii)) Therefore, the claim is ineligible.
Regarding claims 2 and 12, the claims recites generating the first intermediate tensor comprises combining the reference latent tensor and the first latent tensor. The limitation, as drafted, is considered to be, under the broadest reasonable interpretation, a “mental process”, which is a grouping of abstract idea. Therefore, the claims are rejected under the same basis as claims 1 and 11.
Regarding claims 3 and 13, the claims recite combining the reference latent tensor and the first latent tensor comprises at least one of adding, concatenating, or averaging the reference latent tensor and the first latent tensor. The limitation, as drafted, is considered to be, under the broadest reasonable interpretation, a “mental process”, which is a grouping of abstract idea. Therefore, the claims are rejected under the same basis as claims 2 and 12.
Regarding claims 4 and 14, the claims recite the second latent tensor is generated based further on processing a prompt tensor encoding a text input prompt using the auxiliary machine learning model. The limitation, as drafted, is considered to be mere instructions to apply a judicial exception, as it instructs to use the auxiliary model to process a tensor encoding. (See MPEP 2106.05(f)) Therefore, the claims are rejected under the same basis as claims 1 and 11.
Regarding claims 5 and 15, the claims recite generating the second latent tensor comprises providing the first intermediate tensor as input to a first decoder block of the denoising backbone. The limitation, as drafted, is considered to be insignificant extra-solution activity, as well as well-understood, routine, and conventional, as it is considered to be transmitting data over a network. (See MPEP 2106.05(g) and MPEP 2106.05(d)(ii)) Therefore, the claims are rejected under the same basis as claims 1 and 11.
Regarding claims 6 and 16, the claims recite generating a second intermediate tensor based on processing the reference latent tensor and the first latent tensor using the auxiliary machine learning model; and providing the second intermediate tensor as input to a second decoder block of the denoising backbone, wherein the second latent tensor is generated based further on the second intermediate tensor. The limitation, “generating a second intermediate tensor based on processing the reference latent tensor and the first latent tensor” is considered to be, under the broadest reasonable interpretation, a “mental process”, which is a grouping of abstract idea, with the limitation of “using the auxiliary machine learning model” being interpreted to be mere instructed to apply an judicial exception, as it instructs to use the auxiliary machine learning model as a tool to perform the abstract idea. (See MPEP 2106.05(f)) The limitation, “providing the second intermediate tensor as input to a second decoder block of the denoising backbone, wherein the second latent tensor is generated based further on the second intermediate tensor” is considered to be insignificant extra-solution activity, as well as well-understood, routine, and conventional, as it is considered to be transmitting data over a network. (See MPEP 2106.05(g) and MPEP 2106.05(d)(ii)) Therefore, the claims are rejected under the same basis as claims 5 and 15.
Regarding claims 7 and 17, the claims recite the first intermediate tensor is generated by a first encoder block of the auxiliary machine learning model, and generating the second intermediate tensor comprises processing the first intermediate tensor using a second encoder block of the auxiliary machine learning model. The limitation, as drafted, is considered to be mere instructions to apply a judicial exception, as it instructs to use a block of an auxiliary model to process a tensor. (See MPEP 2106.05(f)) Therefore, the claims are rejected under the same basis as claims 6 and 16.
Regarding claims 8 and 18, the claims recite the denoising backbone comprises a sequence of denoiser encoder blocks and a sequence of decoder blocks; the auxiliary machine learning model comprises a sequence of auxiliary encoder blocks; and each decoder block of the sequence of decoder blocks receives input from (i) a corresponding denoiser encoder block of the sequence of denoiser encoder blocks, and (ii) a corresponding encoder block of the sequence of auxiliary encoder blocks. The limitations, “the denoising backbone comprises a sequence of denoiser encoder blocks and a sequence of decoder blocks” and “the auxiliary machine learning model comprises a sequence of auxiliary encoder blocks” are considered to be merely indicating the technological environment or field of use, and “generally links” sequences of encoder blocks to the denoising backbone and auxiliary model. (See MPEP 2106.05(h)) The limitation, “each decoder block of the sequence of decoder blocks receives input from (i) a corresponding denoiser encoder block of the sequence of denoiser encoder blocks, and (ii) a corresponding encoder block of the sequence of auxiliary encoder blocks.” is considered to be insignificant extra-solution activity, as well as well-understood, routine, and conventional, as it is considered to be sending or receiving data over a network. (See MPEP 2106.05(g) and MPEP 2106.05(d)(ii)) Therefore, the claims are rejected under the same basis as claims 1 and 11.
Regarding claims 9 and 19, the claims recite an initial block of the sequence of auxiliary encoder blocks corresponds to a final block of the sequence of decoder blocks; and a final block of the sequence of auxiliary encoder blocks corresponds to an initial block of the sequence of decoder blocks. The limitations, as drafted, are considered to be merely indicating the technological environment or field of use, and “generally links” blocks of a sequence of encoder blocks to blocks of the sequence of decoder blocks. (See MPEP 2106.05(h)) Therefore, the claims are rejected under the same basis as claims 8 and 18.
Regarding claims 10 and 20, the claims recite generating a second intermediate tensor based on processing the reference latent tensor and the second latent tensor using the auxiliary machine learning model; and generating a third latent tensor, during a third iteration of processing data using the denoising backbone, based at least in part on the second intermediate tensor. The limitations, “generating a second intermediate tensor based on processing the reference latent tensor and the second latent tensor” and “generating a third latent tensor, during a third iteration of processing data using the denoising backbone, based at least in part on the second intermediate tensor” are considered to be, under the broadest reasonable interpretation, “mental processes”, which are a grouping of abstract ideas. The limitation, “using the auxiliary machine learning model”, is considered to be mere instructions to apply a judicial exception, as it instructs to use the auxiliary model to process a tensor encoding. (See MPEP 2106.05(f)) Therefore, the claims are rejected under the same basis as claims 1 and 11.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Erik Paul (On Diffusion Models) in view of Jonathan Ho et al. (Herein referred to as Ho) (Cascaded Diffusion Models for High Fidelity Image Generation)
Regarding claim 1, Erik Paul teaches a processing system comprising: one or more memories comprising processor-executable instructions; and one or more processors configured to execute the processor-executable instructions (While the processors and memory are never explicitly mentioned in Erik Paul’s disclosure, one would implicitly require those component to run Erik Paul’s machine learning model.) and cause the processing system to: access a reference latent tensor generated based on a reference input to a diffusion machine learning model; (“For denoising diffusion probabilistic models (DDPMs) [18, 52], we define a Markov chain by gradually adding noise to our training samples until they are (more or less) normally distributed noise and then attempt to model the reversal of this diffusion process. Sampling is achieved by generating random noise and applying the reverse process… The input to the network is a batch of noisy images, i.e., a tensor of shape (B,H,W,3)”, pg. 1, third paragraph; pg. 8, under “6 The U-Net Backbone”) (The input to the network corresponds to a reference latent tensor, with the batch of noisy images corresponding to reference inputs. The denoising diffusion probabilistic model corresponds to a diffusion machine learning model.) access a first latent tensor generated during a first iteration of processing data using a denoising backbone of the diffusion machine learning model; (“A dropout layer [16] randomly sets a fraction p of the weights of its input tensor to 0. Here, p was set to 0.1 for the system trained on the CIFAR10 data set and to 0 on all other data sets (CelebA-HQ 256×256, LSUN) in all dropout layers… The input is a tensor x of shape (B,H,W,C) and the output is of the same shape. Three separate dense layers of output dimension C are used to produce projections of x for use as queries, keys, and values… Sampling from a diffusion model as described in [18] is very time consuming as T = 1000 denoising steps have to be computed.”, pg. 8, under “Dropout”; pg. 10-11; pg. 16, under “Speedups–Strided Sampling, DDIM, and LDM) (The input tensor corresponds to a first latent tensor, which is generated after data processing (via the dropout layer or another data processing layer) using the U-Net Backbone. The backbone is applied to a denoising diffusion model, which means the backbone corresponds to a denoising backbone.) and to generate a first intermediate tensor based on processing the reference latent tensor and the first latent tensor using an auxiliary machine learning model; (“The U-Net consists of a gradual downsampling phase followed by a gradual up sampling phase, where downsampling blocks and upsampling blocks of matching dimensions are also connected by a skip connection, i.e., the output of the down sampling block is concatenated to the (linear) input of the upsampling block. Here, the concatenation of two tensors x,y of shape (B,H,W,C1),(B,H,W,C2) is a tensor z of shape (B,H,W,C1 +C2), where z(b,h,w) is the concatenation of x(b,h,w) and y(b,h,w).”, pg. 12, first and/or second paragraph) (The first intermediate tensor corresponds to the concatenated tensor, which is the concatenation of tensor x and y, performed by the U-Net, which corresponds to an auxiliary machine learning model.)
However, Erik Paul does not explicitly teach to generate a second latent tensor, during a second iteration of processing data using the denoising backbone, based on the first latent tensor and at least in part on the first intermediate tensor.
Ho teaches an iterative refinement method in diffusion model for high fidelity image generation. (“Since the sampling cost increases quadratically with the target image resolution, we attempt to minimize the number of denoising iterations for our 64×64 → 256×256 and 64×64 → 128×128 super-resolution models.”, pg. 21, second paragraph) Combining the iterative refinement of Ho with the diffusion model of Erik Paul would allow for the combination to generate a second latent tensor, during a second iteration of processing data using the denoising backbone, based on the first latent tensor and at least in part on the first intermediate tensor. (“A dropout layer [16] randomly sets a fraction p of the weights of its input tensor to 0. Here, p was set to 0.1 for the system trained on the CIFAR10 data set and to 0 on all other data sets (CelebA-HQ 256×256, LSUN) in all dropout layers… The input is a tensor x of shape (B,H,W,C) and the output is of the same shape. Three separate dense layers of output dimension C are used to produce projections of x for use as queries, keys, and values”, pg. 8, under “Dropout”; pg. 10-11 (Erik Paul)) (After the first iteration, the second iteration would follow similar steps, and use the concatenated tensor (the output from the first iteration) as the input to the diffusion model.)
Therefore, it would have been considered obvious to one of ordinary skill in the art,
prior to the current application’s filing date, to combine the diffusion model of Erik Paul with the iterative method of Ho. One would be motivated to combine the teachings, prior to the filing date of the current application, as this allows for improved sample quality for diffusion models, as described in Ho. (“Our key contribution is the use of cascades to improve the sample quality of diffusion models on class-conditional ImageNet . Here, cascading refers to a simple technique to model high resolution data by learning a pipeline of separately trained models at multiple resolutions; a base model generates low resolution samples, followed by super-resolution models that upsample low resolution samples into high resolution samples.”, pg. 2, second paragraph)
Regarding claim 11, Erik Paul teaches a processor-implemented method, comprising: accessing a reference latent tensor generated based on a reference input to a diffusion machine learning model; (“For denoising diffusion probabilistic models (DDPMs) [18, 52], we define a Markov chain by gradually adding noise to our training samples until they are (more or less) normally distributed noise and then attempt to model the reversal of this diffusion process. Sampling is achieved by generating random noise and applying the reverse process… The input to the network is a batch of noisy images, i.e., a tensor of shape (B,H,W,3)”, pg. 1, third paragraph; pg. 8, under “6 The U-Net Backbone”) (The input to the network corresponds to a reference latent tensor, with the batch of noisy images corresponding to reference inputs. The denoising diffusion probabilistic model corresponds to a diffusion machine learning model.) accessing a first latent tensor generated during a first iteration of processing data using a denoising backbone of the diffusion machine learning model; (“A dropout layer [16] randomly sets a fraction p of the weights of its input tensor to 0. Here, p was set to 0.1 for the system trained on the CIFAR10 data set and to 0 on all other data sets (CelebA-HQ 256×256, LSUN) in all dropout layers… The input is a tensor x of shape (B,H,W,C) and the output is of the same shape. Three separate dense layers of output dimension C are used to produce projections of x for use as queries, keys, and values”, pg. 8, under “Dropout”; pg. 10-11) (The input tensor corresponds to a first latent tensor, which is generated after data processing (via the dropout layer or another data processing layer) using the U-Net Backbone.) and generating a first intermediate tensor based on processing the reference latent tensor and the first latent tensor using an auxiliary machine learning model; (“The U-Net consists of a gradual downsampling phase followed by a gradual up sampling phase, where downsampling blocks and upsampling blocks of matching dimensions are also connected by a skip connection, i.e., the output of the down sampling block is concatenated to the (linear) input of the upsampling block. Here, the concatenation of two tensors x,y of shape (B,H,W,C1),(B,H,W,C2) is a tensor z of shape (B,H,W,C1 +C2), where z(b,h,w) is the concatenation of x(b,h,w) and y(b,h,w).”, pg. 12, first and/or second paragraph) (The first intermediate tensor corresponds to the concatenated tensor, which is the concatenation of tensor x and y, performed by the U-Net, which corresponds to an auxiliary machine learning model.)
However, Erik Paul does not explicitly teach generating a second latent tensor, during a second iteration of processing data using the denoising backbone, based on the first latent tensor and at least in part on the first intermediate tensor.
Ho teaches an iterative refinement method in diffusion model for high fidelity image generation. (“Since the sampling cost increases quadratically with the target image resolution, we attempt to minimize the number of denoising iterations for our 64×64 → 256×256 and 64×64 → 128×128 super-resolution models.”, pg. 21, second paragraph) Combining the iterative refinement of Ho with the diffusion model of Erik Paul would allow for the combination to then teach the step of generating a second latent tensor, during a second iteration of processing data using the denoising backbone, based on the first latent tensor and at least in part on the first intermediate tensor. (“A dropout layer [16] randomly sets a fraction p of the weights of its input tensor to 0. Here, p was set to 0.1 for the system trained on the CIFAR10 data set and to 0 on all other data sets (CelebA-HQ 256×256, LSUN) in all dropout layers… The input is a tensor x of shape (B,H,W,C) and the output is of the same shape. Three separate dense layers of output dimension C are used to produce projections of x for use as queries, keys, and values”, pg. 8, under “Dropout”; pg. 10-11 (Erik Paul)) (After the first iteration, the second iteration would follow similar steps, and use the concatenated tensor (the output from the first iteration) as the input to the diffusion model.)
Therefore, it would have been considered obvious to one of ordinary skill in the art,
prior to the current application’s filing date, to combine the diffusion model of Erik Paul with the iterative method of Ho. One would be motivated to combine the teachings, prior to the filing date of the current application, as this allows for improved sample quality for diffusion models, as described in Ho. (“Our key contribution is the use of cascades to improve the sample quality of diffusion models on class-conditional ImageNet. Here, cascading refers to a simple technique to model high resolution data by learning a pipeline of separately trained models at multiple resolutions; a base model generates low resolution samples, followed by super-resolution models that upsample low resolution samples into high resolution samples.”, pg. 2, second paragraph)
Regarding claim 2, Erik Paul, as modified by Ho, teaches the processing system of claim 1, wherein, to generate the first intermediate tensor, the one or more processors are configured to execute the processor-executable instructions and cause the processing system to combine the reference latent tensor and the first latent tensor. (“The U-Net consists of a gradual downsampling phase followed by a gradual up sampling phase, where downsampling blocks and upsampling blocks of matching dimensions are also connected by a skip connection, i.e., the output of the down sampling block is concatenated to the (linear) input of the upsampling block. Here, the concatenation of two tensors x,y of shape (B,H,W,C1),(B,H,W,C2) is a tensor z of shape (B,H,W,C1 +C2), where z(b,h,w) is the concatenation of x(b,h,w) and y(b,h,w).”, pg. 12, first and/or second paragraph (Erik Paul))
Regarding claim 3, Erik Paul as modified by Ho, teaches the processing system of claim 2, wherein, to combine the reference latent tensor and the first latent tensor, the one or more processors are configured to execute the processor-executable instructions and cause the processing system to at least one of add, concatenate, or average the reference latent tensor and the first latent tensor. (“The U-Net consists of a gradual downsampling phase followed by a gradual up sampling phase, where downsampling blocks and upsampling blocks of matching dimensions are also connected by a skip connection, i.e., the output of the down sampling block is concatenated to the (linear) input of the upsampling block. Here, the concatenation of two tensors x,y of shape (B,H,W,C1),(B,H,W,C2) is a tensor z of shape (B,H,W,C1 +C2), where z(b,h,w) is the concatenation of x(b,h,w) and y(b,h,w).”, pg. 12, first and/or second paragraph (Erik Paul))
Regarding claim 4, Erik Paul, as modified by Ho, teaches the processing system of claim 1, wherein, to generate the second latent tensor, the one or more processors are configured to further execute the processor-executable instructions and cause the processing system to process a prompt tensor encoding a text input prompt using the auxiliary machine learning model. (“it is desirable to be able to model conditional data distributions p∗(x|y) conditioned on some label y such that the model output can be controlled via y. Such a label could be a text encoding for text-to-image synthesis as (Imagen [47], DALLE-2 [41], Stable Diffusion [44]), a partially drawn image to be inpainted [44], or a low resolution image to perform super-resolution on [19, 44]. In the following, we review approaches to enhance the models discussed so far with conditioning mechanisms.”, pg. 20, under “Conditioned Generation” (Eric Paul)) (Using text encodings, the U-Net model of Erik Paul, as modified by Ho, processes the tensor.)
Regarding claim 5, Erik Paul, as modified by Ho, teaches the processing system of claim 1, wherein, to generate the second latent tensor, the one or more processors are configured to execute the processor-executable instructions and cause the processing system to provide the first intermediate tensor as input to a first decoder block of the denoising backbone. (“The decoder D first quantizes the input if it is implemented as a VQ-VAE. A 3×3 convolution restores the shape the encoder produced before forcing the channel number c or 2c. After this, the decoder follows the U-Net layout of Section 6, repeating the ResNet-Attn-ResNet sequence into the upsampling blocks but without any skip connections from the encoder.”, pg. 19, bottom paragraph (Erik Paul)) (Given the configuration, a decoder is implemented with any block that utilizes the decoder corresponding to a decoder block, and tensors are input to the decoder block.)
Regarding claim 6, Erik Paul, as modified by Ho, teaches the processing system of claim 5, wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system to: generate a second intermediate tensor based on processing the reference latent tensor and the first latent tensor using the auxiliary machine learning model; (“The U-Net consists of a gradual downsampling phase followed by a gradual up sampling phase, where downsampling blocks and upsampling blocks of matching dimensions are also connected by a skip connection, i.e., the output of the down sampling block is concatenated to the (linear) input of the upsampling block. Here, the concatenation of two tensors x,y of shape (B,H,W,C1),(B,H,W,C2) is a tensor z of shape (B,H,W,C1 +C2), where z(b,h,w) is the concatenation of x(b,h,w) and y(b,h,w).”, pg. 12, first and/or second paragraph (Erik Paul)) (With the iterative method of Ho, the second intermediate tensor corresponds to the concatenated tensor, which is the concatenation of tensor x and y. It would be easy to configure the concatenation to work by combining the reference latent tensor and the first latent tensor.) and provide the second intermediate tensor as input to a second decoder block of the denoising backbone, wherein the second latent tensor is generated based further on the second intermediate tensor. (“A dropout layer [16] randomly sets a fraction p of the weights of its input tensor to 0. Here, p was set to 0.1 for the system trained on the CIFAR10 data set and to 0 on all other data sets (CelebA-HQ 256×256, LSUN) in all dropout layers… The input is a tensor x of shape (B,H,W,C) and the output is of the same shape. Three separate dense layers of output dimension C are used to produce projections of x for use as queries, keys, and values… The decoder D first quantizes the input if it is implemented as a VQ-VAE. A 3×3 convolution restores the shape the encoder produced before forcing the channel number c or 2c. After this, the decoder follows the U-Net layout of Section 6, repeating the ResNet-Attn-ResNet sequence into the upsampling blocks but without any skip connections from the encoder.”, pg. 8, under “Dropout”; pg. 10-11; pg. 19, bottom paragraph (Erik Paul)) (A decoder is implemented, with any block that utilizes the decoder corresponding to a decoder block, and tensors are input to the decoder block. During a second iteration, the second intermediate tensor would correspond to the input of a second decoder block, which outputs a second latent tensor after data processing (via the dropout layer or another data processing layer).)
Regarding claim 7, Erik Paul, as modified by Ho, teaches the processing system of claim 6, wherein: the first intermediate tensor is generated by a first encoder block of the auxiliary machine learning model, and to generate the second intermediate tensor, the one or more processors are configured to execute the processor-executable instructions and cause the processing system to process the first intermediate tensor using a second encoder block of the auxiliary machine learning model. (“Positional Encoding - The network uses temporal embeddings very similar to those of the original transformer paper [59]”, pg. 8, under “Positional Encoding” (Erik Paul)) (Any block that utilizes positional encoding is interpreted to be an encoder block. During the second iteration, the intermediate tensor is processed via the encoder blocks as part of the U-Net, corresponding to the auxiliary model.)
Regarding claim 8, Erik Paul, as modified by Ho, teaches the processing system of claim 1, wherein: the denoising backbone comprises a sequence of denoiser encoder blocks and a sequence of decoder blocks the auxiliary machine learning model comprises a sequence of auxiliary encoder blocks and each decoder block of the sequence of decoder blocks receives input from (i) a corresponding denoiser encoder block of the sequence of denoiser encoder blocks, and (ii) a corresponding encoder block of the sequence of auxiliary encoder blocks. (The decoder D first quantizes the input if it is implemented as a VQ-VAE. A 3×3 convolution restores the shape the encoder produced before forcing the channel number c or 2c. After this, the decoder follows the U-Net layout of Section 6, repeating the ResNet-Attn-ResNet sequence into the upsampling blocks but without any skip connections from the encoder.”, pg. 19, bottom paragraph; See also Figures on pg. 12 (Erik Paul)) (The VAE has a sequence of encoder and decoder blocks as shown by the figures on pg. 12.)
Regarding claim 9, Erik Paul, as modified by Ho, teaches the processing system of claim 8, wherein: an initial block of the sequence of auxiliary encoder blocks corresponds to a final block of the sequence of decoder blocks; and a final block of the sequence of auxiliary encoder blocks corresponds to an initial block of the sequence of decoder blocks (“The encoder is constructed exactly like the downsampling part of the U-Net in Section 6 including the ResNet-Attn-ResNet sequence after the downsampling)”, pg. 19, third paragraph; See also the figures on pg. 12) (Based on the down sampling and upsampling, there is an initial block and final block for encoders and decoders.)
Regarding claim 10, Erik Paul, as modified by Ho, teaches the processing system of claim 1, wherein the one or more processors are configured to further execute the processor-executable instructions and cause the processing system to: generate a second intermediate tensor based on processing the reference latent tensor and the second latent tensor using the auxiliary machine learning model; (“The U-Net consists of a gradual downsampling phase followed by a gradual up sampling phase, where downsampling blocks and upsampling blocks of matching dimensions are also connected by a skip connection, i.e., the output of the down sampling block is concatenated to the (linear) input of the upsampling block. Here, the concatenation of two tensors x,y of shape (B,H,W,C1),(B,H,W,C2) is a tensor z of shape (B,H,W,C1 +C2), where z(b,h,w) is the concatenation of x(b,h,w) and y(b,h,w).”, pg. 12, first and/or second paragraph) and generate a third latent tensor, during a third iteration of processing data using the denoising backbone, based at least in part on the second intermediate tensor. (“A dropout layer [16] randomly sets a fraction p of the weights of its input tensor to 0. Here, p was set to 0.1 for the system trained on the CIFAR10 data set and to 0 on all other data sets (CelebA-HQ 256×256, LSUN) in all dropout layers… The input is a tensor x of shape (B,H,W,C) and the output is of the same shape. Three separate dense layers of output dimension C are used to produce projections of x for use as queries, keys, and values”, pg. 8, under “Dropout”; pg. 10-11) (After the second iteration, the third iteration would follow similar steps, and use the concatenated tensor (the output from the second iteration) as the input to the diffusion model.)
Regarding claim 12, Erik Paul, as modified by Ho, teaches the processor-implemented method of claim 11 wherein generating the first intermediate tensor comprises combining the reference latent tensor and the first latent tensor. (“The U-Net consists of a gradual downsampling phase followed by a gradual up sampling phase, where downsampling blocks and upsampling blocks of matching dimensions are also connected by a skip connection, i.e., the output of the down sampling block is concatenated to the (linear) input of the upsampling block. Here, the concatenation of two tensors x,y of shape (B,H,W,C1),(B,H,W,C2) is a tensor z of shape (B,H,W,C1 +C2), where z(b,h,w) is the concatenation of x(b,h,w) and y(b,h,w).”, pg. 12, first and/or second paragraph (Erik Paul))
Regarding claim 13, Erik Paul as modified by Ho, teaches the processor-implemented method of claim 12, wherein combining the reference latent tensor and the first latent tensor comprises at least one of adding, concatenating, or averaging the reference latent tensor and the first latent tensor. (“The U-Net consists of a gradual downsampling phase followed by a gradual up sampling phase, where downsampling blocks and upsampling blocks of matching dimensions are also connected by a skip connection, i.e., the output of the down sampling block is concatenated to the (linear) input of the upsampling block. Here, the concatenation of two tensors x,y of shape (B,H,W,C1),(B,H,W,C2) is a tensor z of shape (B,H,W,C1 +C2), where z(b,h,w) is the concatenation of x(b,h,w) and y(b,h,w).”, pg. 12, first and/or second paragraph (Erik Paul))
Regarding claim 14, Erik Paul, as modified by Ho, teaches he processor-implemented method of claim 11, wherein the second latent tensor is generated based further on processing a prompt tensor encoding a text input prompt using the auxiliary machine learning model. (“it is desirable to be able to model conditional data distributions p∗(x|y) conditioned on some label y such that the model output can be controlled via y. Such a label could be a text encoding for text-to-image synthesis as (Imagen [47], DALLE-2 [41], Stable Diffusion [44]), a partially drawn image to be inpainted [44], or a low resolution image to perform super-resolution on [19, 44]. In the following, we review approaches to enhance the models discussed so far with conditioning mechanisms.”, pg. 20, under “Conditioned Generation” (Erik Paul)) (Using text encodings, the U-Net model of Erik Paul, as modified by Ho, processes the tensor.)
Regarding claim 15, Erik Paul, as modified by Ho, teaches the processor-implemented method of claim 11, wherein generating the second latent tensor comprises providing the first intermediate tensor as input to a first decoder block of the denoising backbone. (“The decoder D first quantizes the input if it is implemented as a VQ-VAE. A 3×3 convolution restores the shape the encoder produced before forcing the channel number c or 2c. After this, the decoder follows the U-Net layout of Section 6, repeating the ResNet-Attn-ResNet sequence into the upsampling blocks but without any skip connections from the encoder.”, pg. 19, bottom paragraph (Erik Paul)) (Given the configuration, a decoder is implemented with any block that utilizes the decoder corresponding to a decoder block, and tensors are input to the decoder block.)
Regarding claim 16, Erik Paul, as modified by Ho, teaches the processor-implemented method of claim 15, further comprising: generating a second intermediate tensor based on processing the reference latent tensor and the first latent tensor using the auxiliary machine learning model; (“The U-Net consists of a gradual downsampling phase followed by a gradual up sampling phase, where downsampling blocks and upsampling blocks of matching dimensions are also connected by a skip connection, i.e., the output of the down sampling block is concatenated to the (linear) input of the upsampling block. Here, the concatenation of two tensors x,y of shape (B,H,W,C1),(B,H,W,C2) is a tensor z of shape (B,H,W,C1 +C2), where z(b,h,w) is the concatenation of x(b,h,w) and y(b,h,w).”, pg. 12, first and/or second paragraph (Erik Paul)) (With the iterative method of Ho, the second intermediate tensor corresponds to the concatenated tensor, which is the concatenation of tensor x and y. It would be easy to configure the concatenation to work by combining the reference latent tensor and the first latent tensor.) and providing the second intermediate tensor as input to a second decoder block of the denoising backbone, wherein the second latent tensor is generated based further on the second intermediate tensor. (“A dropout layer [16] randomly sets a fraction p of the weights of its input tensor to 0. Here, p was set to 0.1 for the system trained on the CIFAR10 data set and to 0 on all other data sets (CelebA-HQ 256×256, LSUN) in all dropout layers… The input is a tensor x of shape (B,H,W,C) and the output is of the same shape. Three separate dense layers of output dimension C are used to produce projections of x for use as queries, keys, and values… The decoder D first quantizes the input if it is implemented as a VQ-VAE. A 3×3 convolution restores the shape the encoder produced before forcing the channel number c or 2c. After this, the decoder follows the U-Net layout of Section 6, repeating the ResNet-Attn-ResNet sequence into the upsampling blocks but without any skip connections from the encoder.”, pg. 8, under “Dropout”; pg. 10-11; pg. 19, bottom paragraph (Erik Paul)) (A decoder is implemented, with any block that utilizes the decoder corresponding to a decoder block, and tensors are input to the decoder block. During a second iteration, the second intermediate tensor would correspond to the input of a second decoder block, which outputs a second latent tensor after data processing (via the dropout layer or another data processing layer).)
Regarding claim 17, Erik Paul, as modified by Ho, teaches the processor-implemented method of claim 16, wherein: the first intermediate tensor is generated by a first encoder block of the auxiliary machine learning model, and generating the second intermediate tensor comprises processing the first intermediate tensor using a second encoder block of the auxiliary machine learning model. (“Positional Encoding - The network uses temporal embeddings very similar to those of the original transformer paper [59]”, pg. 8, under “Positional Encoding”) (Any block that utilizes positional encoding is interpreted to be an encoder block. During the second iteration, the intermediate tensor is processed via the encoder blocks as part of the U-Net, corresponding to the auxiliary model.)
Regarding claim 18, Erik Paul, as modified by Ho, teaches the processor-implemented method of claim 11, wherein: the denoising backbone comprises a sequence of denoiser encoder blocks and a sequence of decoder blocks; the auxiliary machine learning model comprises a sequence of auxiliary encoder blocks; and each decoder block of the sequence of decoder blocks receives input from (i) a corresponding denoiser encoder block of the sequence of denoiser encoder blocks, and (ii) a corresponding encoder block of the sequence of auxiliary encoder blocks. (The decoder D first quantizes the input if it is implemented as a VQ-VAE. A 3×3 convolution restores the shape the encoder produced before forcing the channel number c or 2c. After this, the decoder follows the U-Net layout of Section 6, repeating the ResNet-Attn-ResNet sequence into the upsampling blocks but without any skip connections from the encoder.”, pg. 19, bottom paragraph; See also Figures on pg. 12 (Erik Paul)) (The VAE has a sequence of encoder and decoder blocks as shown by the figures on pg. 12.)
Regarding claim 19, Erik Paul, as modified by Ho, teaches the processor-implemented method of claim 18, wherein: an initial block of the sequence of auxiliary encoder blocks corresponds to a final block of the sequence of decoder blocks; and a final block of the sequence of auxiliary encoder blocks corresponds to an initial block of the sequence of decoder blocks. (“The encoder is constructed exactly like the downsampling part of the U-Net in Section 6 including the ResNet-Attn-ResNet sequence after the downsampling)”, pg. 19, third paragraph; See also the figures on pg. 12) (Based on the down sampling and upsampling, there is an initial block and final block for encoders and decoders.)
Regarding claim 20, Erik Paul, as modified by Ho, teaches the processor-implemented method of claim 11, further comprising: generating a second intermediate tensor based on processing the reference latent tensor and the second latent tensor using the auxiliary machine learning model;(“The U-Net consists of a gradual downsampling phase followed by a gradual up sampling phase, where downsampling blocks and upsampling blocks of matching dimensions are also connected by a skip connection, i.e., the output of the down sampling block is concatenated to the (linear) input of the upsampling block. Here, the concatenation of two tensors x,y of shape (B,H,W,C1),(B,H,W,C2) is a tensor z of shape (B,H,W,C1 +C2), where z(b,h,w) is the concatenation of x(b,h,w) and y(b,h,w).”, pg. 12, first and/or second paragraph) and generating a third latent tensor, during a third iteration of processing data using the denoising backbone, based at least in part on the second intermediate tensor. (“A dropout layer [16] randomly sets a fraction p of the weights of its input tensor to 0. Here, p was set to 0.1 for the system trained on the CIFAR10 data set and to 0 on all other data sets (CelebA-HQ 256×256, LSUN) in all dropout layers… The input is a tensor x of shape (B,H,W,C) and the output is of the same shape. Three separate dense layers of output dimension C are used to produce projections of x for use as queries, keys, and values”, pg. 8, under “Dropout”; pg. 10-11) (After the second iteration, the third iteration would follow similar steps, and use the concatenated tensor (the output from the second iteration) as the input to the diffusion model.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Tyler E Iles whose telephone number is (571)272-5442. The examiner can normally be reached 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.E.I./ Patent Examiner, Art Unit 2122
/KAKALI CHAKI/ Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights