DETAILED ACTION
This office action is responsive to applicant’s communication filed 03/24/2026.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see pg. 9, filed 03/24/2026, with respect to the rejection of claims 6-8, 10, and 15 under 35 U.S.C. 112(b) have been fully considered and are persuasive. The rejection of claims 6-8, 10, and 15 under 35 U.S.C. 112(b) has been withdrawn.
Applicant’s arguments with respect to the rejections of claim(s) 1, 12, and 15 under 35 U.S.C. 103 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Objections
Claim 15 objected to because of the following informalities: “denoise intermediate noise map...” is grammatically incorrect; it is interpreted as “denoise an intermediate noise map…”.
Appropriate correction is required.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
The following terms in the claims have been given the following interpretations in light of the specification:
Claims 1 and 15: “Query token” and “key token”: [0055] “In the case of self-attention, Query tokens are the same as Key tokens.”; [0057] “For cross-attention, Query tokens are image tokens and Key tokens are text prompt tokens.”
Claim 1 does not differentiate between self-attention and cross-attention. Therefore, for the purposes of claim 1, either definition of query tokens and key tokens may apply.
Claim 12: “Denoising-Steps-Aware Pruning (DSAP)” schedule: [0045] "The image processing system 300 may include a token pruning scheme applied within each denoising step of the diffusion process, and an adaptive pruning schedule across different denoising steps, such as a Denoising-Steps-Aware Pruning (DSAP) schedule."
This is the only mention of DSAP in the specification. Therefore, a Denoising-Steps-Aware Pruning schedule is “an adaptive pruning schedule across different denoising steps”.
These definitions are used for purposes of searching for prior art, but cannot be incorporated into the claims. Should applicant wish different definitions, applicant should point to the portions of the specification that clearly show a different definition.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-7, 9-11, 15-16, 18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bolya et al. ("Token Merging for Fast Stable Diffusion". arXiv preprint (30 Mar 2023). https://doi.org/10.48550/arXiv.2303.17604; hereinafter "Bolya") in view of Thorsley et al. (US 20220374766 A1; hereinafter "Thorsley") and Wang et al. ("Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers". arXiv preprint (27 May 2023). https://arxiv.org/abs/2305.17328v1; hereinafter "Wang").
Regarding claim 1, Bolya teaches: A method comprising:
obtaining an input prompt (fig. 2 “Prompt”, pg. 2 “Each transformer block has the standard self attention [22] and multi-layer perception (mlp) modules, with the addition of a cross attention module to condition on the prompt.”);
generating a plurality of tokens for an attention layer of a generative machine learning model based on an intermediate noise map (pg. 2 “Stable Diffusion uses a U-Net [16] with transformer-based blocks. Thus, it first encodes the current noised image as a set of tokens, then passes it through a series of transformer blocks.”; transformer blocks contain attention layers as shown in fig. 2);
denoising, using the generative machine learning model, the intermediate noise map based on the pruned set of tokens to obtain a denoised map (pg. 2 “Diffusion models [4,20,21] generate images by repeatedly denoising some initial noise over some number of diffusion steps.”); and
generating, using the generative machine learning model, a synthetic image based on the denoised map (examples of image generation included in all figures except for fig. 2).
Bolya does not explicitly teach: generating, using the attention layer, an attention map based on the plurality of tokens; or
pruning the plurality of tokens based on the attention map to obtain a pruned set of tokens.
Thorsley teaches: generating, using the attention layer, an attention map based on the plurality of tokens ([0024] “Transformer deep-learning models utilize a self-attention mechanism, differentially weighing the significance of each part of the input data. A self-attention mechanism provides context for any position in the input sequence, allowing for parallelization during training of a deep neural network. Self-attention mechanisms receive input sequences that are converted into tokens. Each token is provided a probability and scored based on a relevant metric (also known as the attention-probability and as the attention-score or importance-score)”;
[0031] to [0037] describes using a self-attention mechanism to calculate an attention matrix (or map) which is later used to relate the attention of each token to each other token, specifically referencing a “scaled dot-product attention layer 131”); and
pruning the plurality of tokens based on the attention map to obtain a pruned set of tokens ([0043] “Each head has the attention probabilities determined using the process described with respect to FIG. 1C with the scaled dot-product attention layers 131… The token importance score s.sup.(l) 340 may be calculated from the mean 330 of the attention probability over all the heads.”, where element 131 references the process of generating an attention matrix/map); [0044] “Alternatively, using a threshold token pruning operation 360, a token may be kept at 362 if the importance score of the token exceeds an absolute threshold value, and may be pruned at 364 otherwise.”) using a function that computes an importance score ([0024] “Self-attention mechanisms receive input sequences that are converted into tokens. Each token is provided a probability and scored based on a relevant metric (also known as the attention-probability and as the attention-score or importance-score)”).
Bolya and Thorsley are both analogous to the claimed invention because they are in the same field of increasing the efficiency of a transformer neural network. Furthermore, Bolya also suggests (but does not explicitly teach) the use of token pruning to speed up neural network processing (pg. 1 col. 2 section 1 “Introduction”), as taught by Thorsley. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Bolya with the teachings of Thorsley to use the token pruning method of Thorsley, rather than the token merging method of Bolya, to reduce computations associated with tokens. The motivation would have been to try an alternate method of increasing the efficiency of the neural network.
The combination of Bolya in view of Thorsley does not explicitly teach: wherein the plurality of tokens are pruned by performing a weighted page rank process using a function that computes an importance score of a query token based on the attention map and an importance score of a key token.
PNG
media_image1.png
683
1676
media_image1.png
Greyscale
Wang teaches: wherein the plurality of tokens are pruned by performing a weighted page rank process (fig. 1 “I-stage” step; pg. 4-5 section 3.2 “I-stage: Importance-based Pruning”: “Inspired by the Page Rank [20] algorithm, we propose a WPR algorithm to derive the importance scores… In the original Page Rank algorithm, links between web pages are unweighted. In order to apply it to the weighted and directed attention graph, we consider the signal of each node in this graph as the importance of each token.”) using a function that computes an importance score of a query token based on the attention map and an importance score of a key token (fig. 1 “I-stage” step; pg. 5 section 3.2 “I-stage: Importance-based Pruning” teaches a method of computing the importance score of each token simultaneously using the attention map and the importance scores of all of the other tokens: “We initialize the graph signal uniformly. Then, we use the adjacency matrix as a graph shift operator (GSO). When the GSO is applied to the graph signal, each node votes for which node is more important through the weight assigned to output edges, i.e., the attention that a token pays to other tokens. Note that each node may possibly have a different impact when voting. If the node itself is more important, the voting of this node is more important.”
Wang teaches a self-attention mechanism: “At the heart of a Transformer layer, a multi-head self-attention mechanism enables each token in the input sequence to attend to every other token” (pg. 2). Therefore, each node (token) may be considered a “query token” and a “key token” at the same time (see “Claim Interpretation” section)).
Wang is analogous to the claimed invention because it teaches a method of optimizing a diffusion model via token pruning. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Bolya in view of Thorsley with the teachings of Wang to prune tokens using a weighted page rank process. The motivation would have been “to reduce noise from unimportant tokens during importance assignment” (Wang pg. 2).
Regarding claim 2, the combination of Bolya in view of Thorsley and Wang teaches: The method of claim 1, wherein: each of the plurality of tokens corresponds to one or more pixels of an image (Bolya pg. 1 “All of these methods function by denoising images through several evaluations of a transformer [22] backbone, meaning that computation scales with the square of the number of tokens (and thus also the square of pixels).”; fig. 5 shows correspondence of token locations to image locations.).
Regarding claim 3, the combination of Bolya in view of Thorsley and Wang teaches: The method of claim 1, wherein generating the attention map comprises: performing a self-attention mechanism on the plurality of tokens (Thorsley [0024] “Transformer deep-learning models utilize a self-attention mechanism, differentially weighing the significance of each part of the input data. A self-attention mechanism provides context for any position in the input sequence, allowing for parallelization during training of a deep neural network. Self-attention mechanisms receive input sequences that are converted into tokens. Each token is provided a probability and scored based on a relevant metric (also known as the attention-probability and as the attention-score or importance-score)”; [0031] to [0037] describes using the self-attention mechanism to calculate an attention matrix (or map) which is later used to relate the attention of each token to each other token).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Bolya in view of Thorsley and Wang with the additional teachings of Thorsley to use a self-attention mechanism on the tokens to generate an attention map. The motivation would have been to include a method of establishing metrics or criteria by which tokens are selected for pruning.
Regarding claim 5, the combination of Bolya in view of Thorsley and Wang teaches: The method of claim 1, wherein pruning the plurality of tokens comprises:
computing the importance score for each of the plurality of tokens based on the attention map (Thorsley [0043] “Each head has the attention probabilities determined using the process described with respect to FIG. 1C with the scaled dot-product attention layers 131… The token importance score s.sup.(l) 340 may be calculated from the mean 330 of the attention probability over all the heads.”, where element 131 references the process of generating an attention matrix/map); and
identifying a threshold importance score (Thorsley [0044] “Alternatively, using a threshold token pruning operation 360, a token may be kept at 362 if the importance score of the token exceeds an absolute threshold value, and may be pruned at 364 otherwise.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Bolya in view of Thorsley and Wang with the additional teachings of Thorsley to compute importance scores and a threshold importance score for token pruning. The motivation would have been to include a method of establishing metrics or criteria by which tokens are selected for pruning.
Regarding claim 6, the combination of Bolya in view of Thorsley and Wang teaches: The method of claim 1, wherein generating the synthetic image comprises:
performing, using a subsequent attention layer of the generative machine learning model, an attention mechanism on the pruned set of tokens (Bolya fig. 2, U-Net block includes attention modules; multiple blocks are connected in sequence: pg. 2 “…making the next block faster”).
Regarding claim 7, the combination of Bolya in view of Thorsley and Wang teaches: The method of claim 1, wherein generating the synthetic image comprises:
identifying a plurality of pruned tokens (Bolya pg. 2 section 3.1 “Defining Unmerging”: “And if we have information about what tokens we merged, we have enough information to then unmerge those same tokens.”; pruned tokens could be identified in the same manner);
generating a plurality of replacement tokens corresponding to the plurality of pruned tokens; and adding the plurality of replacement tokens to the pruned set of tokens to obtain an augmented set of tokens (Bolya pg. 2 section 3.1 “Defining Unmerging” describes duplicating existing merged tokens, which were identified based on similarity, and copying them into previously filled token slots; a similar procedure could be applied to pruned tokens).
Regarding claim 9, the combination of Bolya in view of Thorsley and Wang teaches: The method of claim 7, wherein generating the plurality of replacement tokens comprises: identifying a similarity-based copy for each of the plurality of replacement tokens (Bolya pg. 2 section “Token Merging” describes how tokens are identified for merging based on similarity; pg. 2 section 3.1 “Defining Unmerging” describes how these merged tokens are copied in order to generate replacement tokens).
Regarding claim 10, the combination of Bolya in view of Thorsley and Wang teaches: The method of claim 1, wherein generating the synthetic output comprises: performing a diffusion process on a noise input (Bolya pg. 2 “Diffusion models [4,20,21] generate images by repeatedly denoising some initial noise over some number of diffusion steps.”).
Regarding claim 11, the combination of Bolya in view of Thorsley and Wang teaches: The method of claim 1, further comprising:
identifying a first pruning parameter, wherein the pruning is performed based on the first pruning parameter at a first stage of the generative machine learning model (Thorsley [0004] “The input sequences may include first input sequence with a plurality of tokens. The plurality of tokens may be received into the first sub-model, where each of the plurality of tokens are scored for a first token score. At least one of the plurality of tokens below a first predetermined threshold score may be pruned from the first input sequence to form a second input sequence.”); and
identifying a second pruning parameter, wherein a subsequent pruning is performed based on the second pruning parameter at a second stage of the generative machine learning model (Thorsley [0004] “The second input sequence may be input into the second sub-model, where each token of the second input sequence is scored for second token score. At least one of the tokens below a second predetermined threshold score may be pruned from the second input sequence to form a third input sequence. The second predetermined threshold score may differ from the first predetermined threshold score. The second predetermined threshold score may be equal to or greater than the first predetermined threshold score.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Bolya in view of Thorsley and Wang with the additional teachings of Thorsley to adjust the threshold importance score across multiple stages of token pruning. The motivation would have been to optimize the threshold value to balance accuracy and computation time, as taught by Thorsley ([0026]).
Regarding claim 15, it is rejected with the same rationale, references, and motivation to combine as claim 1 because its limitations substantially correspond to the limitations of claim 1, along with the additional limitation of: An apparatus comprising: at least one processor; at least one memory storing instruction executable by the at least one processor (Bolya section 3 “Experimental Details”: “To test speed, we simply average the time taken over all 2,000 samples on a single 4090 GPU.”; GPUs contain processors connected to memory); and a generative machine learning model comprising parameters stored in the at least one memory (Bolya abstract “In the process, we speed up image generation by up to 2× and reduce memory consumption by up to 5.6×.”, caused by reducing count of tokens being stored in memory).
Regarding claim 16, the combination of Bolya in view of Thorsley and Wang teaches: The apparatus of claim 15, wherein:
the generative machine learning model comprises a diffusion model (Bolya pg. 1 section 2 “Background”: “In this work, our goal is to speed up an off-the-shelf Stable Diffusion [15] model…”).
Regarding claim 18, the combination of Bolya in view of Thorsley and Wang teaches: The apparatus of claim 15, wherein:
the generative machine learning model comprises an attention block comprising the attention layer and a subsequent attention layer that processes the pruned set of tokens (Bolya fig. 2, U-Net transformer blocks include multiple attention layers which process merged (or pruned) tokens).
Regarding claim 20, the combination of Bolya in view of Thorsley and Wang teaches: The apparatus of claim 15, wherein:
the generative machine learning model comprises a pre-trained model that is not fine-tuned prior to generating the synthetic output (Bolya pg. 1 section 2 “Background”: “In this work, our goal is to speed up an off-the-shelf Stable Diffusion [15] model without training using ToMe [1]”).
Claim(s) 4 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bolya ("Token Merging for Fast Stable Diffusion") in view of Thorsley (US 20220374766 A1) and Wang ("Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers") as applied to claims 1 and 15 above, and further in view of Azarian Yazdi et al. (US 20250166236 A1, hereinafter "Azarian Yazdi").
Regarding claim 4, the combination of Bolya in view of Thorsley and Wang teaches: The method of claim 1, wherein generating the attention map comprises:
performing a cross-attention mechanism on the plurality of tokens (Bolya fig. 2, transformer block includes cross-attention module).
The combination of Bolya in view of Thorsley and Wang does not explicitly teach: performing a cross-attention mechanism on the plurality of tokens and a plurality of condition tokens.
Azarian Yazdi teaches performing a cross-attention mechanism on a plurality of condition tokens ([0005] “The method includes… for each of one or more patches of a latent image representation: calculating a respective plurality of cross-attention weights corresponding to the plurality of conditioning tokens based on the patch as a query and the plurality of conditioning tokens as a key; and modifying a maximum value cross-attention weight among the respective plurality of cross-attention weights to generate a modified respective plurality of cross-attention weights”).
Azarian Yazdi is analogous to the claimed invention because it is in the same field of image generation using a diffusion model. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Bolya in view of Thorsley and Wang with the teachings of Azarian Yazdi to represent the text input using condition tokens when performing cross-attention with the image tokens. The motivation would have been to maintain a standardized form of input representation for the neural network.
Regarding claim 17, the combination of Bolya in view of Thorsley and Wang teaches: The apparatus of claim 15, but does not explicitly teach: further comprising:
a text encoder configured to generate a plurality of condition tokens.
Azarian Yazdi teaches: a text encoder configured to generate a plurality of condition tokens ([0005] “The method includes obtaining the text prompt; encoding the text prompt into a plurality of conditioning tokens; …”).
Azarian Yazdi is analogous to the claimed invention because it is in the same field of image generation using a diffusion model. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Bolya in view of Thorsley and Wang with the teachings of Azarian Yazdi to encode the text input into condition tokens. The motivation would have been to maintain a standardized form of input representation for the neural network, along with the image tokens.
Claim(s) 8 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bolya ("Token Merging for Fast Stable Diffusion") in view of Thorsley (US 20220374766 A1) and Wang ("Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers") as applied to claims 1 and 15 above, and further in view of Kahatapitiya et al. (US 20250166133 A1, hereinafter "Kahatapitiya").
Regarding claim 8, the combination of Bolya in view of Thorsley and Wang teaches: The method of claim 7, but does not explicitly teach: wherein generating the synthetic output comprises: performing a convolution based on the augmented set of tokens.
Kahatapitiya teaches: wherein generating the synthetic output comprises: performing a convolution based on the augmented set of tokens ([0120] “When applying ToMe on the triplets, the output representation should be unmerged after the attention operation to preserve the original shape (or, resolution) especially for generative tasks. This is done by Token Unmerging, which simply copies merged tokens to their original locations based on the same set of indices.”;
Fig. 4 shows the diffusion model architecture, [0098] “The U-Net architecture 400 includes a contracting path 404 and an expansive path 406 as shown in FIG. 4, which gives it the U-shaped architecture. The contracting path 404 can be a convolutional network that includes repeated convolutional layers (that apply convolutional operations), each followed by a rectified linear unit (ReLU) and a max pooling operation.”).
Kahatapitiya is analogous to the claimed invention because they are in the same field of image generation using a diffusion model. Furthermore, Kahatapitiya and Bolya use the same token merging and unmerging methodology (ToMe). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Bolya in view of Thorsley and Wang with the teachings of Kahatapitiya to perform a convolution operation with the restored set of tokens; this is a standard neural network operation that is enabled by the token unmerging (or un-pruning) operation taught by Bolya, which restores the complete structure of tokens.
Regarding claim 19, it is rejected with the same rationales, references, and motivations to combine as claims 7 and 8 because its limitations substantially correspond to the limitations of claims 7 and 8.
Claim(s) 12-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bolya et al. ("Token Merging for Fast Stable Diffusion". arXiv preprint (30 Mar 2023). https://doi.org/10.48550/arXiv.2303.17604; hereinafter "Bolya") in view of Thorsley et al. (US 20220374766 A1; hereinafter "Thorsley") and Yang et al. ("Denoising Diffusion Step-aware Models". arXiv preprint (24 May 2024). https://arxiv.org/abs/2310.03337v5; hereiniafter "Yang").
Regarding claim 12, Bolya teaches:
obtain an input prompt (fig. 2 “Prompt”, pg. 2 “Each transformer block has the standard self attention [22] and multi-layer perception (mlp) modules, with the addition of a cross attention module to condition on the prompt.”);
generate a plurality of tokens for an attention layer of the generative machine learning model based on intermediate noise map (pg. 2 “Stable Diffusion uses a U-Net [16] with transformer-based blocks. Thus, it first encodes the current noised image as a set of tokens, then passes it through a series of transformer blocks.”; transformer blocks contain attention layers as shown in fig. 2);
denoise, using the generative machine learning model, the intermediate noise map based on the pruned set of tokens to obtain a denoised map (pg. 2 “Diffusion models [4,20,21] generate images by repeatedly denoising some initial noise over some number of diffusion steps.”); and
generate, using the generative machine learning model, a synthetic output based on the denoised map (examples of image generation included in all figures except for fig. 2).
Bolya does not explicitly teach:
A non-transitory computer readable medium storing code for a generative machine learning model, the code comprising instructions executable by at least one processor to:
generate, using the attention layer, an attention map based on the plurality of tokens;
prune the plurality of tokens based on the attention map to obtain a pruned set of tokens;
Thorsley teaches: A non-transitory computer readable medium storing code for a generative machine learning model, the code comprising instructions executable by at least one processor ([0073] “Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer-program instructions, encoded on computer-storage medium for execution by, or to control the operation of data-processing apparatus… The computer-storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).”) to:
generate, using the attention layer, an attention map based on the plurality of tokens ([0024] “Transformer deep-learning models utilize a self-attention mechanism, differentially weighing the significance of each part of the input data. A self-attention mechanism provides context for any position in the input sequence, allowing for parallelization during training of a deep neural network. Self-attention mechanisms receive input sequences that are converted into tokens. Each token is provided a probability and scored based on a relevant metric (also known as the attention-probability and as the attention-score or importance-score)”;
[0031] to [0037] describes using a self-attention mechanism to calculate an attention matrix (or map) which is later used to relate the attention of each token to each other token, specifically referencing a “scaled dot-product attention layer 131”);
prune the plurality of tokens based on the attention map to obtain a pruned set of tokens ([0043] “Each head has the attention probabilities determined using the process described with respect to FIG. 1C with the scaled dot-product attention layers 131… The token importance score s.sup.(l) 340 may be calculated from the mean 330 of the attention probability over all the heads.”, where element 131 references the process of generating an attention matrix/map); [0044] “Alternatively, using a threshold token pruning operation 360, a token may be kept at 362 if the importance score of the token exceeds an absolute threshold value, and may be pruned at 364 otherwise.”).
Bolya and Thorsley are both analogous to the claimed invention because they are in the same field of increasing the efficiency of a transformer neural network. Furthermore, Bolya also suggests (but does not explicitly teach) the use of token pruning to speed up neural network processing (pg. 1 col. 2 section 1 “Introduction”), as taught by Thorsley. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Bolya with the teachings of Thorsley to use the token pruning method of Thorsley, rather than the token merging method of Bolya, to reduce computations associated with tokens. The motivation would have been to try an alternate method of increasing the efficiency of the neural network.
The combination of Bolya and Thorsley does not explicitly teach: prune the plurality of tokens based on the attention map to obtain a pruned set of tokens by identifying a denoising timestep and pruning the plurality of tokens according to a denoising- steps-aware pruning (DSAP) based on the denoising timestep.
Yang teaches: prune the plurality of tokens based on the attention map to obtain a pruned set of tokens by identifying a denoising timestep (pg. 3 “Conventional diffusion models use a heavy network for all denoising steps, ignoring the differences between the steps. However, we hypothesize that some steps in the generation process may be easier to process, and that even a lightweight network can handle these steps.”; section 4 “Step-aware Network for Diffusion Models” discusses determining which steps to prune) and pruning the plurality of tokens according to a denoising- steps-aware pruning (DSAP) based on the denoising timestep (pg. 2 “We show the feasibility of this approach and present the Denoising Diffusion Step-aware Models (DDSM). In DDSM, the neural network is variable and slimmable at different steps to avoid redundant computation at unimportant steps. We determine the capacity of the neural network at each step via evolutionary search and prune the network to various scales accordingly.”).
Yang is analogous to the claimed invention because it is in the same field of pruning a diffusion model to improve efficiency. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Bolya in view of Thorsley with the teachings of Yang to apply Yang’s steps-aware pruning to the token pruning system of Bolya in view of Thorsley. The motivation would have been “to avoid redundant computation at unimportant steps” (Yang pg. 2).
Regarding claim 13, the combination of Bolya in view of Thorsley and Yang teaches: The non-transitory computer readable medium of claim 12, wherein pruning the plurality of tokens comprises:
computing an importance score for each of the plurality of tokens based on the attention map (Thorsley [0043] “Each head has the attention probabilities determined using the process described with respect to FIG. 1C with the scaled dot-product attention layers 131… The token importance score s.sup.(l) 340 may be calculated from the mean 330 of the attention probability over all the heads.”, where element 131 references the process of generating an attention matrix/map); and
identifying a threshold importance score (Thorsley [0044] “Alternatively, using a threshold token pruning operation 360, a token may be kept at 362 if the importance score of the token exceeds an absolute threshold value, and may be pruned at 364 otherwise.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Bolya in view of Thorsley and Yang with the additional teachings of Thorsley to compute importance scores and a threshold importance score for token pruning. The motivation would have been to include a method of establishing metrics or criteria by which tokens are selected for pruning.
Regarding claim 14, the combination of Bolya in view of Thorsley and Yang teaches: The non-transitory computer readable medium of claim 12, wherein generating the synthetic output comprises:
identifying a plurality of pruned tokens (Bolya pg. 2 section 3.1 “Defining Unmerging”: “And if we have information about what tokens we merged, we have enough information to then unmerge those same tokens.”; pruned tokens could be identified in the same manner);
generating a plurality of replacement tokens corresponding to the plurality of pruned tokens; and adding the plurality of replacement tokens to the pruned set of tokens to obtain an augmented set of tokens (Bolya pg. 2 section 3.1 “Defining Unmerging” describes duplicating existing merged tokens, which were identified based on similarity, and copying them into previously filled token slots; a similar procedure could be applied to pruned tokens).
References Cited
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Tewel et al. (US 20250252614 A1) teaches a method of generating images using a pre-trained diffusion model; it describes generating an attention map using tokens corresponding to the image being processed.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENJAMIN STATZ whose telephone number is (571)272-6654. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571)272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BENJAMIN TOM STATZ/Examiner, Art Unit 2611
/TAMMY PAIGE GODDARD/Supervisory Patent Examiner, Art Unit 2611