DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-3, 5-21 are currently pending in the present application, with claims 1, 14, and 19 being independent.
Response to Amendments / Arguments
Applicant's arguments filed 03/23/2026 have been fully considered but they are not persuasive.
Applicant argues: None of the asserted references (Gandikota et al. "Erasing Concepts from Diffusion Models", Cornell University, arXiv Preprint, arxiv.org [retrieved 2024-01-24]. Retrieved from the Internet <https://arxiv.org/pdf/2303.07345.pdf>, 03/13/2023, 23 pages, Meng et al. "Locating and editing factual associations in gpt." Advances in neural information processing systems 35 (2022): 17359-17372. (Year: 2022), Zhang et al. "Continuous layout editing of single images with diffusion models." In Computer Graphics Forum, vol. 42, no. 7, p. e14966. 2023. (Year: 2023)), alone or in combination teach or suggest:
“(1) Form a restored machine-learning model from the corrupted machine-learning model, the corrupted machine-learning model and the restored machine-learning model implemented independent and distinct of the text-to-image model”
“(2) The corrupted machine-learning model implemented independent and distinct from the text-to-image model” and “the restored machine-learning model implemented independent and distinct from the text-to-image model”
Examiner replies: Gandikota expressly teaches a text-to-image synthesis framework having separate model components and separately configured model instances, specifically (Section 4; We focus on Stable Diffusion, an LDM that consists of 3 subnetworks, a text encoder T, a diffusion model (U-Net) 0*, and a decoder model D…”). Gandikota further teaches that “training uses several instances of the diffusion model, with one set of parameters frozen (0*) while training the other set of parameters (0) to erase the concept…we perform inference on the frozen model 0* twice…once conditioned on c and the other unconditioned…”. Thus, Gandikota discloses multiple separately configured and separately executed model instances within a text-to-image diffusion framework.
Meng expressly teaches corruption and restoration operations performed on model activations, specifically (Section 2.1; baseline corrupted run, the subject is obfuscated from G before the network runs…set of corrupted activations…). Meng further teaches a “corrupted-with-restoration run” in which “we hook G so that it is forced to output the clean state h(I)^I” (Section 2.1) and “restore selected internal activations to their clean value” (Fig. 1 description (c)). Meng additionally teaches that restoring the activations causes “the output to return to the original prediction” (). Thus, Meng discloses generating corrupted and restored operational model configurations through activation corruption and activation restoration operations.
The combined teachings of Gandikota and Meng therefore teach a text-to-image model framework (Gandikota Section 4), a corruption of activations to form a corrupted machine-learning model configuration and restoration of activations from the original model configuration to form a restored machine-learning model configuration (Meng Section 2.1), and therefore, separately configured model instances/configurations relative to the underlying text-to-image framework.
Further, in light of Par. 0055-0059 and Fig. 6 of Applicant’s disclosure, describing the corrupted and restored machine-learning models as modified or separately instantiated model configurations derived relative to an underlying text-to-image model, rather than separate independently trained neural-network architectures. Additionally, the claims do not require the corrupted machine-learning model and restored machine-learning model themselves to be structurally independent from one another or separately trained machine-learning models. Under broadest reasonable interpretation, the claimed corrupted and restored machine-learning models reasonably encompass derived or modified instances/configurations of a common model framework generated from one another through activation modification and restoration operations, rather than requiring fully separate standalone neural-network architectures. Accordingly, the combined teachings of Gandikota and Meng teach or at least render obvious “(1) Form a restored machine-learning model from the corrupted machine-learning model, the corrupted machine-learning model and the restored machine-learning model implemented independent and distinct of the text-to-image model” and “(2) The corrupted machine-learning model implemented independent and distinct from the text-to-image model” and “the restored machine-learning model implemented independent and distinct from the text-to-image model”.
To the extent applicant intends to narrow the limitation/relationship between the recited models, examiner suggests amending the claims to further clarify the structural and training independence between the recited machine-learning models (text-to-image, corrupted, restored, edited text-to-image machine-learning models), possibly explicitly reciting each of the machine-learning models are separately trained, structurally distinct neural-network architectures, or otherwise independently implemented relative to one another. Such amendments may assist in clarifying the intended scope of the claims and advancing prosecution to overcome the current rejections.
Regarding the remaining arguments: Applicant argues with respect to the amended claim language, which is fully addressed in the prior art rejections set forth below.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION. —The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding independent claim 1, the recitation of “the corrupted machine-learning model and the restored machine-learning model implemented independent and distinct of the text-to-image model” and “the edited machine-learning model implemented independent and distinct from the corrupted machine-learning model and the restored machine-learning model” renders the scope unclear. It is ambiguous as to what degree or type of “independent” and “distinct” relationship is required between the recited models.
It is unclear whether the corrupted machine-learning model and restored machine-learning model are required to be entirely separate neural-network architectures, separately instantiated copies of the same model, separately executed operational configurations, different internal model states, or distinct portions of a common model-editing framework. The disclosure describes the corrupted machine-learning model and restored machine-learning model as being generated from and modified relative to an underlying machine-learning model through activation corruption and replacement operations (Specification Par. 0055-0062 and Fig. 6), where Fig. 6 shows all the processes as being a part of the encoder-decoder knowledge tracing module, where the results of one model appear to be dependent on another model. It is unclear whether these models are independent and distinct from each other, and if distinct could just be different parts of the same process. Accordingly, one of ordinary skill in the art would not be able to reasonable determine the scope of the claimed relationship between the recited models, as the filed disclosure does not clearly define what degree of “independent and distinct” relationship is required.
The examiner respectfully requests the applicant to clarify the scope of the claimed invention.
Claims 14 and 19 recites substantially similar subject matter as to that of claim 1 and is rejected using substantially similar rationale as to that which was set forth with respect to claim 1.
Claims depending thereon are also rejected for substantially similar reasons as that set forth for the claims from which they depend on.
Claims 1-20 will be examined as best understood by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-7, 12-17, and 19-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gandikota et al. "Erasing Concepts from Diffusion Models", Cornell University, arXiv Preprint, arxiv.org [retrieved 2024-01-24]. Retrieved from the Internet <https://arxiv.org/pdf/2303.07345.pdf>, 03/13/2023, 23 pages, hereinafter referred to as “Gandikota”, in view of Meng et al. "Locating and editing factual associations in gpt." Advances in neural information processing systems 35 (2022): 17359-17372. (Year: 2022), hereinafter referred to as “Meng”, in further view of Zhang et al. "Continuous layout editing of single images with diffusion models." In Computer Graphics Forum, vol. 42, no. 7, p. e14966. 2023. (Year: 2023), hereinafter referred to as “Zhang”.
Regarding claim 1, Gandikota discloses identifying, (Pg. 2429, Section 4; text-to-image diffusion models…diffusion model (U-Net)), the location (see Pg. 2429-2430, Section 4.1 and Fig. 3-4 on attention modules; cross-attention/cross-attention parameters, and self-attention/non-cross-attention parameters, unconditional layers) supporting a trained ability of the text-to-image model to generate a visual attribute in a digital image (Fig. 3; generation of two similar car images conditioned on different prompts…local contributions of the first attention modules of the 3rd up sampling block of the Stable Diffusion U-net while generating the images
(Section 4; training uses several instances of the diffusion model, with one set of parameters frozen (0*) while training the other set of parameters (0) to erase the concept…we perform inference on the frozen model 0* twice…once conditioned on c and the other unconditioned… Examiner’s note: as previously mentioned above in examiner’s response to arguments); and
detecting a change has been made to edit the visual attribute (Fig. 2; The optimization process for erasing undesired visual concepts from the pre-trained diffusion model weights involves using a short text description of the concept as guidance…c --> concept to erase. Pg. 2429, Section 4; editing the pre-trained diffusion U-Net model weights to remove a specific style or concept) by comparing a candidate digital image (Fig. 1, 4-5, 7, 9-11 showing restored/edited outputs as candidate images from the edited model) to a digital image generated by the text-to-image model (Fig. 1, 4-5, 7, 9-11 showing images generated from the original model),
forming, an edited text-to-image model (Pg. 2428, Fig. 2; ESD model. Pg. 2429, Section 4, train new parameters θ) by editing the location of the text-to-image model (Fig. 3; cross-attention and self-attention. Fig. 4; modifying the cross-attention weights, ESD-x…altering the non-cross-attention weights, ESD-u), the editing causing removal of the trained ability of the text-to-image model to generate the visual attribute (Pg. 2429, Section 4 and 4.1; our approach involves editing the pre-trained diffusion U-Net model weights to remove a specific style or concept…fine tuning the cross attentions, ESD-x, when the erasure is required to be controlled and specific to the prompt...Further we propose fine tuning unconditional layers (non-cross-attention modules), ESD-u, when the erasure is required to be independent of the text in the prompt…) in a subsequent digital image (Pg. 2426, Figure 1 edited model images), the edited machine-learning model implemented independent and distinct from the corrupted machine-learning model and the restored machine-learning model (Section 4; training uses several instances of the diffusion model, with one set of parameters frozen (0*) while training the other set of parameters (0) to erase the concept…we perform inference on the frozen model 0* twice…once conditioned on c and the other unconditioned…Examiner’s note: as previously mentioned above in examiner’s response to arguments);
and generating, the subsequent digital image by the edited text-to-image model, in which, the visual attribute is removed (Pg. 2426, Figure 1 edited model images; method fine-tunes model weights to erase the targeted concept. Pg. 2430, Figure 4; the effects of parameter choices on artist style removal are illustrated).
Gandikota does not disclose that the neural network/MLM is to be processed by a processing device. Official Notice is however taken that it is well known and conventional to use neural networks within a computer comprising a processor and memory, and such as implementation would be considered implied or otherwise an obvious implementation according to standard techniques in the art with entirely predictable outcomes.
Additionally, Gandikota does not disclose applying activations from nodes included in at least one layer of the text-to-image model to nodes of the corrupted machine-learning model to form a restored machine learning model, the corrupted machine-learning model and the restored machine-learning model implemented independent and distinct of the text-to-image model, and a candidate digital image generated by the restored machine-learning model.
In the same art of machine learning models, Meng discloses applying activations from nodes included in at least one layer (Pg. 3, Section 2.1; observe all of G's internal activations during 3 runs) l (Fig. 1; once where we corrupt the subject token and then (c) restore selected internal activations to their clean value…The causal impact on output probability is mapped for the effect of (e) each hidden state on the prediction, (f) only MLP activations, and (g) only attention activations.) to form a restored machine learning model (Pg. 3, Section 2.1; corrupted-with-restoration-run…), the corrupted machine-learning model and the restored machine-learning model implemented independent and distinct of the text-to-image model (Section 2.1; clean run…baseline corrupted run…corrupted-with-restoration-run. Examiner’s note; as previously presented in examiner’s response, separately configured, separately instantiated, operationally distinct model configurations relative to the underlying text-to-image model framework reasonably encompass “implemented independent and distinct of the text-to-image model” limitation)
(Pg. 3, Section 2.1; corrupted-with-restoration-run…Fig. 1; restore selected internal activations to their clean value).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Meng’s corrupted and restored model approach within Gandikota’s ESD text-to-image diffusion model in order to generate corrupted model runs with noise, generate restored runs, compare candidate images from restored runs to original uncorrupted generations, and generate location indication corresponding to node activations in at least one layer. Both Meng and Gandikota operate on Transformer-based models, focused on attention layers and internal activations. Gandikota discloses which parameter subsets control visual attributes (Section 4.1, cross-attention layers, self-attention layers, unconditional layers) while Meng provides a corrupted/restored protocol that directly identifies which activations are causally responsible for an attribute (Section 2.1). A person of ordinary skill in the art would have been motivated to apply Meng’s causal tracing method to Gandikota’s U-Net attention layers to provide a precise way to identify which layers directly control a corresponding visual attribute, yielding the predictable results in accurately targeting of attention modules for editing (Additional examiner’s note: Meng teaches generating corrupted and restored operational instances/configurations of an underlying machine-learning model through activation corruption and restoration operations, while Gandikota discloses applying model-editing operations to a text-to-image diffusion model for removing visual concepts. It would have been obvious to one of ordinary skill in the art to apply Meng’s activation-based causal tracing and restoration techniques to Gandikota’s text-to-image model framework to identify and edit locations responsible for generating a visual attribute).
Gandikota in view of Meng does not disclose the identifying performed over a number of iterations until a threshold similarity is achieved, and comparing a candidate digital image generated by the restored machine-learning model as having the threshold similarity achieved for the visual attribute to a digital image.
Zhang discloses the identifying performed over a number of iterations until a threshold similarity is achieved, (Section 3.2, Pg. 4, Right Column, Algorithm 1; a set of thresholds {Qt}…steps 11-18. Section 3.3, Pg. 5, Left Column; We apply iterative optimization for 40 iterations with early stopping thresholds of 0.4, 0.3, and 0.2 when t is 1.0, 0.8, and 0.6, respectively), and comparing a candidate digital image (Section 3.2, Pg. 4, Right Column, Algorithm 1; Step 28, Decode the edited image) (Section 3.2, Pg. 4, Right Column and Fig. 3; Eqns. (6) and (7) the region loss can be calculated…loss for the k-th object with target position specified…is the cross-attention map with the optimized token…at layer l, and I is the spatial position…model can control the positions of each object and simultaneously focus on the object with a large loss…to preserve the original background, we start to blend with the original input image… Section 3.3, Pg. 5, Left Column; We apply iterative optimization for 40 iterations with early stopping thresholds of 0.4, 0.3, and 0.2 when t is 1.0, 0.8, and 0.6, respectively. Examiners note: computing a similarity/match proxy (loss derived from attention-map alignment with target region) and using thresholds/early stopping to determine when the candidate output satisfies the criterion)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Gandikota’s ESD text-to-image diffusion model and Meng’s three-run framework with Zhang’s threshold similarity technique. Doing so allows a repeatable and quantitative stop condition to confirm that the edit has taken effect, yielding predictable results in more precise targeting, reducing unintended side effects on unrelated content, and overall editing accuracy.
Regarding claim 2, Gandikota in view of Meng, and in further view of Zhang discloses the method of claim 1, and Gandikota further discloses the visual attribute (Gandikota Pg. 2432-2433, Section 5.3 on Object. Pg. 2430-2431 and Fig. 4-5 style. Pg. 2429; Section 4; c represents the concept to erase) and the text-to-image-model (Pg. 2428-2429, Section 3.2 and 4; LDM that consists of 3 subnetworks: a text encoder T, a diffusion model (U-Net) 8* and a decoder model D).
Gandikota does not disclose wherein the identifying is performed using causal mediation analysis by analyzing causal inference through change in a response variable following an intervention on intermediate variables of interest
In the same art of machine learning models, Meng discloses wherein the identifying is performed using causal mediation analysis (Fig. 1-2 and Section 2.1; wish to understand if there are specific hidden state variables that are more important than others when recalling a fact. As Vig et al. (2020b) have shown, this is a natural case for causal mediation analysis, which quantifies the contribution of intermediate variables in causal graphs (Pearl, 2001)) by analyzing causal inference through change in a response variable following an intervention on intermediate variables of interest (Pg. 2, Section 2 Interventions of Activations and Pg. 5, Section 3 Interventions on Weights)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate Meng’s causal tracing approach for language models into Gandikota’s ESD model. The motivation lies in the advantage of enabling more accurate localization of visual attributes, yielding predictable results in editing targeted objects, concepts, or styles while still preserving unrelated content.
Regarding claim 3, Gandikota in view of Meng, and in further view of Zhang discloses the method of claim 1, and Gandikota further discloses wherein the location is identified as corresponding to one or more nodes included in at least one layer of the text-to-image model (Fig. 3; first attention modules of the 3rd upsampling block of the Stable Diffusion U-net while generating the images. Examiner's note: these are "nodes in a layer") and the editing the location includes editing one or more activations associated with the one or more nodes included in the at least one layer (Fig. 4; modifying the cross-attention weights, ESD-x…altering the non-cross-attention weights, ESD-u. Examiner's note: weight edits change neuron activations during inference).
The motivation to combine would’ve been the same as that set forth above with respect to claim 1.
Regarding claim 5, Gandikota in view of Meng, and in further view of Zhang discloses the method of claim 1, and Gandikota further discloses wherein the corrupted machine-learning model is generated by applying Gaussian noise to the text-to-image model (Pg. 2428-2429, Section 3.2; for an image x, noise is added to its encoded latent, z = E(x) leading to zt where the noise level increases with t…The inference process starts from a Gaussian noise zT ~ N(0, 1)).
The motivation to combine would’ve been the same as that set forth above with respect to claim 1.
As described above in examiner’s response to arguments, while Gandikota does not explicitly disclose a “corrupted model,” under broadest reasonable interpretation, the application of noise, perturbation of latent variables, and induced changes to internal activations constitute a corrupted operational state of the model relative to its baseline behavior.
Regarding claim 6, Gandikota in view of Meng, and in further view of Zhang discloses the method of claim 5, and Gandikota further discloses wherein the Gaussian noise (Pg. 2429, Section 3.2; The inference process starts from a Gaussian noise zT ~ N(0, 1)) is applied to:
a symmetric encoder-decoder machine learning model of the text-to-image model; or
a text encoder machine learning model of the text-to-image model (Pg. 2428-2429, Section 3.2; for an image x, noise is added to its encoded latent, z = E(x) leading to zt where the noise level increases with t…Pg. 2429, Section 4; We focus on Stable Diffusion (SD), an LDM that consists of 3 subnetworks: a text encoder T, a diffusion model (U-Net) θ* and a decoder model D).
The motivation to combine would’ve been the same as that set forth above with respect to claim 1.
Regarding claim 7, Gandikota in view of Meng, and in further view of Zhang discloses the method of claim 1, and Gandikota further discloses wherein the editing the location within the text-to-image model includes editing at least one weight matrix associated with a layer of the text-to-image model (Pg. 2429, Section 4; Our approach involves editing the pre-trained diffusion U-Net model weights to remove a specific style or concept. Fig. 4; modifying the cross-attention weights, ESD-x…altering the non-cross-attention weights, ESD-u).
The motivation to combine would’ve been the same as that set forth above with respect to claim 1.
Regarding claim 12, Gandikota in view of Meng, and in further view of Zhang discloses the method of claim 1, and Gandikota further discloses wherein the text-to-image model (Fig. 1; Original model. Fig. 2; pre-trained diffusion model) is configured to generate the digital image (Fig. 1; original model image. Fig. 2; xt --> (image), generated by θ) as having the visual attribute based on a text input (Fig. 1; short text description. Fig. 2; c--> "Van Gogh", concept to erase…short description of the concept as guidance) and the edited text-to-image model (Fig. 1; Edited model. Fig. 2; ESD model) is configured to generate the subsequent digital image as having a change to the visual attribute based on the text input (Fig. 1; given only a short text description of an undesired visual concept and no additional data, our method fine-tunes model weights to erase the targeted concept. Fig. 2; The optimization process for erasing undesired visual concepts from pre-trained diffusion model weights involves using a short text description of the concept as guidance).
The motivation to combine would’ve been the same as that set forth above with respect to claim 1.
Regarding claim 13, Gandikota in view of Meng, and in further view of Zhang discloses the method of claim 1, and Gandikota further discloses wherein the location is configured to influence generation of the visual attribute by the text-to-image model (Pg. 2429, Section 4.1 Importance of Parameter Choice) and the forming is based on the location and causes causing a change to the visual attribute in generating the subsequent digital image by the edited text-to-image model (Fig. 3; when comparing generation of two similar car images conditioned on different prompts…Pg. 2429-2430, Section 4.1; The effects of parameter choices on artist style removal are illustrated in Fig. 4: when erasing the "Van Gogh" style ESD-u and other unconditional parameter choices erase aspects of the style globally…tuning the cross-attention parameters only (ESD-x) erases the distinctive style of Van Gogh specifically when his name is mentioned in the prompt).
The motivation to combine would’ve been the same as that set forth above with respect to claim 1.
Regarding claim 14, Gandikota in view of Meng, and in further view of Zhang discloses the method of claim 1, and Gandikota further discloses generating a corrupted machine-learning model by applying Gaussian noise to activations of nodes from the text-to-image model (Pg. 2428-2429, Section 3.2; for an image x, noise is added to its encoded latent, z = E(x) leading to zt where the noise level increases with t…The inference process starts from a Gaussian noise zT ~ N(0, 1)), the corrupted machine-learning model implemented independent and distinct from the text-to-image model (Pg. 2428, Fig. 2; ESD model. Section 4; We focus on Stable Diffusion, an LDM that consists of 3 subnetworks, a text encoder T, a diffusion model (U-Net) 0*, and a decoder model D…training uses several instances of the diffusion model, with one set of parameters frozen (0*) while training the other set of parameters (0) to erase the concept…we perform inference on the frozen model 0* twice…once conditioned on c and the other unconditioned…Examiner’s note: as previously mentioned above in examiner’s response to arguments), generating a candidate digital image (Fig. 1, 4-5, 7, 9-11 showing restored/edited outputs as candidate images from the edited model), and indicating a location within the text-to-image model corresponding to a visual attribute based on the candidate digital image (Fig. 3 and Section 4.1; attention modules and parameter choices).
As described above in examiner’s response to arguments, while Gandikota does not explicitly disclose a “corrupted model,” under broadest reasonable interpretation, the application of noise, perturbation of latent variables, and induced changes to internal activations constitute a corrupted operational state of the model relative to its baseline behavior.
Gandikota does not appear to explicitly disclose a system comprising: a processing device; and a computer-readable storage medium storing instructions that are non-transitory and, responsive to execution by the processing device, causes the processing device to perform operations.
Official Notice is however taken that it is well known and conventional to use neural networks within a computer comprising a processor and memory, and such as implementation would be considered implied or otherwise an obvious implementation according to standard techniques in the art with entirely predictable outcomes.
Additionally, Gandikota does not disclose generating a restored machine-learning model by applying one or more activations from one or more nodes of the text-to-image model to the corrupted machine-learning model, the restored machine-learning model implemented independent and distinct from the text-to-image model.
In the same art of machine learning models, Meng discloses generating a restored machine-learning model (Pg. 3, Section 2.1; corrupted-with-restoration-run…) by applying one or more activations from one or more nodes (Fig. 1; once where we corrupt the subject token and then (c) restore selected internal activations to their clean value…The causal impact on output probability is mapped for the effect of (e) each hidden state on the prediction, (f) only MLP activations, and (g) only attention activations). the restored machine-learning model implemented independent and distinct from the text-to-image model (Section 2.1; clean run…baseline corrupted run…corrupted-with-restoration-run. Examiner’s note; as previously presented in examiner’s response, separately configured, separately instantiated, operationally distinct model configurations relative to the underlying text-to-image model framework reasonably encompass “implemented independent and distinct of the text-to-image model” limitation).
The motivation to combine would’ve been the same as that set forth above with respect to claim 1.
Regarding claim 15, Gandikota in view of Meng, and in further view of Zhang discloses the system of claim 14, and Gandikota further discloses wherein the generating a corrupted machine-learning model includes applying Gaussian noise (Gandikota Pg. 2428-2429, Section 3.2; for an image x, noise is added to its encoded latent, z = E(x) leading to zt where the noise level increases with t...The inference process starts from a Gaussian noise zT ~ N(O, 1)) to a symmetric encoder-decoder machine-learning model of the text-to-image model or a text encoder machine-learning model of the text-to-image model (Gandikota Pg. 2428-2429, Section 3.2; for an image x, noise is added to its encoded latent, z = E(x) leading to zt where the noise level increases with t...Pg. 2429, Section 4; We focus on Stable Diffusion (SD), an LDM that consists of 3 subnetworks: a text encoder T, a diffusion model (U-Net) @* and a decoder model D).
As described above in examiner’s response to arguments, while Gandikota does not explicitly disclose a “corrupted model,” under broadest reasonable interpretation, the application of noise, perturbation of latent variables, and induced changes to internal activations constitute a corrupted operational state of the model relative to its baseline behavior.
Regarding claim 16, Gandikota in view of Meng, and in further view of Zhang discloses the system of claim 14, and Gandikota further discloses wherein the operations further comprising forming an edited text-to-image model (Gandikota Pg. 2429; Section 4; to remove a specific style or concept...c represents the concept to erase) by editing the location within the text-to-image model (Gandikota Pg. 2429, Section 4.1 Importance of Parameter Choice), the editing causing a change to the visual attribute in generating a subsequent digital image by the edited text-to-image model (Gandikota Fig. 3; when comparing generation of two similar car images conditioned on different prompts...Pg. 2429-2430, Section 4.1; The effects of parameter choices on artist style removal are illustrated in Fig. 4: when erasing the "Van Gogh" style ESD-u and other unconditional parameter choices erase aspects of the style globally...tuning the cross-attention parameters only (ESD-x) erases the distinctive style of Van Gogh specifically when his name is mentioned in the prompt).
The motivation to combine would’ve been the same as that set forth above with respect to claim 1.
Regarding claim 17, Gandikota in view of Meng, and in further view of Zhang discloses the system of claim 14, and Gandikota further discloses wherein the editing the location within the text-to-image model includes editing at least one weight matrix associated with a layer of the text-to-image model (Gandikota Pg. 2429, Section 4; Our approach involves editing the pre-trained diffusion U-Net model weights to remove a specific style or concept. Fig. 4; modifying the cross-attention weights, ESD-x... altering the non-cross- attention weights, ESD-u).
The motivation to combine would’ve been the same as that set forth above with respect to claim 1.
Regarding claim 19, Gandikota discloses generating a corrupted machine-learning model based on a text-to-image model by applying Gaussian noise to activations of nodes from the text-to-image model (Pg. 2428-2429, Section 3.2; for an image x, noise is added to its encoded latent, z = E(x) leading to zt where the noise level increases with t…The inference process starts from a Gaussian noise zT ~ N(0, 1. Pg. 2428-2429, Section 3.2 and 4; LDM that consists of 3 subnetworks: a text encoder T, a diffusion model (U-Net) θ* and a decoder model D), the corrupted machine-learning model implemented independent and distinct from the text-to-image model
identifying a location (Fig. 3; first attention modules of the 3rd upsampling block of the Stable Diffusion U-net while generating the images. Examiner's note: these are "nodes in a layer") by detecting a change has been made to edit a visual attribute (Fig. 2; The optimization process for erasing undesired visual concepts from the pre-trained diffusion model weights involves using a short text description of the concept as guidance…c --> concept to erase. Pg. 2429, Section 4; editing the pre-trained diffusion U-Net model weights to remove a specific style or concept) by comparing a candidate digital image (Fig. 1, 4-5, 7, 9-11 showing restored/edited outputs as candidate images from the edited model) to a digital image generated by the text-to-image model (Fig. 1, 4-5, 7, 9-11 showing images generated from the original model), and forming an edited text-to-image model (Pg. 2428, Fig. 2; ESD model. Pg. 2429, Section 4, train new parameters θ) by editing the text-to-image model based on the location (Fig. 3; cross-attention and self-attention. Fig. 4; modifying the cross-attention weights, ESD-x…altering the non-cross-attention weights, ESD-u), the editing causing a change to the visual attribute in generating a subsequent digital image by the edited text-to-image model (Pg. 2429, Section 4 and 4.1; our approach involves editing the pre-trained diffusion U-Net model weights to remove a specific style or concept…fine tuning the cross attentions, ESD-x, when the erasure is required to be controlled and specific to the prompt...Further we propose fine tuning unconditional layers (non-cross-attention modules), ESD-u, when the erasure is required to be independent of the text in the prompt…Pg. 2426, Figure 1 edited model images)
Gandikota does not appear to explicitly disclose one or more computer-readable media storing instructions that are non-transitory and, responsive to execution by a processing device, causes the processing device to perform operations.
Official Notice is however taken that it is well known and conventional to use neural networks within a computer comprising a processor and memory, and such as implementation would be considered implied or otherwise an obvious implementation according to standard techniques in the art with entirely predictable outcomes.
Additionally, Gandikota does not disclose applying activations from nodes included in at least one layer of the text-to-image model to nodes of the corrupted machine-learning model over a plurality of iterations to form, respectively, a plurality of iterations of a restored machine-learning model, the restored machine learning model implemented independent and distinct from the text-to-image model.
In the same art of machine learning models, Meng discloses applying activations from nodes included in at least one layer (Pg. 3, Section 2.1; observe all of G's internal activations during 3 runs) to nodes of the corrupted machine-learning model (Fig. 1; once where we corrupt the subject token and then (c) restore selected internal activations to their clean value…The causal impact on output probability is mapped for the effect of (e) each hidden state on the prediction, (f) only MLP activations, and (g) only attention activations.) over a plurality of iterations to form, respectively, a plurality of iterations of a restored machine-learning models (Section 2.1; corrupted-with-restoration-run…Fig. 1; restore selected internal activations to their clean value), the restored machine-learning model implemented independent and distinct from the text-to-image model (Section 2.1; clean run…baseline corrupted run…corrupted-with-restoration-run. Examiner’s note; as previously presented in examiner’s response, separately configured, separately instantiated, operationally distinct model configurations relative to the underlying text-to-image model framework reasonably encompass “implemented independent and distinct of the text-to-image model” limitation).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Meng’s corrupted and restored model approach within Gandikota’s ESD text-to-image diffusion model in order to generate corrupted model runs with noise, generate restored runs, compare candidate images from restored runs to original uncorrupted generations, and generate location indication corresponding to node activations in at least one layer. Both Meng and Gandikota operate on Transformer-based models, focused on attention layers and internal activations. Gandikota discloses which parameter subsets control visual attributes (Section 4.1, cross-attention layers, self-attention layers, unconditional layers) while Meng provides a corrupted/restored protocol that directly identifies which activations are causally responsible for an attribute (Section 2.1). A person of ordinary skill in the art would have been motivated to apply Meng’s causal tracing method to Gandikota’s U-Net attention layers to provide a precise way to identify which layers directly control a corresponding visual attribute, yielding the predictable results in accurately targeting of attention modules for editing.
Gandikota in view of Meng does not disclose comparing a candidate digital image generated by the restored machine-learning model as having the threshold similarity achieved for the visual attribute to a digital image.
Zhang discloses comparing a candidate digital image (Section 3.2, Pg. 4, Right Column, Algorithm 1; Step 28, Decode the edited image) (Section 3.2, Pg. 4, Right Column, Algorithm 1; a set of thresholds {Qt}…steps 11-18. Section 3.2, Pg. 4, Right Column and Fig. 3; Eqns. (6) and (7) the region loss can be calculated…loss for the k-th object with target position specified…is the cross-attention map with the optimized token…at layer l, and I is the spatial position…model can control the positions of each object and simultaneously focus on the object with a large loss…to preserve the original background, we start to blend with the original input image… Section 3.3, Pg. 5, Left Column; We apply iterative optimization for 40 iterations with early stopping thresholds of 0.4, 0.3, and 0.2 when t is 1.0, 0.8, and 0.6, respectively. Examiners note: computing a similarity/match proxy (loss derived from attention-map alignment with target region) and using thresholds/early stopping to determine when the candidate output satisfies the criterion)
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to combine Gandikota’s ESD text-to-image diffusion model and Meng’s three-run framework with Zhang’s threshold similarity technique. Doing so allows a repeatable and quantitative stop condition to confirm that the edit has taken effect, yielding predictable results in more precise targeting, reducing unintended side effects on unrelated content, and overall editing accuracy.
Regarding claim 20, Gandikota in view of Meng, and in further view of Zhang discloses the one or more computer-readable media of claim 19, and Gandikota further discloses discloses wherein the editing the location within the text-to-image model includes editing at least one weight matrix associated with a layer of the text-to- image model (Gandikota Pg. 2429, Section 4; Our approach involves editing the pre-trained diffusion U- Net model weights to remove a specific style or concept. Fig. 4; modifying the cross-attention weights, ESD-x... altering the non-cross-attention weights, ESD-u).
The motivation to combine would’ve been the same as that set forth above with respect to claim 1.
Regarding claim 21, Gandikota in view of Meng, and in further view of Zhang discloses the one or more computer-readable media as described in claim 19, and further discloses wherein the corrupted machine-learning model (Meng Pg. 3, Section 2.1; baseline corrupted run, the subject is obfuscated...immediately after x is embedded...set of corrupted activations...noisy embeddings as in the corrupted baseline) is generated by applying the Gaussian noise (Gandikota Pg. 2428-2429, Section 3.2; for an image x, noise is added to its encoded latent, z = E(x) leading to zt where the noise level increases with t...The inference process starts from a Gaussian noise zT ~ N(O, 1)) to: a symmetric encoder-decoder machine learning model of the text-to-image model; or a text encoder machine learning model of the text-to-image model (Gandikota Pg. 2428-2429, Section 3.2; for an image x, noise is added to its encoded latent, z = E(x) leading to zt where the noise level increases with t...Pg. 2429, Section 4; We focus on Stable Diffusion (SD), an LDM that consists of 3 subnetworks: a text encoder T, a diffusion model (U-Net) @* and a decoder model D)).
The motivation to combine would’ve been the same as that set forth above with respect to claim 1.
Claim(s) 8-9, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gandikota et al. "Erasing Concepts from Diffusion Models", Cornell University, arXiv Preprint, arxiv.org [retrieved 2024-01-24]. Retrieved from the Internet <https://arxiv.org/pdf/2303.07345.pdf>, 03/13/2023, 23 pages, hereinafter referred to as “Gandikota”, in view of Meng et al. "Locating and editing factual associations in gpt." Advances in neural information processing systems 35 (2022): 17359-17372. (Year: 2022), hereinafter referred to as “Meng”, in further view of Zhang et al. "Continuous layout editing of single images with diffusion models." In Computer Graphics Forum, vol. 42, no. 7, p. e14966. 2023, hereinafter referred to as “Zhang”, and in further view of Vaswani et al. "Attention is all you need." Advances in neural information processing systems 30 (2017), hereinafter referred to as “Vaswani”.
Regarding claim 8, Gandikota in view of Meng, and in further view of Zhang discloses the method as described in claim 7, and further discloses the text-to-image model (Gandikota Pg. 2429, Section 4; LDM that consists of 3 subnetworks: a text encoder T, a diffusion model (U-Net) θ* and a decoder model D).
Gandikota in view of Meng and in further view of Zhang does not disclose wherein the weight matrix is a projection matrix associated with a self-attention layer of a text-encoder machine-learning model.
In the same art of attention mechanisms, Vaswani discloses wherein the weight matrix (Pg. 3, Section 3.2; weight assigned to each value is computed by a compatibility function of the query with the corresponding key. Pg. 4, Section 3.2.1; compute the attention function on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V) is a projection matrix associated with a self-attention layer (Pg. 4-5, Section 3.3.2; linearly project the queries, keys, and values h times…on each projected versions of queries, keys, and values we then perform the attention function in parallel…projections are parameter matrices…) of a text-encoder machine-learning model (Pg. 5, Section 3.2.3; encoder-decoder attention mechanisms…encoder contains self-attention layers…).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate Vaswani’s teaching of a projection matrix associated with the self-attention layer of a text-encoder model into Gandikota’s ESD model and Meng’s corrupted-with-restoration technique. Since Gandikota already modifies cross-attention and self-attention weights and parameters to remove concepts (Fig. 3 and 4), applying Vaswani’s approach using projection matrices yields a predictable way to implement the weight matrices with standard structure used in Transformer models. Doing so allows efficient computation of query-key-value relationships and improved accuracy in editing targeted concepts/objects in text-to-image diffusion models. Thus, the combination would have been obvious as it applies a well-known structural implementation.
Regarding claim 9, Gandikota in view of Meng, in further view of Zhang, and in further view of Vaswani discloses the method as described in claim 8, Gandikota, Meng, and Zhang does not disclose wherein the self-attention layer is a first layer of the text-encoder machine-learning model.
In the same art of attention mechanisms, Vaswani discloses wherein the self-attention layer is a first layer of the text-encoder machine-learning model (Vaswani Pg. 2-3, Section 3.1 and Fig. 1; The encoder is composed of a stack of N=6 identical layers. Each layer has two sub-layers. The first is a multihead self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate Vaswani’s teaching of self-attention layer being the first layer of the encoder MLM into Gandikota’s ESD model and Meng’s corrupted-with-restoration technique. Doing so allows conditioning image generation on text prompts. It is well known for Transformer model architectures to use self-attention as the first layer in each block, therefore, applying the conventional ordering of attention mechanisms would have been a routine and predictable design choice.
Regarding claim 18, claim 18 has similar limitations as of claim 8, except it is a system claim, therefore it is rejected under the same rationale as claim 8.
Claim(s) 10-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gandikota et al. "Erasing Concepts from Diffusion Models", Cornell University, arXiv Preprint, arxiv.org [retrieved 2024-01-24]. Retrieved from the Internet <https://arxiv.org/pdf/2303.07345.pdf>, 03/13/2023, 23 pages, hereinafter referred to as “Gandikota”, in view of Meng et al. "Locating and editing factual associations in gpt." Advances in neural information processing systems 35 (2022): 17359-17372. (Year: 2022), hereinafter referred to as “Meng”, in further view of Zhang et al. "Continuous layout editing of single images with diffusion models." In Computer Graphics Forum, vol. 42, no. 7, p. e14966. 2023, hereinafter referred to as “Zhang”, and in further view of Li et al. (US 20210334708), hereinafter referred to as “Li”.
Regarding claim 10, Gandikota in view of Meng, and in further view of Zhang discloses the method as described in claim 1, Gandikota further discloses receiving a visual attribute input (Gandikota Section 4.1; prompt), and that identifies the visual attribute and wherein the identifying is performed responsive to the receiving (Gandikota Pg. 2429, Section 4; a text encoder T, diffusion model (U-Net) θ∗ and a decoder model D. Fig. 2; The optimization process for erasing undesired visual concepts from the pre-trained diffusion model weights involves using a short text description of the concept as guidance. Pg. 2426, Figure 1; Erased from the model: "Nudity", "Van Gogh", "Car").
Gandikota in view of Meng, and in further view of Zhang does not disclose via a user interface.
In the same art of text-to-image models, Li discloses receiving a visual attribute input via a user interface (Par. 0033; receive user input via a user interface 210. Fig. 2; Car, Fish, Flight).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate Li’s user interface into Gandikota’s diffusion-based text-to- image model and Meng’s corrupted-with-restoration technique. Incorporating a user interface provides the predictable benefits of allowing user interactivity to supply the system with user desired inputs (attributes to edit). Thus, the combinations yield predictable results as receiving input via a user interface is a common practice in the art of interactive generative models.
Regarding claim 11, Gandikota in view of Meng, in further view of Zhang and in further view of Li discloses the method as described in claim 10. Gandikota further discloses wherein the visual attribute input (Gandikota prompt) indicates an object (Gandikota Pg. 2432-2433, Section 5.3 on Object removal and Table 2 class name), style (Gandikota Pg. 2430-2431 and Fig. 4-5 style removal), color, viewpoint, action, or concept (Gandikota Pg. 2429; Section 4; to remove a specific style or concept…c represents the concept to erase…nudity).
The motivation to combine would’ve been the same as that set forth above with respect to claim 10.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JENNY NGAN TRAN whose telephone number is (571)272-6888. The examiner can normally be reached Mon-Thurs 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JENNY N TRAN/Examiner, Art Unit 2615
/ALICIA M HARRINGTON/Supervisory Patent Examiner, Art Unit 2615
Prosecution Insights