DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 5-6, 8, and 14 are rejected under 35 U.S.C. 102(a)(1) as being anticipate by Moayeri et al. (Moayeri, Mazda, et al. Rethinking Artistic Copyright Infringements in the Era of Text-to-Image Generative Models. April 2024, https://doi.org/10.48550/arxiv.2404.08030. Hereinafter “Moayeri”).
Regarding claim 1,
Moayeri teaches:
A computer-implemented method comprising (Moayeri: Page 1 Abstract, “. . . We leverage two complementary methods to perform artistic style classification over image sets, including TagMatch. . .”; Page 31 Section D.5, “TagMatch is surprisingly fast. The longest step by far is computing CLIP embeddings for the reference artworks. This takes us about 5 minutes using one rtx2080 GPU with four CPU cores to embed the 73k training split images using a CLIP ViT-B\16 model. . .”; Page 12-13 Section 5.1, “. . . employ DeepMatch, . . . to quantify the degree to which the unique styles of artists from our dataset are reproduced by generative models prompted to recreate works from these artists . . .”):
determining, by at least one processor, a digital image generated by an image generation neural network (Moayeri: Page 12, Section 5, “. . . images from 3 text-to-image generative models: (i) Stable-Diffusion v1.4; (ii) Stable-Diffusion-v2.0; and (iii) OpenJourney from PromptHero. . . obtain a corresponding set of generated images D′ai, per generative model. . .”);
processing the digital image utilizing an artist prompt prediction neural network trained to detect synthetic images generated from artist prompts comprising indications of specific artists (Moayeri: Page 7 Section 4.2, “DeepMatch consists of a light-weight artist classifier . . . allowing for abstention in case no specific style is recognized with sufficient confidence. . .For our classifier, we train a two layer MLP on top of embeddings from a frozen CLIP ViT-B\16 vision encoder [17] to classify artwork to their respective artist. . .”; Page 12-13 Section 5.1, “. . . employ DeepMatch, . . . to quantify the degree to which the unique styles of artists from our dataset are reproduced by generative models prompted to recreate works from these artists. Recall that DeepMatch predicts an artist from a set of test images by first inferring the artist for each image. . .”; Page 6 Section 4, “. . . DeepMatch and TagMatch, two complementary methods (w.r.t. accuracy and interpretability) that detect artistic styles. . .”; NOTE: Also see Fig. 8 in Page 18 illustrating processing generated art and detect style copying. Moayeri uses DeepMatch, which is also called the Black-Box Detector in section 4.2, which is based on trained MLP (Multilayer Perceptron Neural Network) to process synthetic images generated by generative models prompted to recreate works of specific artist. The recognized artist style and classifications by Deepmatch are the indications that the generated image is prompted indicating a specific artist. Deepmatch is used to classify synthetic images generated from artist prompt that allows abstention if no specific style is recognized based on a confidence threshold. (Syntethic image >> Deepmatch classification >> detects if a specific artist style is recognized/not recognized. Because Deepmatch recognizes an artist style from synthetic imaged prompted to recreate other artists work, based on a confidence level, therefore, it detects a synthetic image generated from artist prompts comprising indications of specific artists. Additionally, Moayeri uses TagMatch as described in page 24 Section D, which is based on CLIP neural network defined in page 33 “CLIP (Contrastive Language-Image Pre-training) is a powerful model that can effectively capture the semantic similarity between text and images”, which generates tags from analyzing synthetic images. Example of tags is disclosed in pages 25-26 such as realism style, animal painting genre, dark colors, etc. ));
and generating, utilizing the artist prompt prediction neural network, a prompt type label for the digital image indicating whether the digital image was generated from an artist prompt comprising an indication of a specific artist (NOTE: As shown in Fig. 8 in page 18, and as discussed above, Deepmatch detects, based on a confidence level, whether a synthetic image is prompted with a specific artistic style, or abstain in case no specific style is recognized. As shown in Fig. 8. The output of the Deepmatch is the prompt type label indicating whether the digital image was generated from an artist prompt because it detects style copying, or not. The output of the TagMatch are also prompt type labels including explanation and attribution of style copying such as disclosed in pages 25-26.).
Regarding claim 5, depending on 1
Moyaeri teaches:
The computer-implemented method of claim 1,
Moyaeri further teaches:
wherein generating the prompt type label for the digital image comprises utilizing a prompt type classification head of the artist prompt prediction neural network (Moyaeri: Page 3 Paragraph 2, “Our tool’s first component– DeepMatch– is a neural network which classifies an artwork to its corresponding artist. DeepMatch implicitly maps each artist to a vector (via the classification head) during training. . .”; Page 7 Section 4.2, “. . . For our classifier, we train a two layer MLP on top of embeddings from a frozen CLIP ViT-B\16 vision encoder [17] to classify artwork to their respective artist. . .”; NOTE: The MLP in DeepMatch acts as the prompt type classification head generating a prompt label for the digital image if it a style corresponding to a respective artist is recognized.)
to generate a prompt type classification from embeddings of the digital image (Moayeri: Page 7 Section 4.2, “. . . For our classifier, we train a two layer MLP on top of embeddings from a frozen CLIP ViT-B\16 vision encoder [17] to classify artwork to their respective artist. . .”; NOTE: The MLP processes the embeddings extracted by CLIP ViT-B\16 vision encoder from the generated digital image to output the classification, which is the prompt type classification.)
Regarding claim 6, depending on 5;
Moyaeri teaches:
The computer-implemented method of claim 5,
further comprising generating a prompt name label for the digital image by utilizing a prompt name classification head of the artist prompt prediction neural network (Moyaeri: Page 12 Section 5 Setup, “. . . quantify the frequency with which generated art prompted to be in the style of a specific artist is matched to that artist; we call this the match rate. Match rate is a percentage over 372 artists, as each artist is either matched correctly or not (i.e. to the wrong artist or no artist at all)”; NOTE: The MLP as discussed in the rejection of claim 5 also generates a prompt name label corresponding to the name of the artist in order to compare and matched to a specific name of an artist. The MLP as discussed in the rejection of claim 5 also acts as the prompt name classification head as it distinguishes artists to their respective artwork.)
to generate a prompt name classification from the embeddings of the digital image (NOTE: As referenced in the rejection of claim 5, the MLP processes the embeddings extracted by CLIP ViT-B\16 vision encoder from the generated digital image to output the name of an artist, which is the prompt name classification)
Regarding claim 8, depending on 1,
Moayeri teaches:
The computer-implemented method of claim 1,
Moayeri further teaches the method of claim 1:
further comprising:
determining the digital image in response to a user query for prompt information about the digital image (Moayeri: Page 18, section A, “. . . a hypothetical artist who is concerned with generative models potentially copying their styles. In figure 8, we outline the general flow of how our tool can be used. The concerned artist would first present a corpus of their works, along with their own name and the titles of each work. Then, ArtSavant would create an easy-to-understand report characterizing the degree to which generative models copy the styles of the artist. The artist can present a set of generated images. . .”; NOTE: The user is the artist concerned of copying their style. They query for prompt information by presenting generated digital images. The system then outputs prompt information if the presented generated digital images are prompted using the artist style that matches the artist. The output of the DeepMatch/TagMatch are the prompt information about the digital image. );
and providing, in response to determining the digital image, the prompt type label for display via a graphical user interface of a client device (NOTE: The report created by Moayeri’s tool providing prompt type label for display via a graphical user interface of a client device is shown in page 19 Fig 9. The report provides prompt type labels such as the matched artistic styles described in the report.).
Regarding claim 14,
Moayeri teaches:
A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising: (Moayeri: page 3 paragraph 3, “. . . the CLIP embeddings are computed and cached,. . .”;page 7 section 4.1, “. . . We provide an easy-to-execute script . . .”; page 31 section D5, “computing CLIP embeddings for the reference artworks. . . using one rtx2080 GPU with four CPU cores. . .”; NOTE: Moayeri’s system uses an rtx2080 which inherently has 8GB of GDDR6 memory coupled with four CPU cores to perform operations such as extracting embeddings. The GDDR6 memory is non transitory CRM because it is soldered within the GPU. The scripts are the operations executed by the processor. A computing system inherently includes a memory to store the scripts, such as a RAM, or the GPU’s memory to cache or store data and GPU instructions or scripts.):
The operations performed by the processor of claim 14 correspond to the methods of using same as claimed in claim 1, and is rejected for the same reasons of anticipation as used above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 2-3. And 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Moayeri in view of Zhou et al. (Zhou, Chao, et al. Plug-in Authorization for Human Content Copyright Protection in Text-to-Image Model. April 24 2024, https://doi.org/10.48550/arxiv.2404.11962, hereinafter “Zhou”)
Regarding claim 2, depending on 1,
Moayeri teaches:
The computer-implemented method of claim 1,
Moayeri further teaches:
further comprising generating, a first batch of images generated from artist prompts (Moayeri: Page 12 Section 5 para 3, “. . . Using these prompts (e.g.“the starry night by Vincent Van Gogh” or “the water lillies by Claude Monet”), we generate images from 3 text-to-image generative models: (i) Stable-Diffusionv1.4; (ii) Stable-Diffusion-v2.0; and (iii) OpenJourney from PromptHero. . .”).
However, Moayeri fails to disclose if they also generate images from non-artistic prompts, and therefore fails to teach generating a second batch of images generated from non-artist prompts.
The analogous art Zhou teaches:
a second batch of images generated from non-artist prompts (Zhou: Page 6, Section 2.2 (3), “After adding such a LoRA component, the base model can generate Picasso-style images even when the prompts do not explicitly mention “Picasso”. Hence, the LoRA represents the Picasso style, and w −wL would give us a non-infringing model, which can be thought of as an analogy of a negative LoRA. However, directly using w−wL as the non-infringing model compromises its ability to generate images with surrounding contexts, e.g., “the painting of a building”, as shown in Figure 5 in Appendix B. This observation motivates us to further tune the LoRA with pairs of images and texts of surrounding contexts... . .”; NOTE: Zhou also generates a first batch of images generated from artist prompts also disclosed in page 6, “the image generated by the infringing model with the prompt “the painting of building by Picasso”. Both batches are used as training data set for a non-infringing model neural network. The prompt “the painting of a building” is an non-artist prompt because it does not specify a particular artist, and is used to generate an image by the non-infringing model, and also used as training data to further tune the neural network).
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Moayeri, and Zhou and include: generating a second batch of images generated from non-artist prompts.
The reason for doing so is “to generate images with surrounding contexts” (Zhou: page 6, section 2.2 (4)).
Although the combination of Moayeri and Zhou teaches generating images from artist and non-artist prompts; Moayeri teaches Deepmatch as an artist prompt prediction neural network; and Zhou teaches using the generated batches of images to as training dataset to train a non-infringing model; the combination still fails to teach using the generated batches of images as training dataset for an artist prompt prediction neural network, such as Moayeri’s DeepMatch (Moayeri’s Deepmatch is only trained on real artwork of the artist), and therefore, the combination fails to teach: generating, for the artist prompt prediction neural network, a training dataset comprising a first batch of images generated from artist prompts and a second batch of images generated from non-artist prompts.
It would have been an obvious design choice between using/not using as training dataset for the artist prediction neural network, the generated images from an artist/non-artist prompts to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention and include: generating, for the artist prompt prediction neural network, a training dataset comprising a first batch of images generated from artist prompts and a second batch of images generated from non-artist prompts.
The reason for doing so is for “facilitating proper credit attribution” and “to reclaim copyright from infringing models”, “incentivizing fair use and providing flexibility in authorization” to address contentions “issue of copyright infringement in images generated by text-to-image models” (Zhou: Page 1, Abstract.)
Regarding claim 3, depending on 2,
The combination of Moayeri, and Zhou teaches:
The computer-implemented method of claim 2, wherein generating the training dataset comprises:
Moayeri further teaches:
accessing, for the first batch of images, a first set of images generated from artist prompts by a first image generation neural network (Moayeri: Page 12 Section 5 para 3, “. . . Using these prompts (e.g.“the starry night by Vincent Van Gogh” or “the water lillies by Claude Monet”), we generate images from 3 text-to-image generative models: (i) Stable-Diffusionv1.4; (ii) Stable-Diffusion-v2.0; and (iii) OpenJourney from PromptHero. . .”; NOTE: The first image generation neural network is Stable Diffusion v1.4);
and accessing, for the first batch of images, a second set of images generated from artist prompts by a second image generation neural network (NOTE: The second generation neural network is either the Stable-Diffusion v.20 or the OpenJourney from PromptHero.).
Regarding claim 15, depending on 14,
Moayeri teaches: The non-transitory computer-readable medium of claim 14,
The limitation: wherein the operations further comprise generating, for the artist prompt prediction neural network, a training dataset comprising a first batch of images generated from artist prompts, a second batch of images generated from style prompts, have similar scope to the methods recited in claim 2: “further comprising generating, for the artist prompt prediction neural network, a training dataset comprising a first batch of images generated from artist prompts and a second batch of images generated from non-artist prompts” (NOTE: The style prompts of claim 15 is included in the non-artist prompt discussed in the rejection of claim 2, also referencing Zhou page 6 paragraph 2, “. . . the base model can generate Picasso-style images even when the prompts do not explicitly mention “Picasso”. . .” and page 8 section 3.2, “. . . compare images generated by the base model and the non infringing model using target and surrounding style prompts. . .”),
Zhou further teaches:
and a third batch of images generated from content prompts (Zhou: page 6 paragraph 4, “. . . This step involves fine-tuning the LoRA component with images and textual prompts of surrounding contexts, e.g., “the painting of a building”. To curate the dataset, we randomly generate images with the base model using the contextual prompt “the painting of a building”. . .”; The prompt “the painting of a building” is the content prompt to generate an image used for training the LoRA).
It would have been an obvious design choice between using/not using as training dataset for the artist prediction neural network, the generated images from an artist/non-artist prompts to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention and include: generating, for the artist prompt prediction neural network, a training dataset comprising a first batch of images generated from artist prompts, a second batch of images generated from style prompts, and a third batch of images generated from content prompts..
The reason for doing so is for “facilitating proper credit attribution” and “to reclaim copyright from infringing models”, “incentivizing fair use and providing flexibility in authorization” to address contentions “issue of copyright infringement in images generated by text-to-image models” (Zhou: Page 1, Abstract.)
Regarding claim 16, depending on 15,
The combination of Moayeri, and Zhou teaches:
The non-transitory computer-readable medium of claim 15,
Moayeri further teaches:
wherein generating the training dataset comprises:
accessing, for the first batch of images, a first set of images generated from artist prompts by a first image generation neural network (Moayeri: Page 12 Section 5 para 3, “. . . Using these prompts (e.g.“the starry night by Vincent Van Gogh” or “the water lillies by Claude Monet”), we generate images from 3 text-to-image generative models: (i) Stable-Diffusionv1.4; (ii) Stable-Diffusion-v2.0; and (iii) OpenJourney from PromptHero. . .”; NOTE: The first image generation neural network is Stable Diffusion v1.4);
Although Moayeri teaches generating images using the same artist prompt using all 3 disclosed image generation neural networks, Moayeri fails to access a second set of images generated from style prompts for the second batch of images.
Although Zhou teaches accessing a second set of images generated from style prompts for the second batch of images used as a training dataset for LoRA based model, Zhou fails to teach generating images using style prompts using same image generation neural network the same as in Moayeri’s disclosure, therefore failing to teach: accessing, for the second batch of images, a second set of images generated from style prompts by the first image generation neural network.
It would have been obvious an obvious design choice (between using the same/different image generation neural network to generate images based on different prompt styles including an artist prompt as taught by Moyaeri, and style prompt as taught by Zhou) to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Moyaeri, and Zhou to include: accessing, for the second batch of images, a second set of images generated from style prompts by the first image generation neural network, having predictable results, generated images using style prompt of Zhou, using Moayeri’s 3 text-to-image generative models. (NOTE: The combination results in both the artist and style prompts are used by the same image generator, for example, stable diffusion 1.4 disclosed by Moayeri.)
The reason for doing so is for “facilitating proper credit attribution” and “to reclaim copyright from infringing models”, “incentivizing fair use and providing flexibility in authorization” to address contentions “issue of copyright infringement in images generated by text-to-image models” (Zhou: Page 1, Abstract.)
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Moyaeri in view of Mahajan et al. (Mahajan, Shweta, et al. “Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models., December 2023. https://arxiv.org/abs/2312.12416, hereinafter “Mahajan”)
Regarding claim 4, depending on 1,
Moyaeri teaches:
The computer-implemented method of claim 1,
further comprising adjusting parameters of the artist prompt prediction neural network (Moyaeri: Page 7 Section 4.2, “For our classifier, we train a two layer MLP on top of embeddings from a frozen CLIP ViT-B\16 vision encoder [17] to classify artwork to their respective artist, using a train split containing 80% of our dataset. We employ weighted sampling to account for class imbalance”; NOTE: Moyaeri’s DeepMatch is a trained MLP neural network similar to the applicant’s MLP included in the image source classification disclosed in paragraph 46. Training a neural network with a data set inherently adjusts the parameters of that neural network.)
comparing the prompt type label for the digital image with a ground truth prompt type label (Moyaeri: Page 7 Section 4.1 “. . . compare a test set of images to our reference corpus so to detect if any of the reference styles reappear . . .”; NOTE: In Fig. 18 of Moyaeri, It shows a detection output labeling if the comparison of the generated image (digital image) matches a style of a specific artist (ground-truth). Specific data must have labels in order to make comparison such as tags or style).
However, Moyaeri fails to disclose, if the MLP in DeepMatch used for classification is trained to reduce a measure of loss.
The analogous art Mahajan teaches:
adjusting parameters of a neural network to reduce a measure of loss (Mahajan: Page 5 col 1 paragraph 4, “. . . since we are optimizing for the initial layers of a very deep neural network, the gradients are very small. By optimizing for larger values of loss, we get better gradients. . .”; NOTE: Optimizing a neural network constitutes to adjusting parameters of a neural network, and optimizing larger values of loss to get better gradients is reducing a measure of loss.)
determined by comparing the prompt type label for the digital image with a ground truth prompt type label (Mahajan: Page 7 Col 2 Quality of the prompts, “. . . We compare the contextual similarity between the prompts inverted for the COCO dataset and the ground-truth annotations (captions) of the corresponding images, with the BertScore. . .”)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Moayeri, and Mahajan to apply neural network optimization to Moyaeri’s DeepMatch: adjusting parameters of the artist prediction neural network to reduce a measure of loss comparing the prompt type label for the digital image with a ground truth prompt type label.
The reason for doing so is “to obtain interpretable language prompts directly” (Mahajan: Page 1 Abstract).
Claims 7, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Moayeri in view of Liu et al. (Liu, Fengyuan, et al. “Which Model Generated This Image? A Model-Agnostic Approach for Origin Attribution.” Computer Vision - ECCV 2024, edited by Stefan Roth et al., vol. 15120, Springer, 2024, pp. 282–301, https://doi.org/10.1007/978-3-031-73033-7_16., hereinafter “Liu”)
Regarding claim 7, depending on 5,
Moayeri teaches:
The computer-implemented method of claim 5,
Although Moayeri teaches an MLP that generates prompt name/type label acting as a prompt name/type classification head of the artist prompt prediction neural network from the embeddings of the digital image, Moayeri’s MLP does not identify and label the source of an image, and therefore fails to teach: generating an image source label for the digital image by utilizing an image source classification head of the artist prompt prediction neural network to generate an image source classification from the embeddings of the digital image.
The analogous art Liu teaches:
generating an image source label for the digital image by utilizing an image source classification head the neural network (Liu: Page 2 paragraph 1-4, “. . . identify DALL·E-3 as the origin model of the query image. . . a CLIP-based frame . . . identify the model that generated the few images as the origin model of the given image. . .”; Page 6 section 3.3, “to determine if the origin of an image can be attributed to ProGAN [25], Stable Diffusion [46], or Vector Quantized Diffusion [19], we can employ three one-class classifiers corresponding to these models for classification. Given a set of trained K one-class classifiers. . .”; NOTE: Liu’s CLIP Based neural network acts a an image source classification head because it identifies the origin model that generated an image. The generated image source label is the identified model, such as “DALL E-3” )
to generate an image source classification from the embeddings of the digital image (Liu: Page 13 section 4.5, “We utilize the embeddings from CLIP’s image encoder for visualization.”; NOTE: Similar to Moayeri, Liu uses a CLIP Based framework to extract embeddings from a digital image to generate image source labels.).
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Moayeri, and Liu, implementing an image source classification head as taught by Liu to Moayeri’s ClIP Based prompt/name classifier to include: generating an image source label for the digital image by utilizing an image source classification head of the artist prompt prediction neural network to generate an image source classification from the embeddings of the digital image.
The reason for doing so is because “it is important to identify the origin model that generates them”, “to prevent the misuse of generated images”, and “to check if a given image is generated by the source model” (Liu: Page 1 Abstract).
Regarding CRM claim 18, CRM claim 18 is drawn to the CRM corresponding to the methods of using same as claimed in claims 5-7. Therefore, CRM claim 18 corresponds to the methods of claims 5-7, and is rejected for the same reasons of obviousness as used above.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Moayeri in view of Liu and Mahajan.
Regarding claim 19, depending on 18,
The combination of Moayeri, and Liu teaches:
The non-transitory computer-readable medium of claim 18,
wherein the operations further comprise adjusting parameters of the artist prompt prediction neural network by:
adjusting prompt type parameters of the prompt type classification head;
adjusting prompt name parameters of the prompt name classification head; and
adjusting image source parameters of the image source classification head. (NOTE: The parameters of the prompt type classification head, the prompt name classification head, and image source classification head have their parameters inherently adjusted once training is initiated. The training inherently adjusts the parameters of the classification head based on the embeddings extracted from the digital images.)
However, the combination of Moayeri, and Liu fails to teach: adjusting parameters of the artist prompt prediction neural network to reduce a measure of loss.
The analogous art Mahajan teaches:
adjusting parameters of a neural network to reduce a measure of loss (Mahajan: Page 5 col 1 paragraph 4, “. . . since we are optimizing for the initial layers of a very deep neural network, the gradients are very small. By optimizing for larger values of loss, we get better gradients. . .”; page 5 col 2, “. . . the loss is computed with respect to the embeddings in the feasible set i.e. the embeddings representing the vocabulary. During optimization, the variable ˆe is updated. . .”; NOTE: Also see page 4 section 3.2 Equation 2, the optimization formula. Optimizing a neural network constitutes to adjusting parameters of a neural network, and optimizing larger values of loss to get better gradients is reducing a measure of loss.)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Moayeri, Liu, and Mahajan to implement the loss optimization of Mahajan to the classification head neural networks, such that: wherein the operations further comprise adjusting parameters of the artist prompt prediction neural network to reduce a measure of loss by: adjusting prompt type parameters of the prompt type classification head; adjusting prompt name parameters of the prompt name classification head; and adjusting image source parameters of the image source classification head.
The reason for doing so is “to obtain interpretable language prompts directly” (Mahajan: Page 1 Abstract).
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Moayeri in view of Liu and Mahajan, further in view of Cui et al. (US 20240185578 A1, hereinafter “Cui”).
Regarding claim 20, depending on 19,
The combination of Moayeri, Liu, and Mahajan teaches:
The non-transitory computer-readable medium of claim 19, wherein the operations further comprise adjusting the parameters of the artist prompt prediction neural network to reduce the measure of loss,
However, the combination of Moayeri, Liu, and Mahajan fails to teach adjusting the parameters: by adjusting encoder parameters of an image encoder of the artist prompt prediction neural network. (NOTE: The CLIP model, which is implemented in the DeepMatch is the image encoder of Moayeri extracting embeddings from a digital image. Moayeri does not teach updating parameters for the CLIP model)
The analogous art Cui teaches:
adjusting encoder parameters of a neural network to reduce the measure of loss (Cui: ¶61, “. . . the image encoder 310 and the image decoder 320 may be configured as machine learning models or neural networks that are suitable for processing visual data. . .”; ¶57, “. . . training of the image encoder 310 may include updating the parameter values of the image encoder 310 in such a direction that the text error (e.g., the corresponding loss function) is interactively decreased to an expected or minimum value. . . ”).
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Moayeri, Liu, Mahajan, and Cui to include: wherein the operations further comprise adjusting the parameters of the artist prompt prediction neural network to reduce the measure of loss by adjusting encoder parameters of an image encoder of the artist prompt prediction neural network.
The reason for doing so is “to accurately describe the visual information presented by the sample image” (Cui: ¶54).
Claims 9-11, and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Moayeri in view of Wang et al. (Wang, Zhenting, et al. “DIAGNOSIS: Detecting Unauthorized Data Usages in Text-to-Image Diffusion Models.”, Feb 2024, https://arxiv.org/abs/2307.03108v2, hereinafter “Wang”)
Regarding claim 9,
Moayeri teaches:
A system comprising: one or more memory devices; and one or more processors coupled to the one or more memory devices that cause the system to perform operations comprising (Moayeri: page 31 section D5, “computing CLIP embeddings for the reference artworks. . . using one rtx2080 GPU with four CPU cores. . .”; NOTE: Moayeri’s system uses an rtx2080 which inherently has 8GB of GDDR6 memory coupled with four CPU cores to perform operations such as extracting embeddings.):
determining a digital image generated by an image generation neural network (Moayeri: Page 12, Section 5, “. . . images from 3 text-to-image generative models: (i) Stable-Diffusion v1.4; (ii) Stable-Diffusion-v2.0; and (iii) OpenJourney from PromptHero. . . obtain a corresponding set of generated images D′ai, per generative model. . .”);
However, Moayeri fails to teach: processing the digital image utilizing an artist customized model prediction neural network trained to detect synthetic images generated by one or more neural networks customized with a set of artist-specific training data; and generating, utilizing the artist customized model prediction neural network, a customized model label for the digital image indicating whether the digital image was generated by a customized image generation neural network finetuned with artist-specific image data (NOTE: As discussed in independent claim 1, Moayeri’s method only generates labels corresponding to classification of artistic styles to match an artist, it does not use an artist customized model prediction neural network to generate a label if the generated image was generated using a customized image generation neural network.).
The analogous art Wang also teaches:
determining a digital image generated by an image generation neural network (Wang: Page 2: Paragraph 3, “. . . More specifically, when a set of protected images are uploaded to the internet, it will be processed by a specific function (called signal function) that is stealthy to humans but can be captured and memorized for diffusion models. Therefore, after the models are trained or fine-tuned on the “coated images” (i.e., images processed by the signal function), it will memorize the added signal function so that the unauthorized data usages can be detected by using a binary classifier (called signal classifier) to analyze if the given model has the memorization on the signal function (i.e., if the images generated by the model contains the signal function). . .”; NOTE: The image generation neural network is the diffusion models trained or fine-tuned to generate a synthetic digital image. A signal classifier is used to detect a signal function to determine if a digital image generated by an image generation neural network)
Wang further teaches:
processing the digital image utilizing an artist customized model prediction neural network trained to detect synthetic images generated by one or more neural networks customized with a set of artist-specific training data (Wang: Page 1 Abstract: “. . . when a model trainer collects a set of images created by a particular artist and attempts to train a model capable of generating similar images without obtaining permission and giving credit to the artist. To address this issue, we propose a method for detecting such unauthorized data usage by planting the injected memorization into the text-to-image diffusion models trained on the protected dataset. . . ”; Page 5 section 3.3, “. . . To trace the unauthorized data usages, we train a binary classifier Cθ to distinguish if the image generated by the inspected model contains the signal function S or not. . . we use the ResNet18 (He et al., 2016) model pretrained on the ImageNet. . . dataset and fine-tuned by the procedure described above as the signal classifier.”; NOTE: Wang processes a digital image using a ResNet18 trained on the Imagenet as the artist customized model predicition neural network. It is trained to detect signal function, via a binary classifier, in a synthetic generated image by a diffusion model indicating that the synthetic generated image is generated by one or more neural network customized with a particular artist set of images as training data.);
and generating, utilizing the artist customized model prediction neural network, a customized model label for the digital image indicating whether the digital image was generated by a customized image generation neural network finetuned with artist-specific image data (Wang: Page 4 Section 3.1 Protector’s Goal, “Given an text-to-image diffusion model M and a set of protected images D, the goal of the protector is to detect whether the protected images are used as (part of) the training data for the pre-training or fine-tuning phase of the model. Formally, the goal can be written as constructing an inference algorithm A : M → {0,1} that receives a model M as the input, and returns the inference result (i.e., 0 denotes the model did not use the unauthorized data , and 1 denotes unauthorized usage is detected).
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Moayeri, and Wang; and include: determining a digital image generated by an image generation neural network; processing the digital image utilizing an artist customized model prediction neural network trained to detect synthetic images generated by one or more neural networks customized with a set of artist-specific training data; and generating, utilizing the artist customized model prediction neural network, a customized model label for the digital image indicating whether the digital image was generated by a customized image generation neural network finetuned with artist-specific image data.
The reason for doing so is to “detect models that had illegally utilized the unauthorized data” (Wang: Page 1 Abstract.).
Regarding claim 10, depending on 9,
The combination of Moyaeri, and Wang teaches:
The system of claim 9,
Wang further teaches:
wherein generating the customized model label for the digital image comprises utilizing a customized model prediction head of the artist customized model prediction neural network to generate a source classification indicating that the digital image was generated by the customized image generation neural network (NOTE: As discussed in the rejection of claim 9, Wang’s signal classifier or the binary classifier is the customized model prediction head that detects a signal function to determine if a digital image was generated by the customized image generation neural network trained using images created by a particular artist.).
Regarding claim 11, depending on 9,
The combination of Moyaeri, and Wang teaches:
The system of claim 9,
However, Moyaeri fails to teach: wherein the operations further comprise adjusting parameters of the artist customized model prediction neural network to reduce a cross-entropy loss based on the customized model label and a ground truth customized model label.
Wang further teaches:
wherein the operations further comprise adjusting parameters of the artist customized model prediction neural network to reduce a cross-entropy loss based on the customized model label and a ground truth customized model label (Wang: Page 5, Section 3.3, “. . . To trace the unauthorized data usages, we train a binary classifier Cθ to distinguish if the image generated by the inspected model contains the signal function S or not. The training process of the binary classifier Cθ is formalized in Eq. 2, where yn is the label denoting normal samples and ys is the label standing for the samples processed by signal function S. D is the set of protected images and L is the cross-entropy loss function. . .”; NOTE: Training the binary classifier using a set of images constitute in adjusting the parameters of the artist customized model prediction neural network. In reference to equation 2, “Cθ = argminθ [L(Cθ(x),yn) + L(Cθ(S(x)),ys)], x ∈ D”; the Cθ(x)/Cθ(s(x)) correspond the customized model label 0/1 indicating if the input image x is generated using a customized model. The Yn and Ys labels correspond to the ground truth customized model label denoting normal samples and samples processed by signal function, meaning processed by a customized model trained using an unauthorized particular artist image. L is the cross-entropy loss function. The reduced loss function is represented by argmin.)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Moyaeri, and Wang to include: wherein the operations further comprise adjusting parameters of the artist customized model prediction neural network to reduce a cross-entropy loss based on the customized model label and a ground truth customized model label.
The reason for doing so is to train a classifier such as Moayeri’s DeepMatch neural network improving its classification capabilities (Wang: Page 5 Section 3.3).
Regarding claim 13, depending on 9,
The combination of Moayeri, and Wang teaches:
The system of claim 9,
Moayeri further teaches:
wherein the operations further comprise:
determining the digital image in response to a user query for customization information about the digital image (Moayeri: Page 18, section A, “. . . a hypothetical artist who is concerned with generative models potentially copying their styles. In figure 8, we outline the general flow of how our tool can be used. The concerned artist would first present a corpus of their works, along with their own name and the titles of each work. Then, ArtSavant would create an easy-to-understand report characterizing the degree to which generative models copy the styles of the artist. The artist can present a set of generated images. . .”; NOTE: The user is the artist concerned of copying their style. They query for customization information by presenting a set generated digital images generated by specific models. The system then outputs customization information if the presented generated digital images are prompted using the artist style that matches the artist. The output of the DeepMatch/TagMatch are the customization prompt information about the digital image because it customizes the prompt to specific attributions.);
Although Moayeri provides a report including customization information as shown in Fig. 9 in page 19, it does not include the customized model label. Although Wang teaches generating customized model label denoting a 0/1 as discussed in the rejection of claim 9, Wang does not generate a report via a GUI of a client device.
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Moayeri, and Wang, and include: providing, in response to determining the digital image, the customized model label for display via a graphical user interface of a client device.
The reason for doing so is to easily visualize if an image generated by a model used protected images to generate them and “detect models that had illegally utilized the unauthorized data” (Wang: Page 1 Abstract).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Moayeri in view of Wang further in view of Zhou.
Regarding claim 12, depending on 9,
The combination of Moayeri, and Wang teaches:
The system of claim 9,
However, Moayeri fails to teach: generating, for the artist customized model prediction neural network, a training dataset by: accessing a first batch of images generated by a first image generation neural network customized with artist-specific training data; and accessing a second batch of images generated by a second image generation neural network trained with artist-agnostic training data
Wang teaches
wherein the operations further comprise generating, for the artist customized model prediction neural network, a training dataset by:
accessing a first batch of images generated by a first image generation neural network customized with artist-specific training data (Wang: page 5 section 3.3, “. . . To trace the unauthorized data usages, we train a binary classifier Cθ to distinguish if the image generated by the inspected model contains the signal function S or not . . .”; NOTE: As discussed in the rejection of claim 9, the binary classifier accesses the first batch of images generated by a first image generation neural network customized with artist-specific training data (images generated by a model trained using set of images created by a particular artist to create similar images) in order to detect unauthorized use. In reference to equation 2, the images used in detecting the unauthorized usage is used as a training data denoted inside the cross entropy loss function.)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Moayeri, and Wang to include: generating, for the artist customized model prediction neural network, a training dataset by: accessing a first batch of images generated by a first image generation neural network customized with artist-specific training data.
The reason for doing so is to “train a binary classifier Cθ to distinguish if the image generated by the inspected model contains the signal function S or not”, “to trace the unauthorized data usages” (Wang: Page 5 Section 3.3)
However, the combination of Moayeri, and Wang still fails to teach: accessing a second batch of images generated by a second image generation neural network trained with artist-agnostic training data.
The analogous art Zhou teaches:
accessing a second batch of images generated by a second image generation neural network trained with artist-agnostic training data (Zhou: Page 6, Section 2.2 (3), “After adding such a LoRA component, the base model can generate Picasso-style images even when the prompts do not explicitly mention “Picasso”. Hence, the LoRA represents the Picasso style, and w −wL would give us a non-infringing model, which can be thought of as an analogy of a negative LoRA. However, directly using w−wL as the non-infringing model compromises its ability to generate images with surrounding contexts, e.g., “the painting of a building”, as shown in Figure 5 in Appendix B. This observation motivates us to further tune the LoRA with pairs of images and texts of surrounding contexts... . .”; NOTE: Zhou also generates a first batch of images generated from artist prompts also disclosed in page 6, “the image generated by the infringing model with the prompt “the painting of building by Picasso”. Both batches are used as training data set for a non-infringing model neural network. The prompt “the painting of a building” is an artist-agnostic prompt because it does not specify a particular artist, and is used to generate an image by the non-infringing model, and also used as training data to further tune the neural network.).
The reason for doing so is for “facilitating proper credit attribution” and “to reclaim copyright from infringing models”, “incentivizing fair use and providing flexibility in authorization” to address contentions “issue of copyright infringement in images generated by text-to-image models” (Zhou: Page 1, Abstract.)
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Moayeri in view of Mahajan further in view of Wang.
Regarding claim 17, depending on 14,
Moayeri teaches:
The non-transitory computer-readable medium of claim 14,
Moayeri further teaches:
further comprise adjusting parameters of the artist prompt prediction neural network (Moyaeri: Page 7 Section 4.2, “For our classifier, we train a two layer MLP on top of embeddings from a frozen CLIP ViT-B\16 vision encoder [17] to classify artwork to their respective artist, using a train split containing 80% of our dataset. We employ weighted sampling to account for class imbalance”; NOTE: Moyaeri’s DeepMatch is a trained MLP neural network similar to the applicant’s MLP included in the image source classification disclosed in paragraph 46. Training a neural network with a data set inherently adjusts the parameters of that neural network.)
comparing the prompt type label for the digital image with a ground truth prompt type label (Moyaeri: Page 7 Section 4.1 “. . . compare a test set of images to our reference corpus so to detect if any of the reference styles reappear . . .”; NOTE: In Fig. 18 of Moyaeri, It shows a detection output labeling if the comparison of the generated image (digital image) matches a style of a specific artist (ground-truth). Specific data must have labels in order to make comparison such as tags or style).
However, Moyaeri fails to disclose, if the MLP in DeepMatch used for classification is trained to reduce a cross-entropy loss determined by comparing the prompt type label for the digital image with a ground truth prompt type label.
The analogous art Mahajan teaches:
adjusting parameters of a neural network to reduce a measure of loss (Mahajan: Page 5 col 1 paragraph 4, “. . . since we are optimizing for the initial layers of a very deep neural network, the gradients are very small. By optimizing for larger values of loss, we get better gradients. . .”; NOTE: Optimizing a neural network constitutes to adjusting parameters of a neural network, and optimizing larger values of loss to get better gradients is reducing a measure of loss.)
determined by comparing the prompt type label for the digital image with a ground truth prompt type label (Mahajan: Page 7 Col 2 Quality of the prompts, “. . . We compare the contextual similarity between the prompts inverted for the COCO dataset and the ground-truth annotations (captions) of the corresponding images, with the BertScore. . .”)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Moayeri, and Mahajan to apply neural network optimization to Moyaeri’s DeepMatch: adjusting parameters of the artist prediction neural network to reduce a measure of loss comparing the prompt type label for the digital image with a ground truth prompt type label. (NOTE: Also see the rejection of claim 4)
The reason for doing so is “to obtain interpretable language prompts directly” (Mahajan: Page 1 Abstract).
However still, the loss function of Mahajan is not a cross-entropy loss function, and therefore fails to teach training a neural network to reduce a cross-entropy loss.
The analogous art Wang teaches:
Training a neural network with a classifier with an implementation to reduce a cross-entropy loss represented by argmin in Equation 2 described in page 5. (Wang: Page 5 section 3.3, “we train a binary classifier Cθ to distinguish if the image generated by the inspected model contains the signal function S or not. The training process of the binary classifier Cθ is formalized in Eq. 2, where yn is the label denoting normal samples and ys is the label standing for the samples processed by signal function S. D is the set of protected images and L is the cross-entropy loss function.”; NOTE: Also see the rejection of claim 11.)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Moayeri, Mahajan, and Wang, and reduce a cross-entropy loss function as taught by Wang, to train the DeepMatch MLP of Moayeri, determined by comparing the prompt type label for the digital image with a ground truth prompt type label as taught by Mahajan reducing a measure of loss.
The reason for doing so “to effectively detect unauthorized data usages in text-to-image diffusion models” and “strengthen the responsible development of the text-to-image diffusionmodels,andsafeguardtheintellectualpropertyofthevaluabletrainingdata”. (Wang: Page 9 Ethic Statement, Conclusion).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PATRICK GALERA whose telephone number is (571)272-5070. The examiner can normally be reached Mon-Fri 0800-1700 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PATRICK P GALERA/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights