DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to pending claims 1-20 filed 8/8/2023.
Claim Rejections - 35 USC § 103
The following is a quotation of 3a5 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-17, 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Balan ("EKILA: Synthetic media provenance and attribution for generative art", published 4/10/2023) in view of Lyons (US 20240362968 A1).
For claim 1, Balan discloses: a method comprising:
determining an input provided to a generative artificial intelligence to generate an output (§4.2 goes over various exemplary implementations of image attribution and the models used, i.e., for generation, a Latent Diffusion Model (LDM) is trained on various image corpuses including LAION-400M (§4.1 ¶1), IPF-STOCK (¶2); further, product names or image captions are sampled for each data set, respectively, to form generate queries for experimentation);
processing the input to determine (§4.2 ¶1-2: text input such as product names, image captions are parsed for image generation by the LDM):
a type of content to generate (ibid: product name and image caption constitutes a type);
a content description (ibid: product names and image caption contain content descriptions); and
embedding the input into a shared language-image space using an encoder to create an input embedding (§4.2 “Attribution Baselines”. §4.3 ¶1: contemplates using the ViT-CLIP encoder, the ViT-CLIP encoder being a shared image and language encoder, see Radford fig.1);
determining, by the one or more processors, a creator description comprising a creator-based embedding associated with individual creators identified by the one or more creator identifiers (fig.2, §3.1.1-3.1.2 discloses using manifests on training image data to identify creator information including NFT information, hence, §4.3 disclosing generating patch or image embeddings via ViT-CLIP constitutes determining creator-based embeddings associated with creators identified by creator identifiers);
performing, by the one or more processors, a comparison of the input embedding to the creator-based embedding associated with individual creators (§4.3 discloses cluster or distance analysis to generate K closest neighbors in the individual creator embeddings and hence attributions in the embedding space);
determining, by the one or more processors and based on the comparison, a distance between a creator embedding of the individual creators and the input embedding (ibid: distances are determined in order to determine ranking and closest neighbors);
determining, by the one or more processors, one or more creator attributions based on the distance of an amount of the embedding of the individual creators in the input embedding (ibid: various creator attributions (see fig.2, §3.1.1) are determined for top K embeddings based on the distance);
determining, by the one or more processors, a creator attribution vector that includes the one or more creator attributions (§4.1: “Apportioning Credit via Patch-Based Attribution”, fig.5 discloses generating a creator attribution vector comprising normalized weightings of the various creator attributions); and
initiating providing compensation to one or more creators based on the creator attribution vector (§4.1 last ¶: compensating creators accordingly).
Balan does not disclose: wherein the steps are performed by one or more processors (fig.2, 0025 gives hardware overview, including processor, memory); wherein the processing comprises parsing (fig.4, 0048-54: gathering various prompt related data in order to generate a prompt, including user interface input (0054, 0057), hence, analyzing data into components to generate prompt tokens); wherein the input includes one or more creator identifiers (0064: famous artist or musician names and styles).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Balan by incorporating the prompt inputting technique of Lyons. Both concern the art of generative text and image ai, and the incorporation would have, according to Lyons, improve user entertainment experience by producing contextually relevant and ad hoc art (0005-6).
For claim 2, Balan modified by Lyons discloses the method of claim 1, as described above. Balan further discloses: wherein the generative artificial intelligence comprises:
a latent diffusion model (§4.2 ¶1);
a generative adversarial network;
a generative pre-trained transformer;
a variational autoencoders;
a multimodal model (LDM is a multimodal text-image model);
or any combination thereof.
For claim 3, Balan modified by Lyons discloses the method of claim 1, as described above. Balan further discloses: selecting a particular creator of the one or more creators (§3.1.1 discloses selecting works of a creator for purposes of embedding creator manifests, etc., hence, selecting creators to associated with the manifests);
performing, using a neural network, an analysis of content items created by the particular creator (§4.2 ¶1, §4.3 ¶133 performing, using ViT-CLIP neural network, embeddings as analytics of training data including those created by the creator);
determining, based on the analysis, a plurality of captions describing the content items (ibid: as CLIP is a language-image embedding space, associated captions for the content items are determine for training, see Radford fig.1);
creating, based on the plurality of captions, a particular creator description (Radford fig.1: text embeddings are associated with the particular creator in the shared image text space, hence, various respective particular creator descriptions for each creator-tagged work are created); and
associating the particular creator description with the particular creator (Balan fig.4: embedding are associated with the particular creator-tagged works via the embedding space for attribution queries, see fig.4).
For claim 4, Balan modified by Lyons discloses the method of claim 3, as described above. Balan further discloses: the neural network is implemented using a Contrastive Language Image Pretraining encoder (§4.2 ¶1, §4.3: ViT-CLIP; and
the encoder comprises a transformer neural network (ViT-CLIP uses a vision transformer (ViT), see Radford §2.4).
For claim 5, Balan modified by Lyons discloses the method of claim 1, as described above. Balan further discloses: wherein the type of content comprises:
a digital image having an appearance of a work of art (fig.4, §4.2);
a digital visual image (fig.4, §4.2 discloses visual image various datasets);
a digital text-based book;
a digital music composition;
a digital video; or
any combination thereof.
For claim 6, Balan modified by Lyons discloses the method of claim 1, as described above. Balan further discloses: wherein the distance comprises:
a cosine similarity (Radford §2.2 ¶3 contemplates use of cosine similarity for CLIP embedding metric),
a contrastive learning encoding distance (ibid: CLIP uses a contrastive learning distance);
a simple matching coefficient,
a Hamming distance,
a Jaccard index,
an Orchini similarity,
a Sorensen-Dice coefficient,
a Tanimoto distance,
a Tucker coefficient of congruence,
a Tversky index, or
any combination thereof.
For claim 7, Balan discloses: determining an input provided to a generative artificial intelligence to generate an output (§4.2 goes over various exemplary implementations of image attribution and the models used, i.e., for generation, a Latent Diffusion Model (LDM) is trained on various image corpuses including LAION-400M (§4.1 ¶1), IPF-STOCK (¶2); further, product names or image captions are sampled for each data set, respectively, to form generate queries for experimentation);
processing the input to determine (§4.2 ¶1-2: text input such as product names, image captions are parsed for image generation by the LDM):
a type of content to generate (ibid: product name and image caption constitutes a type);
a content description (ibid: product names and image caption contain content descriptions); and
embedding the input into a shared language-image space using an encoder to create an input embedding (§4.2 “Attribution Baselines”. §4.3 ¶1: contemplates using the ViT-CLIP encoder, the ViT-CLIP encoder being a shared image and language encoder, see Radford fig.1);
determining a creator description comprising a creator-based embedding associated with individual creators identified by the one or more creator identifiers (fig.2, §3.1.1-3.1.2 discloses using manifests on training image data to identify creator information including NFT information, hence, §4.3 disclosing generating patch or image embeddings via ViT-CLIP constitutes determining creator-based embeddings associated with creators identified by creator identifiers);
performing a comparison of the input embedding to the creator-based embedding associated with individual creators (§4.3 discloses cluster or distance analysis to generate K closest neighbors in the individual creator embeddings and hence attributions in the embedding space);
determining based on the comparison, a distance of an amount of an embedding of the individual creators in the input embedding (ibid: distances are determined in order to determine ranking and closest neighbors; such as similarity or cosine distance, these distance being projection distances, hence, an amount of an embedding in the input embedding, see Radford §2.2 ¶3 as used in the CLIP contrastive learning embedding);
determining, by the one or more processors, one or more creator attributions based on the distance of an amount of the embedding of the individual creators in the input embedding (ibid: various creator attributions (see fig.2, §3.1.1) are determined for top K embeddings based on the distance);
determining, by the one or more processors, a creator attribution vector that includes the one or more creator attributions (§4.1: “Apportioning Credit via Patch-Based Attribution”, fig.5 discloses generating a creator attribution vector comprising normalized weightings of the various creator attributions); and
initiating providing compensation to one or more creators based on the creator attribution vector (§4.1 last ¶: compensating creators accordingly).
Balan does not disclose: a server comprising:
one or more processors;
a non-transitory memory device to store instructions executable by the one or more processors to perform operations comprising:
wherein the processing comprises parsing; wherein the input includes one or more creator identifiers.
Lyons discloses: a server (0045) comprising:
one or more processors; a non-transitory memory device to store instructions executable by the one or more processors to perform operations comprising: (fig.2, 0025 gives hardware overview, including processor, non-transitory memory);
wherein the processing comprises parsing (fig.4, 0048-54: gathering various prompt related data in order to generate a prompt, including user interface input (0054, 0057), hence, analyzing data into components to generate prompt tokens);
wherein the input includes one or more creator identifiers (0064: famous artist or musician names and styles).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Balan by incorporating the prompt inputting technique of Lyons. Both concern the art of generative text and image ai, and the incorporation would have, according to Lyons, improve user entertainment experience by producing contextually relevant and ad hoc art (0005-6).
For claim 11, Balan modified by Lyons discloses the server of claim 7, as described above. Balan further discloses: the one or more creators comprise one or more artists (Balan §3.1.1 contemplates creators of images, such as shown in fig.4, hence, artists, photographers);
the one or more creators comprise one or more authors;
the one or more creators comprise one or more musicians;
the one or more creators comprise one or more visual content creators (ibid);
or any combination thereof.
For claim 12, Balan modified by Lyons discloses the server of claim 7, as described above. Balan further discloses: wherein the content description comprises:
a noun comprising a name of a living creature, an object, a place, or any combination thereof (§4.2 ¶1-2 contemplates product names, hence, name of objects); and
zero or more adjectives to qualify the noun (ibid: image captions include adjectives).
For claim 17, Balan modified by Lyons discloses the server of claim 14, as described above. Balan further discloses: wherein: the type of content comprises a digital image having an appearance of a work of art and the one or more creators comprise one or more artists (Balan §3.1.1 contemplates creators of images, such as shown in fig.4, hence, artists, photographers).
For claim 19, Balan discloses the server of claim 14, as described above. Balan modified by Lyons further discloses: wherein: the type of content comprises a digital music composition and the one or more creators comprise one or more musicians (0053, 0064, 0076).
For claim 20, Balan modified by Lyons discloses the server of claim 14, as described above. Balan further discloses: wherein: the type of content comprises visual content and the one or more creators comprise one or more visual content creators (Balan §3.1.1 contemplates creators of images, such as shown in fig.4, hence, visual content creators).
The remaining claims 8-10, 13-16 recite analogous systems and computer readable media and are hence rejected for the same reason.
Claim(s) 18 are rejected under 35 U.S.C. 103 as being unpatentable over Balan ("EKILA: Synthetic media provenance and attribution for generative art", published 4/10/2023) in view of Lyons (US 20240362968 A1) in view of Todasco ("Yes, AI Can Create a Comic Book", published 2/27/2023).
For claim 18, Balan modified by Lyons discloses the server of claim 14, as described above. Balan modified by Lyons further discloses: wherein: the type of content comprises a digital media and the one or more creators comprise one or more creators (Balan §3.1.1 contemplates generation of images, Lyons 0064 contemplates generation of media in an creator’s style).
Balan modified by Lyons does not disclose, wherein the media is a book and wherein the creator is an author.
Todasco discloses: wherein the media is a book and wherein the creator is an author (p.2 discloses the use of text-prompt based interface MidJourney, hence, combination with Lyons yielding a technique where particular creators are authors of books).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Balan modified by Lyons by incorporating book generation technique of Todasco. Both concern the art of generative text and image ai, and the incorporation would have, according to Todasco, provide various advantages such as improve user story ideas, provide user entertainment (p.2).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Conde ("CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification", published 2022) discloses a technique of associating particular artistic styles with test descriptions via CLIP.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LIANG LI whose telephone number is (303)297-4263. The examiner can normally be reached Mon-Fri 9-12p, 3-11p MT (11-2p, 5-1a ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. The examiner is available for interviews Mon-Fri 6-11a, 2-7p MT (8-1p, 4-9p ET).
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Jennifer Welch can be reached on (571)272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center or Private PAIR to authorized users only. Should you have questions about access to Patent Center or the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/LIANG LI/
Primary examiner AU 2143
Prosecution Insights