DETAILED ACTION
This Office Action is in response to the Applicants' communication filed on April 8, 2026, which amends the independent claims 1, 11, and 16, and presents arguments, is hereby acknowledged. Claims 1-20 are currently pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s arguments filed on April 8, 2026, have been fully considered.
Applicant argues that by this response, the independent claims 1, 11, and 16 are hereby amended to add a new limitation “generating, using a language model comprising an attention mechanism, an image generation prompt based on having the image generation prompt attribute by processing the text generation prompt and the target token using the attention mechanism, wherein the image generation prompt has the image generation prompt attribute comprises a sequence of words indicating a target image characteristic” in order to overcome the 35 U.S.C. §103 rejection.
Examiner replies that the amended claims add two new limitations “generating, using a language model comprising an attention mechanism, an image generation prompt based on having the image generation prompt attribute by processing the text generation prompt and the target token using the attention mechanism” and “wherein the image generation prompt has the image generation prompt attribute comprises a sequence of words indicating a target image characteristic”. The primary art, Aberman, etc. (US 20250349040 A1) teaches that “generating, using a language model comprising an attention mechanism (See Aberman: [0147], “The decoupled cross-attention mechanism 510 provides separate pathways for processing image and text features, namely image feature cross-attention 522 and text feature cross-attention 524, as shown in FIG. 5. The decoupled cross-attention mechanism 510 separates attention processes for different data modalities, allowing each to contribute effectively to the final output. In some examples, both the identity representation and the text prompt representation are fed as latent-space representations via the decoupled cross-attention mechanism 510. FIG. 6 illustrates a decoupled cross-attention mechanism, according to some examples, in more detail”. Note that the image feature cross-attention and the text feature cross-attention are still attention mechanism).
Examiner further replies that the second newly added limitation “wherein the image generation prompt has the image generation prompt attribute comprises a sequence of words indicating a target image characteristic” may overcome the cited portions of the prior arts. However, a newly found art, Kowalski, etc. (US 20210335029 A1) teaches that t wherein the image generation prompt has the image generation prompt attribute comprises a sequence of words indicating a target image characteristic (See Kowalski: Fig. 1, and [0026], “In the example of FIG. 1 an end user computing device 110 such as a smart phone shown on the left hand side of the figure displays a real image of a child's face in a neutral expression with eyes open and with no facial hair. A user inputs values of parameters including “no smile”, “no beard” and “eyes shut”. The neural renderer 102 generates an output image which is displayed at the smart phone on the right hand side in FIG. 1. The output image depicts the child's face with eyes shut, no smile and no beard. Previously it has not been possible to achieve this type of functionality using neural network technology. A significant level of control over generative neural network technology is achieved without sacrificing realism. Previous approaches using conditional models trained with detailed hand labelling of the dataset struggle to generalize to out of distribution combinations of control parameters such as children with facial hair. In contrast the present technology does not need detailed hand labeled datasets and performs well for combinations of control parameters such as children with facial hair”; [0004], “Even when conditional models are trained with detailed labels, they struggle to generalize to out-of-distribution combinations of control parameters such as children with extensive facial hair or young people with gray hair. Thus it has not previously been possible for GAN based rendering techniques to replace traditional rendering pipelines”; and [0023], “FIG. 1 is a schematic diagram of a neural renderer 102 deployed as a cloud service and/or at an end user computing device 110. The neural renderer comprises one or more neural networks used for various image processing tasks including: generating an image of an object using neural networks and where it is possible to control attributes of the image using semantically meaningful parameters”. Note that “the child's face with eyes shut, no smile and no beard” for synthetic image generation is mapped to “a sequence of words indicating a target image characteristic”). The remaining arguments of the applicant are muted in view of the newly found art.
Examiner respectfully further replies that the Applicant's arguments have been fully considered and a new ground of rejections have been made. Accordingly, new grounds of rejection are set forth below. Since the new grounds of rejection are necessitated by Applicant's amendments to the claims, the present action is made final.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 6-7, 9-13, and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Aberman, etc. (US 20250349040 A1) in view of Lester, etc. (US 20230325725 A1), further in view of Kowalski, etc. (US 20210335029 A1).
Regarding claim 1, Aberman teaches that a method for image generation (See Aberman: Fig. 1, and [0046], "FIG. 1 is a block diagram showing an example interaction system 100 for facilitating interactions (e.g., exchanging text messages, conducting text, audio and video calls, or playing games) over a network. The interaction system 100 includes multiple user systems 102, each of which hosts multiple applications, including an interaction client 104 (as an example of an interaction application) and other applications 106. Each interaction client 104 is communicatively coupled, via one or more communication networks including a network 108 (e.g., the Internet), to other instances of the interaction client 104 (e.g., hosted on respective other user systems 102), an interaction server system 110 and third-party servers 112. An interaction client 104 can also communicate with locally hosted applications 106 using Application Programming Interfaces (APls)"), comprising:
obtaining a text generation prompt (See Aberman: Fig. 5, and [0145], "In addition to the combined identity representation, the diffusion model 508 receives a text prompt representation. The text prompt representation is generated by the text encoder 512 based on a text prompt that, for example, describes a scenario and/or style for the output image 526. In some examples, the user provides the text prompt via the interaction client 104") and a target token (See Aberman: Figs. 2 and 11, and [0224], "In some examples, the image encoder is pre-trained to encode each input image into a set of latent tokens (e.g., a latent-space representation). The merging component can then be trained to merge multiple sets of these tokens into a single set of tokens to define a combined identity representation. The combined identity representation captures facial features of the subject depicted in the input images (e.g., the three reference images referred to above)". Note that the latent tokens may not be the target token, and another art herein is used for this cited limitation) indicating an image generation prompt attribute (See Aberman: Fig. 4, and [0112], "The user interface component 402 enables interactions between a user of a user system 102 and the image generation system 230. The user interface component 402 is configured to receive user inputs, such as image uploads, text prompts, and selections of desired image attributes. The user interface component 402 facilitates navigation and operation of image generation system 230, providing tools and options, for example via the interaction client 104, that are useful for running or customizing the image generation process". Note that the desired image attributes may be the image generation attributes);
generating, using a language model (See Aberman: Fig. 5, and [0137], "The image encoder 502 is configured to extract image features from input images. For example, a pre-trained CLIP (Contrastive Language-Image Pre-training) encoder can be utilized as the image encoder 502, either as is or with additional fine-tuning to focus on specific features (e.g., facial features). The image encoder 502 may encode images into a vector space, e.g., to produce a feature vector that represents the visual content". Note that CLIP encoder may be the language model) comprising an attention mechanism (See Aberman: [0147], “The decoupled cross-attention mechanism 510 provides separate pathways for processing image and text features, namely image feature cross-attention 522 and text feature cross-attention 524, as shown in FIG. 5. The decoupled cross-attention mechanism 510 separates attention processes for different data modalities, allowing each to contribute effectively to the final output. In some examples, both the identity representation and the text prompt representation are fed as latent-space representations via the decoupled cross-attention mechanism 510. FIG. 6 illustrates a decoupled cross-attention mechanism, according to some examples, in more detail”. Note that the image feature cross-attention and the text feature cross-attention are still attention mechanism), an image generation prompt having the image generation prompt attribute by processing the text generation prompt and the target token using the attention mechanism (See Aberman: Figs. 1-5, and [0113], "In some examples, the user interface component 402 provides an automated prompt generator component to allow a user to request a prompt, e.g., a sample text prompt or a suggested text prompt, in response to which the image generation system 230 automatically generates and presents a prompt to the user. The user can use such a prompt as a starting point (or as inspiration) to create a final prompt, or may submit such a prompt directly for image generation. The user interface component 402 also provides an upload component for uploading multiple input images (e.g., images depicting the face of the user from different angles or depicting various facial expressions of the user)"; and [0135], "At a high-level, the image generation process 500 involves separately processing each of a plurality of input images via an image encoder 502 and a projection network 504 to obtain identity representations. The identity representations are combined by a merge block 506 to obtain a combined identity representation, which is fed to a diffusion model 508 via a decoupled cross-attention mechanism 510. The decoupled cross-attention mechanism 510 allows the diffusion model 508 to process the combined identity representation and text features. The text features are processed via a text encoder 512". Note that the final prompt or the text prompt from text processing plus the combined identity representation from the image processing which is fed to the diffusion model, for image generation may be the image generation prompt),
wherein the image generation prompt (See Aberman: Fig. 5, and [0146], "In some examples, the text encoder 512 is a CLIP text encoder. The text encoder 512 processes the text prompt 520, converting textual information (e.g., "me at the beach in a photorealistic style") into a set of text features that describe, for example, the desired thematic or stylistic elements of the output image". Note that the text prompt has features for image generation which may map to the image generation prompt attributes) comprises a sequence of words indicating a target image characteristic; and
generating, using an image generation model, a synthetic image based on the image generation prompt (See Aberman: Fig. 5, and [0148], "The diffusion model 508 uses a diffusion process to generate the output image 526. Based on the combined identity representation, the output image 526 reflects the identity of the subject. Since the text prompt is also processed by the diffusion model 508, thematic or stylistic preferences are also incorporated into the output image 526. The output image 526 can thus be referred to as a personalized output image". Note that the personalized output image may map to the synthetic image, and the synthetic image is generated based on the image generation prompt which may be the combination of the text prompts and the combined identify representation), wherein the synthetic image depicts the target image characteristic (See Aberman: Fig. 11, and [0227], “One or more loss functions are employed to measure discrepancies between the features of the generated image and those of the target image, thereby encouraging the model to minimize these discrepancies. As a result, parameters of the merging component and parameters of the new layers of the diffusion model are adjusted as training progresses. By repeatedly training on various sets of images and corresponding targets, the image generation system learns to abstract essential, characterizing, and/or non-temporary identity characteristics that are consistent across different images of the same person, and thereby obtain a personalized output image depicting an AI-generated person with facial features that closely match those of the subject in the target image”).
However, Aberman fails to explicitly disclose that obtaining a target token; and wherein the image generation prompt comprises a sequence of words indicating a target image characteristic.
However, Lester teaches that obtaining a target token (See Lester: Figs.1-2, and [0137], "In some implementations, the prompt representation initialization can include training from scratch, using random initialization. Another option can include initializing each prompt token to an embedding drawn from the model's vocabulary. In some implementations, the soft-prompt can modulate the frozen network's behavior in the same way as textual context preceding the input, therefore, a word-like representation may serve as a good initialization spot. For classification tasks, a third option can be to initialize the prompt with embeddings that represent an enumeration of the output classes. Initializing the prompt with the embeddings of the valid target tokens can prime the model to restrict the output to the legal output classes").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Aberman to have obtaining a target token as taught by Lester in order to reduce the computational cost for generating and tuning prompts (See Lester: Figs. 1A-C, and [0220], "The systems and methods disclosed herein can be utilized for generating a meta-prompt, which can reduce the computational cost for generating and tuning prompts. A meta-prompt can be a prompt, learned via prompt tuning, that when processed with a few input examples produces a prompt. The output prompt can be used by the machine-learned model (e.g., the frozen model) to solve the task sketched by the input examples. The meta-prompt can enable the scaling to a library of millions of prompts"). Aberman teaches a method and system that may generate a synthetic image using generative model based on the text prompts extracted from the text inputs and image tokens (image representations) extracted from the image inputs; while Lester teaches a system and method that may generate an output for a specific task based on the target token. Therefore, it is obvious to one of ordinary skill in the art to modify Aberman by Lester to use the target token to generate the legal output classes. The motivation to modify Aberman by Lester is "Use of known technique to improve similar devices (methods, or products) in the same way".
However, Aberman, modified by Lester, fails to explicitly disclose that wherein the image generation prompt comprises a sequence of words indicating a target image characteristic.
However, Kowalski teaches that wherein the image generation prompt comprises a sequence of words indicating a target image characteristic (Kowalski: Fig. 1, and [0026], “In the example of FIG. 1 an end user computing device 110 such as a smart phone shown on the left hand side of the figure displays a real image of a child's face in a neutral expression with eyes open and with no facial hair. A user inputs values of parameters including “no smile”, “no beard” and “eyes shut”. The neural renderer 102 generates an output image which is displayed at the smart phone on the right hand side in FIG. 1. The output image depicts the child's face with eyes shut, no smile and no beard. Previously it has not been possible to achieve this type of functionality using neural network technology. A significant level of control over generative neural network technology is achieved without sacrificing realism. Previous approaches using conditional models trained with detailed hand labelling of the dataset struggle to generalize to out of distribution combinations of control parameters such as children with facial hair. In contrast the present technology does not need detailed hand labeled datasets and performs well for combinations of control parameters such as children with facial hair”; [0004], “Even when conditional models are trained with detailed labels, they struggle to generalize to out-of-distribution combinations of control parameters such as children with extensive facial hair or young people with gray hair. Thus it has not previously been possible for GAN based rendering techniques to replace traditional rendering pipelines”; and [0023], “FIG. 1 is a schematic diagram of a neural renderer 102 deployed as a cloud service and/or at an end user computing device 110. The neural renderer comprises one or more neural networks used for various image processing tasks including: generating an image of an object using neural networks and where it is possible to control attributes of the image using semantically meaningful parameters”. Note that “the child's face with eyes shut, no smile and no beard” for synthetic image generation is mapped to “a sequence of words indicating a target image characteristic”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Aberman to have wherein the image generation prompt comprises a sequence of words indicating a target image characteristic as taught by Kowalski in order to enable generating realistic head and face images without the need to author expensive three-dimensional (3D) assets by training on curated datasets of 2D images of real human faces by using generative adversarial networks (GANs) in an easy manner (See Kowalski: Fig. 1, and [0003], "Recent advances in generative adversarial networks (GANs) have enabled the production of realistic high resolution images of smooth organic objects such as faces. Generating photorealistic human bodies, and faces in particular, with traditional rendering pipelines that do not use neural networks is notoriously difficult, requiring hand-crafted three dimensional (3D) assets "). Aberman teaches a method and system that may generate a synthetic image using generative model based on the text prompts extracted from the text inputs and image tokens (image representations) extracted from the image inputs; while Kowalski teaches a system and method that may generate text prompts with control parameters to control the attributes of the synthetic images. Therefore, it is obvious to one of ordinary skill in the art to modify Aberman by Kowalski to generate text prompts with a sequence of words to control the attributes of the synthetic images. The motivation to modify Aberman by Kowalski is "Use of known technique to improve similar devices (methods, or products) in the same way".
Regarding claim 2, Aberman, Lester, and Kowalski teach all the features with respect to claim 1 as outlined above. Further, Lester teaches that thee method of claim 1, wherein obtaining the target token comprises:
identifying a target attribute for the synthetic image (See Lester: Fig. 2, and [0197], "For example, a user can have one-or-few-shot examples for a task they want to do well on. Instead of collecting more data, the users can utilize a prompt tuning semantic search feature to find datasets, tasks, and prompts that are similar to their task. In some implementations, the prompt tuning semantic search can begin with a user sending a prompt tuning API a small dataset of examples". Note that that tuning prompts may be the target attributes); and
selecting the target token based on the target attribute (See Lester: Figs. 1A-C, and [0040], "More specifically, the systems and methods can include obtaining input data and a prompt. The prompt can include one or more learned parameters associated with a particular task. In some implementations, the prompt can prime a pre-trained machine-learned model for the particular task. The prompt may be a prompt obtained from a prompt database based on one or more user selections. Additionally and/or alternatively, the prompt may be a prompt generated based on a training dataset that includes a plurality of training examples and a plurality of respective labels. In some implementations, the input data can include text data, image data, video data, audio data, and/or latent encoding data"; and Fig. 2, and [0103], "In particular, FIG. 2 can depict a comparison 200 between model tuning and prompt tuning according to example embodiments of the present disclosure. Model tuning 202 can rely on making a task-specific copy of the entire pre-trained model for each downstream task, and inference may be performed in separate batches. Prompt tuning 204 may only rely on storing a small task-specific prompt for each task and may enable mixed-task inference using the original pre-trained model. Using a TS "XXL" model, each additional copy of the tuned model may rely on 11 billion parameters. By contrast, the tuned prompts may only rely on 81,920 parameters per task-a reduction of over five orders of magnitude-given a prompt length of 20 tokens and embedding dimension 4,096". Note that the prompt is selected based on users selections, and the prompt has a given tokens, which may map to the selecting target tokens).
Regarding claim 3, Aberman, Lester, and Kowalski teach all the features with respect to claim 1 as outlined above. Further, Aberman, Lester, and Kowalski teach that the method of claim 1, wherein obtaining the target token comprises:
receiving a user input indicating the image generation prompt attribute (See Aberman: Fig. 5, and [0136], "The components mentioned above work together to process image and text inputs, ultimately synthesizing these inputs into a final output image that reflects both the identity of a subject and aspects specified by a user via a text prompt, such as thematic, stylistic, spatial, structural, or scenario-related guidance or instructions"); and
selecting the target token based on the user input (See Lester: Figs. 1A-C, and [0040], "More specifically, the systems and methods can include obtaining input data and a prompt. The prompt can include one or more learned parameters associated with a particular task. In some implementations, the prompt can prime a pre-trained machine-learned model for the particular task. The prompt may be a prompt obtained from a prompt database based on one or more user selections. Additionally and/or alternatively, the prompt may be a prompt generated based on a training dataset that includes a plurality of training examples and a plurality of respective labels. In some implementations, the input data can include text data, image data, video data, audio data, and/or latent encoding data"; and Fig. 2, and [0103], "In particular, FIG. 2 can depict a comparison 200 between model tuning and prompt tuning according to example embodiments of the present disclosure. Model tuning 202 can rely on making a task-specific copy of the entire pre-trained model for each downstream task, and inference may be performed in separate batches. Prompt tuning 204 may only rely on storing a small task-specific prompt for each task and may enable mixed-task inference using the original pre-trained model. Using a TS "XXL" model, each additional copy of the tuned model may rely on 11 billion parameters. By contrast, the tuned prompts may only rely on 81,920 parameters per task-a reduction of over five orders of magnitude-given a prompt length of 20 tokens and embedding dimension 4,096". Note that the prompt is selected based on users selections, and the prompt has a given tokens, which may map to the selecting target tokens).
Regarding claim 6, Aberman, Lester, and Kowalski teach all the features with respect to claim 1 as outlined above. Further, Aberman teaches that the method of claim 1, further comprising: filtering the text generation prompt based on a content filter (See Aberman: Fig. 2, and [0086], "In some examples, the image generation system 230 is also responsible for content checking or filtering, such as checking of a prompt for objectionable language or checking of an input image for unwanted content before allowing a new output image to be generated. In some examples, the image generation system 230 provides an automatic prompt generation feature by enabling a user to request a prompt, e.g., a sample text prompt or a suggested text prompt, which can assist the user in obtaining a new output image").
Regarding claim 7, Aberman, Lester, and Kowalski teach all the features with respect to claim 1 as outlined above. Further, Aberman teaches that the method of claim 1, further comprising:
filtering the image generation prompt based on a content filter (See Aberman: Fig. 2, and [0087], "An artificial intelligence and machine learning system 232 provides a variety of services to different subsystems within the interaction system 100. For example, the artificial intelligence and machine learning system 232 operates with the image processing system 202 and the camera system 204 to analyze images and extract information such as objects, text, or faces. This information can then be used by the image processing system 202 to enhance, filter, or manipulate (e.g., apply a visual augmentation to) images. The artificial intelligence and machine learning system 232 may be used by the augmentation system 206 to generate augmented content and augmented reality experiences, such as adding virtual objects or animations to real-world images. The communication system 208 and messaging system 210 may use the artificial intelligence and machine learning system 232 to analyze communication patterns and provide insights into how users interact with each other and provide intelligent message classification and tagging, such as categorizing messages based on sentiment or topic").
Regarding claim 9, Aberman, Lester, and Kowalski teach all the features with respect to claim 1 as outlined above. Further, Aberman teaches that the method of claim 1, wherein: the language model is trained to generate the image generation prompt having the image generation prompt attribute (See Aberman: Fig. 5, and [0140], "For each of the first input image 514, the second input image 516, and the third input image 518, the image features as obtained via the image encoder 502 and the projection network 504 are referred to as a respective identity representation. The identity representation captures characteristics or attributes of the subject as extracted from the particular input image. Since the images all depict the same subject, but are not identical, the identity representations may also be similar but not identical (with variations depending on the extent of differences between images). The merge block 506 receives the respective identity representations from the projection network 504 and combines them into a single, unified set of image features. This merging process causes synthesizing of the identity information from multiple images, creating a comprehensive, accurate, and/or consistent representation that captures the essence of the subject's identity across different visual contexts". Note that the identity representation is obtained by processing the input image and it captures the features (attributes) of the input image, and the image identity representation and text prompt may be mapped to the image generation prompt, which is inputted to the diffusion model to generate synthetic image).
Regarding claim 10, Aberman, Lester, and Kowalski teach all the features with respect to claim 1 as outlined above. Further, Aberman teaches that the method of claim 1, wherein: the language model is trained based on the synthetic image (See Aberman: Fig. 10, and [0132], "Generally, in order to train a model to provide one or more functionality as described in examples of the present disclosure, training data in the form of prompts, images, and/or metadata may be used. A training data set for a generative model may include thousands or millions of images or sets of images paired. Training data may include real and/or synthetic (e.g., Al-generated) data. In some examples, a training data set includes, for a particular image, a caption/prompt. The caption/prompt can be from real data, or can be generated by an automated caption generator, such as an image-to-text model. These captions may be used in the training process. A caption can, for example, be automatically generated for an image using a multimodal encoder-decoder". Note that the synthetic data for training may be mapped to the synthetic image for training).
Regarding claim 11, Aberman, Lester, and Kowalski teach all the features with respect to claim 1 as outlined above. Further, Aberman, Lester, and Kowalski teach that the method for training a machine learning model (See Aberman: Fig. 1, and [0046], "FIG. 1 is a block diagram showing an example interaction system 100 for facilitating interactions (e.g., exchanging text messages, conducting text, audio and video calls, or playing games) over a network. The interaction system 100 includes multiple user systems 102, each of which hosts multiple applications, including an interaction client 104 (as an example of an interaction application) and other applications 106. Each interaction client 104 is communicatively coupled, via one or more communication networks including a network 108 (e.g., the Internet), to other instances of the interaction client 104 (e.g., hosted on respective other user systems 102), an interaction server system 110 and third-party servers 112. An interaction client 104 can also communicate with locally hosted applications 106 using Application Programming Interfaces (APls)"), comprising:
obtaining training data (See Aberman: Fig. 5, and [0145], "In addition to the combined identity representation, the diffusion model 508 receives a text prompt representation. The text prompt representation is generated by the text encoder 512 based on a text prompt that, for example, describes a scenario and/or style for the output image 526. In some examples, the user provides the text prompt via the interaction client 104") comprising a training input text (See Aberman: Fig. 4, and [0118], "The text prompt processing component 408 handles text inputs received from the user interface component 402. The text prompt processing component 408 processes these inputs (e.g., via a text encoder provided by the artificial intelligence and machine learning system 232) to extract relevant information and parameters that will guide the image generation process. This component ensures that the text prompts are correctly interpreted to align the generated images with the user's textual descriptions and intentions") and a target token (See Lester: Figs.1-2, and [0137], "In some implementations, the prompt representation initialization can include training from scratch, using random initialization. Another option can include initializing each prompt token to an embedding drawn from the model's vocabulary. In some implementations, the soft-prompt can modulate the frozen network's behavior in the same way as textual context preceding the input, therefore, a word-like representation may serve as a good initialization spot. For classification tasks, a third option can be to initialize the prompt with embeddings that represent an enumeration of the output classes. Initializing the prompt with the embeddings of the valid target tokens can prime the model to restrict the output to the legal output classes") indicating an image generation prompt attribute (See Aberman: Fig. 4, and [0112], "The user interface component 402 enables interactions between a user of a user system 102 and the image generation system 230. The user interface component 402 is configured to receive user inputs, such as image uploads, text prompts, and selections of desired image attributes. The user interface component 402 facilitates navigation and operation of image generation system 230, providing tools and options, for example via the interaction client 104, that are useful for running or customizing the image generation process". Note that the desired image attributes may be the image generation attributes); and
training (See Aberman: Fig. 10, and [0197], "FIG. 10 illustrates further details of two example phases, namely a training phase 1004 (e.g., part of model selection and training 906) and a prediction phase 1010 (part of prediction 910). Prior to the training phase 1004, feature engineering 904 is used to identify features 1008. This may include identifying informative, discriminating, and independent features for effectively operating the trained machine learning program 1002 in pattern recognition, classification, and regression. In some examples, the training data 1006 includes labeled data, known for pre-identified features 1008 and one or more outcomes. Each of the features 1008 may be a variable or attribute, such as an individual measurable property of a process, article, system, or phenomenon represented by a data set (e.g., the training data 1006). Features 1008 may also be of different types, such as numeric features, strings, and graphs, and may include one or more of content 1012, concepts 1014, attributes 1016, historical data 1018, and/or user data 1020, merely for example"),
using the training data (See Aberman: Fig. 10, and [0197], "FIG. 10 illustrates further details of two example phases, namely a training phase 1004 (e.g., part of model selection and training 906) and a prediction phase 1010 (part of prediction 910). Prior to the training phase 1004, feature engineering 904 is used to identify features 1008. This may include identifying informative, discriminating, and independent features for effectively operating the trained machine learning program 1002 in pattern recognition, classification, and regression. In some examples, the training data 1006 includes labeled data, known for pre-identified features 1008 and one or more outcomes. Each of the features 1008 may be a variable or attribute, such as an individual measurable property of a process, article, system, or phenomenon represented by a data set (e.g., the training data 1006). Features 1008 may also be of different types, such as numeric features, strings, and graphs, and may include one or more of content 1012, concepts 1014, attributes 1016, historical data 1018, and/or user data 1020, merely for example"),
a language model (See Aberman: Fig. 5, and [0137], "The image encoder 502 is configured to extract image features from input images. For example, a pre-trained CLIP (Contrastive Language-Image Pre-training) encoder can be utilized as the image encoder 502, either as is or with additional fine-tuning to focus on specific features (e.g., facial features). The image encoder 502 may encode images into a vector space, e.g., to produce a feature vector that represents the visual content". Note that CLIP encoder may be the language model) comprising an attention mechanism (See Aberman: [0147], “The decoupled cross-attention mechanism 510 provides separate pathways for processing image and text features, namely image feature cross-attention 522 and text feature cross-attention 524, as shown in FIG. 5. The decoupled cross-attention mechanism 510 separates attention processes for different data modalities, allowing each to contribute effectively to the final output. In some examples, both the identity representation and the text prompt representation are fed as latent-space representations via the decoupled cross-attention mechanism 510. FIG. 6 illustrates a decoupled cross-attention mechanism, according to some examples, in more detail”. Note that the image feature cross-attention and the text feature cross-attention are still attention mechanism) to generate an image generation prompt using the attention mechanism based on the training input text (See Aberman: Figs. 1-5, and [0113], "In some examples, the user interface component 402 provides an automated prompt generator component to allow a user to request a prompt, e.g., a sample text prompt or a suggested text prompt, in response to which the image generation system 230 automatically generates and presents a prompt to the user. The user can use such a prompt as a starting point (or as inspiration) to create a final prompt, or may submit such a prompt directly for image generation. The user interface component 402 also provides an upload component for uploading multiple input images (e.g., images depicting the face of the user from different angles or depicting various facial expressions of the user)"; and [0135], "At a high-level, the image generation process 500 involves separately processing each of a plurality of input images via an image encoder 502 and a projection network 504 to obtain identity representations. The identity representations are combined by a merge block 506 to obtain a combined identity representation, which is fed to a diffusion model 508 via a decoupled cross-attention mechanism 510. The decoupled cross-attention mechanism 510 allows the diffusion model 508 to process the combined identity representation and text features. The text features are processed via a text encoder 512". Note that the final prompt or the text prompt from text processing plus the combined identity representation from the image processing which is fed to the diffusion model, for image generation may be the image generation prompt) and the target token (See Lester: Figs.1-2, and [0137], "In some implementations, the prompt representation initialization can include training from scratch, using random initialization. Another option can include initializing each prompt token to an embedding drawn from the model's vocabulary. In some implementations, the soft-prompt can modulate the frozen network's behavior in the same way as textual context preceding the input, therefore, a word-like representation may serve as a good initialization spot. For classification tasks, a third option can be to initialize the prompt with embeddings that represent an enumeration of the output classes. Initializing the prompt with the embeddings of the valid target tokens can prime the model to restrict the output to the legal output classes". Note that the target token of Lester may replace the token of Aberman in training the image generation model), wherein the image generation prompt comprises a sequence of words indicating a target image characteristic (Kowalski: Fig. 1, and [0026], “In the example of FIG. 1 an end user computing device 110 such as a smart phone shown on the left hand side of the figure displays a real image of a child's face in a neutral expression with eyes open and with no facial hair. A user inputs values of parameters including “no smile”, “no beard” and “eyes shut”. The neural renderer 102 generates an output image which is displayed at the smart phone on the right hand side in FIG. 1. The output image depicts the child's face with eyes shut, no smile and no beard. Previously it has not been possible to achieve this type of functionality using neural network technology. A significant level of control over generative neural network technology is achieved without sacrificing realism. Previous approaches using conditional models trained with detailed hand labelling of the dataset struggle to generalize to out of distribution combinations of control parameters such as children with facial hair. In contrast the present technology does not need detailed hand labeled datasets and performs well for combinations of control parameters such as children with facial hair”; [0004], “Even when conditional models are trained with detailed labels, they struggle to generalize to out-of-distribution combinations of control parameters such as children with extensive facial hair or young people with gray hair. Thus it has not previously been possible for GAN based rendering techniques to replace traditional rendering pipelines”; and [0023], “FIG. 1 is a schematic diagram of a neural renderer 102 deployed as a cloud service and/or at an end user computing device 110. The neural renderer comprises one or more neural networks used for various image processing tasks including: generating an image of an object using neural networks and where it is possible to control attributes of the image using semantically meaningful parameters”. Note that “the child's face with eyes shut, no smile and no beard” for synthetic image generation is mapped to “a sequence of words indicating a target image characteristic”).
Regarding claim 12, Aberman, Lester, and Kowalski teach all the features with respect to claim 11 as outlined above. Further, Aberman, Lester, and Kowalski teach that the method of claim 11, wherein the training comprises: generating, using an image generation model, a synthetic image based on an input prompt (See Aberman: Fig. 5, and [0148], "The diffusion model 508 uses a diffusion process to generate the output image 526. Based on the combined identity representation, the output image 526 reflects the identity of the subject. Since the text prompt is also processed by the diffusion model 508, thematic or stylistic preferences are also incorporated into the output image 526. The output image 526 can thus be referred to as a personalized output image". Note that the personalized output image may map to the synthetic image, and the synthetic image is generated based on the image generation prompt which may be the combination of the text prompts and the combined identify representation);
classifying the synthetic image according to an image attribute (See Lester: Fig. 2, and [0029], "In some implementations, a training dataset can be obtained. The training dataset can include a plurality of training examples and a plurality of training labels for the respective training examples. In some implementations, the plurality of training examples can include a plurality of text datasets. The particular task can be a natural language processing task. In some implementations, the training dataset can include a plurality of text examples and a plurality of classifications associated with the plurality of text examples. Alternatively and/or additionally, the training dataset can include a plurality of visual examples (e.g., a plurality of images) and a plurality of classifications (e.g., object classifications in an image, an image classification, a semantic classification, etc.) associated with the plurality of visual examples"); and
classifying the input prompt according to the image attribute based on the classification of the synthetic image, wherein the training is based on the classification of the input prompt (See Lester: Fig. 2, and [0033], "In some implementations, the particular task can include a classification task (e.g., a text classification task, a syntactical classification task, or a sentiment analysis task that classifies whether the input text has a positive sentiment or a negative sentiment). Alternatively and/or additionally, the particular task can include determining a response and/or a follow-up to the input text. For example, the output may be a predicted answer or generated response to an input open ended question. Alternatively and/or additionally, the output may include an augmented version of the input data, which can include correcting data or adjusting data based on the specific task or training dataset. The particular task may include a translation task". Note that the text prompt classification is associated with the image with positive or negative sentiments, which is mapped to this claimed limitation of classifying the input prompt according to the image attributes).
Regarding claim 13, Aberman, Lester, and Kowalski teach all the features with respect to claim 12 as outlined above. Further, Lester teaches that the method of claim 12, wherein the training comprises: computing an image attribute loss based on the classification of the input prompt (See Lester: Figs. 1A-C, and [0031], "A prompt gradient can then be determined based at least in part on a comparison between the training output and one or more training labels associated with the one or more training examples. In some implementations, the prompt gradient can be determined by evaluating a loss function that is evaluated based on a difference between the training output and the one or more training labels. The loss function can include a perceptual loss or another loss function. In some implementations, the labels can include ground truth outputs for the respective training examples"; and [0054], "To create a soft prompt for a given task, the system may first initialize the prompt as a fixed-length sequence of vectors (e.g., 20 tokens long). In some implementations, the systems and methods can attach these vectors to the beginning of each embedded input and feed the combined sequence into the model. Alternatively and/or additionally, the systems and methods can put the prompts at different parts of the input and analyze the effect of the different positions. The model's prediction can be compared to the target to calculate a loss, and the error can be back-propagated to calculate gradients, however the system may only apply these gradient updates to our new learnable vectors-keeping the core model frozen. While soft prompts learned in this way may not be immediately interpretable, at an intuitive level, the soft prompt can be extracting evidence about how to perform a task from the labeled dataset, performing the same role as a manually written text prompt, but without the need to be constrained to discrete language". Note that the learnable vectors may be the image attributes associated with the prompt input classification or categories); and
updating parameters of the language model based on the image attribute loss (See Lester: Fig. 1, and [0077], "The training computing system 150 can include a model trainer 160 that trains the machine-learned models 120 and/or 140 stored at the user computing device 102 and/or the server computing system 130 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, a ranking loss, and/or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations").
Regarding claim 15, Aberman, Lester, and Kowalski teach all the features with respect to claim 11 as outlined above. Further, Aberman, Lester, and Kowalski teach that the method of claim 11, wherein the training comprises:
generating, using the language model, a predicted image generation prompt (See Lester: Fig. 6, and [0116], "At 604, the computing system can process one or more training examples of the plurality of training examples and a prompt with a pre-trained machine-learned model to gene rate a training output. The plurality of pre-trained parameters for the pre-trained machine-learned model can be fixed during prompt tuning (e.g., the pre-trained machine-learned model can be frozen such that the parameters are not adjusted during training of the prompt parameters). In some implementations, the pre-trained machine-learned model can include a model adapted to generate a text prediction output for text that follows an input text (e.g., the input text can include "the sky is" and the output can be "blue"). Alternatively and/or additionally, the pre-trained machine-learned model may have been trained with text masking (e.g., the input text can include "The man old" and the output can be "is"). The pre-trained machine-learned model can include one or more encoder blocks and one or more decoder blocks. For example, the pre-trained machine-learned model can include an encoder-decoder model such as a transformer model. The prompt can be associated with a particular task, and the particular task may be associated with the one or more training examples");
determining whether the predicted image generation prompt includes inappropriate content (See Aberman: Fig. 2, and [0075], "A collection management system 220 is operationally responsible for managing sets or collections of media (e.g., collections of text, image video, and audio data). A collection of content (e.g., messages, including images, video, text, and audio) may be organized into an "event gallery" or an "event story." Such a collection may be made available for a specified time period, such as the duration of an event to which the content relates. For example, content relating to a music concert may be made available as a "story" for the duration of that music concert. The collection management system 220 may also be responsible for publishing an icon that provides notification of a particular collection to the user interface of the interaction client 104. The collection management system 220 includes a curation function that allows a collection manager to manage and curate a particular collection of content. For example, the curation interface enables an event organizer to curate a collection of content relating to a specific event (e.g., delete inappropriate content or redundant messages). Additionally, the collection management system 220 employs machine vision (or image recognition technology) and content rules to curate a content collection automatically. In certain examples, compensation may be paid to a user to include user-generated content into a collection. In such cases, the collection management system 220 operates to automatically make payments to such users to use their content");
computing a content loss based on the determination (See Aberman: Fig. 1, and [0227], "One or more loss functions are employed to measure discrepancies between the features of the generated image and those of the target image, thereby encouraging the model to minimize these discrepancies. As a result, parameters of the merging component and parameters of the new layers of the diffusion model are adjusted as training progresses. By repeatedly training on various sets of images and corresponding targets, the image generation system learns to abstract essential, characterizing, and/or non-temporary identity characteristics that are consistent across different images of the same person, and thereby obtain a personalized output image depicting an Al-generated person with facial features that closely match those of the subject in the target image"); and
updating parameters of the language model based on the content loss (See Aberman: Fig. 5, and [0128], "Once trained, in order to generate an image, the diffusion model uses the relevant input and applies the trained sequence of transformations to generate an output image. The model generates the image in a step-by-step manner, updating the image sequentially with additional information until the image is fully generated. In some examples, this process may be repeated to produce a set of candidate images, from which the final image is chosen based on criteria such as a likelihood score. The resulting image is intended to represent a visual interpretation of the input").
Regarding claim 16, Aberman, Lester, and Kowalski teach all the features with respect to claim 1 as outlined above. Further, Aberman, Lester, and Kowalski teach that the system for image generation (See Aberman: Fig. 1, and [0046], "FIG. 1 is a block diagram showing an example interaction system 100 for facilitating interactions (e.g., exchanging text messages, conducting text, audio and video calls, or playing games) over a network. The interaction system 100 includes multiple user systems 102, each of which hosts multiple applications, including an interaction client 104 (as an example of an interaction application) and other applications 106. Each interaction client 104 is communicatively coupled, via one or more communication networks including a network 108 (e.g., the Internet), to other instances of the interaction client 104 (e.g., hosted on respective other user systems 102), an interaction server system 110 and third-party servers 112. An interaction client 104 can also communicate with locally hosted applications 106 using Application Programming Interfaces (APls)"), comprising:
at least one memory component (See Aberman: Fig. 14, and [0266], "The memory 1406 includes a main memory 1416, a static memory 1418, and a storage unit 1420, both accessible to the processors 1404 via the bus 1410. The main memory 1406, the static memory 1418, and storage unit 1420 store the instructions 1402 embodying any one or more of the methodologies or functions described herein. The instructions 1402 may also reside, completely or partially, within the main memory 1416, within the static memory 1418, within machine-readable medium 1422 within the storage unit 1420, within at least one of the processors 1404 (e.g., within the processor's cache memory), or any suitable combination thereof, during execution thereof by the machine 1400");
at least one processor executing instructions stored in the at least one memory component (See Aberman: Fig. 14, and [0266], "The memory 1406 includes a main memory 1416, a static memory 1418, and a storage unit 1420, both accessible to the processors 1404 via the bus 1410. The main memory 1406, the static memory 1418, and storage unit 1420 store the instructions 1402 embodying any one or more of the methodologies or functions described herein. The instructions 1402 may also reside, completely or partially, within the main memory 1416, within the static memory 1418, within machine-readable medium 1422 within the storage unit 1420, within at least one of the processors 1404 (e.g., within the processor's cache memory), or any suitable combination thereof, during execution thereof by the machine 1400");
a language model (See Aberman: Fig. 5, and [0137], "The image encoder 502 is configured to extract image features from input images. For example, a pre-trained CLIP (Contrastive Language-Image Pre-training) encoder can be utilized as the image encoder 502, either as is or with additional fine-tuning to focus on specific features (e.g., facial features). The image encoder 502 may encode images into a vector space, e.g., to produce a feature vector that represents the visual content". Note that CLIP encoder may be the language model) comprising an attention mechanism (See Aberman: [0147], “The decoupled cross-attention mechanism 510 provides separate pathways for processing image and text features, namely image feature cross-attention 522 and text feature cross-attention 524, as shown in FIG. 5. The decoupled cross-attention mechanism 510 separates attention processes for different data modalities, allowing each to contribute effectively to the final output. In some examples, both the identity representation and the text prompt representation are fed as latent-space representations via the decoupled cross-attention mechanism 510. FIG. 6 illustrates a decoupled cross-attention mechanism, according to some examples, in more detail”. Note that the image feature cross-attention and the text feature cross-attention are still attention mechanism) and text generation parameters stored in the at least one memory component (See Aberman: Fig. 3, and [0090], "FIG. 3 is a schematic diagram illustrating data structures 300, which may be stored in a database, such as the database 128 of the interaction server system 110, according to certain examples. While the content of the database 128 is shown to comprise multiple tables, it will be appreciated that the data could be stored in other types of data structures (e.g., as an object-oriented database)"; and [0107], "A prompts table 320 may store one or more prompts that are or may be used with respect to the image generation system 230 and/or artificial intelligence and machine learning system 232. For example, the prompts table 320 may store prompts that can be selected by a user (or have previously been selected) for automatic image generation via the image generation system 230, where the interaction client 104 provides the user with access to automatic image generation functionality"),
the language model trained to generate an image generation prompt having an image generation prompt attribute having an image generation prompt attribute based on a text generation prompt and a target token (See Lester: Figs.1-2, and [0137], "In some implementations, the prompt representation initialization can include training from scratch, using random initialization. Another option can include initializing each prompt token to an embedding drawn from the model's vocabulary. In some implementations, the soft-prompt can modulate the frozen network's behavior in the same way as textual context preceding the input, therefore, a word-like representation may serve as a good initialization spot. For classification tasks, a third option can be to initialize the prompt with embeddings that represent an enumeration of the output classes. Initializing the prompt with the embeddings of the valid target tokens can prime the model to restrict the output to the legal output classes") indicating the image generation prompt attribute (See Aberman: Figs. 1-5, and [0113], "In some examples, the user interface component 402 provides an automated prompt generator component to allow a user to request a prompt, e.g., a sample text prompt or a suggested text prompt, in response to which the image generation system 230 automatically generates and presents a prompt to the user. The user can use such a prompt as a starting point (or as inspiration) to create a final prompt, or may submit such a prompt directly for image generation. The user interface component 402 also provides an upload component for uploading multiple input images (e.g., images depicting the face of the user from different angles or depicting various facial expressions of the user)"; and [0135], "At a high-level, the image generation process 500 involves separately processing each of a plurality of input images via an image encoder 502 and a projection network 504 to obtain identity representations. The identity representations are combined by a merge block 506 to obtain a combined identity representation, which is fed to a diffusion model 508 via a decoupled cross-attention mechanism 510. The decoupled cross-attention mechanism 510 allows the diffusion model 508 to process the combined identity representation and text features. The text features are processed via a text encoder 512". Note that the final prompt or the text prompt from text processing plus the combined identity representation from the image processing which is fed to the diffusion model, for image generation may be the image generation prompt),
wherein the image generation prompt generation prompt attribute (See Aberman: Fig. 5, and [0146], "In some examples, the text encoder 512 is a CLIP text encoder. The text encoder 512 processes the text prompt 520, converting textual information (e.g., "me at the beach in a photorealistic style") into a set of text features that describe, for example, the desired thematic or stylistic elements of the output image". Note that the text prompt has features for image generation which may map to the image generation prompt attributes) comprises a sequence of words indicating a target image characteristic (Kowalski: Fig. 1, and [0026], “In the example of FIG. 1 an end user computing device 110 such as a smart phone shown on the left hand side of the figure displays a real image of a child's face in a neutral expression with eyes open and with no facial hair. A user inputs values of parameters including “no smile”, “no beard” and “eyes shut”. The neural renderer 102 generates an output image which is displayed at the smart phone on the right hand side in FIG. 1. The output image depicts the child's face with eyes shut, no smile and no beard. Previously it has not been possible to achieve this type of functionality using neural network technology. A significant level of control over generative neural network technology is achieved without sacrificing realism. Previous approaches using conditional models trained with detailed hand labelling of the dataset struggle to generalize to out of distribution combinations of control parameters such as children with facial hair. In contrast the present technology does not need detailed hand labeled datasets and performs well for combinations of control parameters such as children with facial hair”; [0004], “Even when conditional models are trained with detailed labels, they struggle to generalize to out-of-distribution combinations of control parameters such as children with extensive facial hair or young people with gray hair. Thus it has not previously been possible for GAN based rendering techniques to replace traditional rendering pipelines”; and [0023], “FIG. 1 is a schematic diagram of a neural renderer 102 deployed as a cloud service and/or at an end user computing device 110. The neural renderer comprises one or more neural networks used for various image processing tasks including: generating an image of an object using neural networks and where it is possible to control attributes of the image using semantically meaningful parameters”. Note that “the child's face with eyes shut, no smile and no beard” for synthetic image generation is mapped to “a sequence of words indicating a target image characteristic”); and
an image generation model comprising image generation parameters stored in the at least one memory component (See Aberman: Fig. 12, and [0240], "Message augmentation data 1212: augmentation data (e.g., filters, stickers, or other annotations or enhancements) that represents augmentations to be applied to message image payload 1206, message video payload 1208, or message audio payload 1210 of the message 1200. Augmentation data fora sent or received message 1200 may be stored in the augmentation table 310"; and [0241], "Message duration parameter 1214: parameter value indicating, in seconds, the amount of time for which content of the message (e.g., the message image payload 1206, message video payload 1208, message audio payload 1210) is to be presented or made accessible to a user via the interaction client 104". Note that various data and parameters are stored in tables),
the image generation model trained to generate a synthetic image based on the image generation prompt (See Aberman: Fig. 5, and [0128], "Once trained, in order to generate an image, the diffusion model uses the relevant input and applies the trained sequence of transformations to generate an output image. The model generates the image in a step-by-step manner, updating the image sequentially with additional information until the image is fully generated. In some examples, this process may be repeated to produce a set of candidate images, from which the final image is chosen based on criteria such as a likelihood score. The resulting image is intended to represent a visual interpretation of the input"; and [0148], "The diffusion model 508 uses a diffusion process to generate the output image 526. Based on the combined identity representation, the output image 526 reflects the identity of the subject. Since the text prompt is also processed by the diffusion model 508, thematic or stylistic preferences are also incorporated into the output image 526. The output image 526 can thus be referred to as a personalized output image". Note that the personalized output image may map to the synthetic image, and the synthetic image is generated based on the image generation prompt which may be the combination of the text prompts and the combined identify representation).
Regarding claim 17, Aberman, Lester, and Kowalski teach all the features with respect to claim 16 as outlined above. Further, Aberman teaches that the system of claim 16, further comprising: a language verification component configured to filter the text generation prompt or the image generation prompt (See Aberman: Fig. 2, and [0086], "In some examples, the image generation system 230 is also responsible for content checking or filtering, such as checking of a prompt for objectionable language or checking of an input image for unwanted content before allowing a new output image to be generated. In some examples, the image generation system 230 provides an automatic prompt generation feature by enabling a user to request a prompt, e.g., a sample text prompt or a suggested text prompt, which can assist the user in obtaining a new output image". Note that filtering out the unwanted text may be mapped to the language verification).
Regarding claim 18, Aberman, Lester, and Kowalski teach all the features with respect to claim 16 as outlined above. Further, Lester teaches that the system of claim 16, further comprising: a classification network (See Lester: Fig. 6, and [0115], "At 602, a computing system can obtain a training dataset. The training dataset can include a plurality of training examples and a plurality of training labels for the respective training examples. In some implementations, the plurality of training examples can include a plurality of text datasets. The particular task can be a natural language processing task. In some implementations, the training dataset can include a plurality of text examples and a plurality of classifications associated with the plurality of text examples. Alternatively and/or additionally, the training dataset can include a plurality of visual examples (e.g., a plurality of images) and a plurality of classifications (e.g., object classifications in an image, an image classification, a semantic classification, etc.) associated with the plurality of visual examples") configured to generate information for the target token (See Lester: Figs.1-2, and [0137], "In some implementations, the prompt representation initialization can include training from scratch, using random initialization. Another option can include initializing each prompt token to an embedding drawn from the model's vocabulary. In some implementations, the soft-prompt can modulate the frozen network's behavior in the same way as textual context preceding the input, therefore, a word-like representation may serve as a good initialization spot. For classification tasks, a third option can be to initialize the prompt with embeddings that represent an enumeration of the output classes. Initializing the prompt with the embeddings of the valid target tokens can prime the model to restrict the output to the legal output classes").
Regarding claim 19, Aberman, Lester, and Kowalski teach all the features with respect to claim 16 as outlined above. Further, Aberman teaches that the system of claim 16, wherein: the language model comprises a transformer model (See Aberman: Fig. 9, and [0187], "Principles discussed herein for one machine learning algorithm can be applied to at least some other machine learning algorithms as well. Deep learning algorithms such as convolutional neural networks, recurrent neural networks, and transformers, as well as more traditional machine learning algorithms like decision trees, random forests, and gradient boosting may be used in various machine learning applications").
Regarding claim 20, Aberman, Lester, and Kowalski teach all the features with respect to claim 16 as outlined above. Further, Aberman teaches that the system of claim 16, wherein: the image generation model comprises a diffusion model (See Aberman: Fig. 5, and [0145], "In addition to the combined identity representation, the diffusion model 508 receives a text prompt representation. The text prompt representation is generated by the text encoder 512 based on a text prompt that, for example, describes a scenario and/or style for the output image 526. In some examples, the user provides the text prompt via the interaction client 104").
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Aberman, etc. (US 20250349040 A1) in view of Lester, etc. (US 20230325725 A1), further in view of Kowalski, etc. (US 20210335029 A1), and Hu, etc. (US 20250094816 A1).
Regarding claim 4, Aberman, Lester, and Kowalski teach all the features with respect to claim 1 as outlined above. Further, Lester teaches that the method of claim 1, wherein: the image generation prompt attribute comprises an image description attribute (See Lester: Fig. 3, and [0105], "In particular, prompt tuning 304 can retain the strong task performance of model tuning 302, while keeping the pre-trained model frozen, enabling efficient multitask serving. FIG. 3 can depict that although model tuning 302 can have strong task performance, model tuning 302 is computationally expensive and can require a newly trained model for each new task. Alternatively, engineered prompt design 306 (e.g., the use of several canonical examples with a task description) can allow the use of a single model for multiple tasks; however, task performance can be relatively weak due to the heavy reliance on the task description and the number of examples. However, prompt tuning 304 can utilize a single pre-trained model for a plurality of downstream tasks while maintaining strong task performance as tunable soft prompts are learned for each task in which each tunable soft prompt includes a limited number of learned parameters"), a vocabulary attribute (See Lester: Figs. 1-2, and [0137], "In some implementations, the prompt representation initialization can include training from scratch, using random initialization. Another option can include initializing each prompt token to an embedding drawn from the model's vocabulary. In some implementations, the soft-prompt can modulate the frozen network's behavior in the same way as textual context preceding the input, therefore, a word-like representation may serve as a good initialization spot. For classification tasks, a third option can be to initialize the prompt with embeddings that represent an enumeration of the output classes. Initializing the prompt with the embeddings of the valid target tokens can prime the model to restrict the output to the legal output classes"), or a prompt length.
However, Aberman, modified by Lester and Kowalski, fails to explicitly disclose that a prompt length.
However, Hu teaches that the image generation prompt attribute comprises a prompt length (See Hu: Fig. 1, and [0024], "In one embodiment, length loss 140 is configured to determine whether and/or to quantify to what extent the generated response 136 generated by the LLM complies with a word count, sentence/paragraph length, or other constraint on length of the text, and to dock or penalize the text generation loss function 134 consistent with the extent of any noncompliance with the specified constraint on length. In one embodiment, length loss 140 is configured to generate a value of length loss that indicates an extent to which the generated text response complies with one or more of the extracted instructions that specify a length of the generated text response". Note that the LLM use the length of the text to evaluate if the evaluation the loss function which is a measure if the generated text prompts complies with the instructions).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention was effectively filed to modify Aberman to have the image generation prompt attribute comprises a prompt length as taught by Hu in order to enable automatically valuating improvement of LLM text generation performance to control deployment of the improved LLM to a production environment (See Hu: Fig. 1, and [0008], "Systems, methods, and other embodiments are described herein that provide automated fine-tuning of text generation by large language models (LLMs). In one embodiment, a text generation tuning system automatically fine-tunes an LLM to improve performance of natural language text generation by the LLM. For example, the text generation tuning system automatically adjusts the LLM to cause the LLM to generate outputs that are more closely aligned with expectations for generating a body of natural language text in response to natural language requirements or instructions to the LLM. And, for example, the text generation tuning system automatically evaluates improvement of LLM text generation performance to control deployment of the improved LLM to a production environment. In one embodiment, the LLM text generation tuning system quantifies improvement to the performance of the LLM at the task of text generation, rendering the improvement verifiable"). Aberman teaches a method and system that may generate a synthetic image using generative model based on the text prompts extracted from the text inputs and image tokens (image representations) extracted from the image inputs; while Hu teaches a system and method that may process the text inputs using LLM to generate texts, tokens, and instructions and evaluate the improvements in text processing using text length measurements (loss). Therefore, it is obvious to one of ordinary skill in the art to modify Aberman by Hu to use the text prompt length as an attribute of the text prompts or the image generation prompt attribute. The motivation to modify Aberman by Hu is "Use of known technique to improve similar devices (methods, or products) in the same way".
Regarding claim 14, Aberman, Lester, and Kowalski teach all the features with respect to claim 11 as outlined above. Further, Lester and Hu teach that the method of claim 11, wherein the training comprises:
generating, using the language model, a predicted image generation prompt (See Lester: Fig. 6, and [0116], "At 604, the computing system can process one or more training examples of the plurality of training examples and a prompt with a pre-trained machine-learned model to gene rate a training output. The plurality of pre-trained parameters for the pre-trained machine-learned model can be fixed during prompt tuning (e.g., the pre-trained machine-learned model can be frozen such that the parameters are not adjusted during training of the prompt parameters). In some implementations, the pre-trained machine-learned model can include a model adapted to generate a text prediction output for text that follows an input text (e.g., the input text can include "the sky is" and the output can be "blue"). Alternatively and/or additionally, the pre-trained machine-learned model may have been trained with text masking (e.g., the input text can include "The man old" and the output can be "is"). The pre-trained machine-learned model can include one or more encoder blocks and one or more decoder blocks. For example, the pre-trained machine-learned model can include an encoder-decoder model such as a transformer model. The prompt can be associated with a particular task, and the particular task may be associated with the one or more training examples");
computing a length of the predicted image generation prompt (See Hu: Fig. 2, and [0064], "In one embodiment, length detector 230 is configured to determine whether text generated by the LLM complies with a word count, sentence/paragraph length, or other constraint on length of the text. In one embodiment, the length detector 230 is a rule engine separate from the LLM that is configured to apply rules or heuristics to validate that the length constraints are satisfied by a text. Length detector 230 checks lengths of words, sentences, paragraphs, other portions of and/or the entirety of the generated response. To determine compliance, length detector 230 may count characters, tokens, words, phrases, or other text elements specified by an instruction, and compare that to a maximum (cap), minimum (floor) or other constraint specified by the instruction. In one embodiment, the constraint on length may be included in the instructions. In one embodiment, whether the constraint on length is a binary true/false analysis");
computing a length loss based on the length (See Hu: Fig. 2, and [0066], "In one embodiment, length loss function 235 is configured to assign a penalty where the length constraint is not satisfied. In one embodiment, assignment of a penalty includes an analysis of the extent to which the length constraint is satisfied, for example with a score for length loss normalized between zero and 1. In one embodiment, the score, penalty, or other value for length loss may be obtained by prompting an additional LLM to generate the value for length loss based on inputs of a length instruction that specifies the length constraint and the generated output. Where multiple length instructions are extracted from the text prompt, the values of length loss for individual length instructions may be averaged to produce a merged, overall value of length loss"); and
updating parameters of the language model (See Lester: Figs. 1A-C, and [0034], "In some implementations, prompt tuning can involve inputting parameters with the input data into the frozen model such that only those parameters are updated. Additionally and/or alternatively, the systems and methods may involve only training an initial learnable layer that either precedes the pre-trained machine-learned model or is an initialization layer of the pre-trained machine-learned model. Therefore, only the initial block (e.g., a small set of parameters at the beginning) may be written and/or overwritten, not the entire model. In some implementations, prompt tuning can include learning vectors for new words and tasks. The parameters may be learned directly based on the label comparison. The prompts can include a plurality of values and/or functions") based on the length loss (See Hu: Fig.1, and [0032], "In one embodiment, LLM fine-tuner 108 is configured to update, adjust, optimize, further train or otherwise fine-tune large language model 130 so as to improve performance of LLM 130 at the task of text generation. In other words, LLM fine-tuner 108 is equipped to tailor a configuration of LLM 130 for more accurate (e.g., human-like) text generation. LLM fine-tuner 108 is configured to fine-tune LLM 130 based on the text generation loss output of text generation loss function 134. The text generation loss outputs combined values of format loss 138, length loss 140, embedding loss 146, narrative loss 142, and repetitive loss 144, for example in a weighted average. In one embodiment, LLM-fine tuner 108 may be configured to generate adjustments 148 to weights (and/or other parameters) of large language model 130. LLM fine tuner 108 is configured to generate adjustments 148 that improve performance of large language model 130 with respect to text generation, based on one or more text samples 116 and the extracted instructions 120 obtained from the text samples 116. LLM fine tuner 108 is configured to produce adjustments 148 to large language model 130 so as to minimize text generation loss function 134 over the course of training". The various loss functions are evaluated and used to update the LLM model).
Claims 5 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Aberman, etc. (US 20250349040 A1) in view of Lester, etc. (US 20230325725 A1), further in view of Kowalski, etc. (US 20210335029 A1), and Harang, etc. (US 20250131261 A1).
Regarding claim 5, Aberman, Lester, and Kowalski teach all the features with respect to claim 1 as outlined above. However, Aberman, modified by Lester and Kowalsk, fails to explicitly disclose that the method of claim 1, wherein: the target token comprises a nonce token used to train the language model.
However, Harang teaches that the method of claim 1, wherein: the target token comprises a nonce token used to train the language model (See Harang: Fig. 1, and [0030], "The tokenizer therefore may provide an intermediate language model system 102 step which converts strings of text into a list of numbers, and vice versa. A salient aspect of security for instructions to language models is the token IDs provided to the language model. Since the translation step of tokenization converts text into numbers, where the numbers are the actual input into the language model 102, purposeful application of tokenization may improve security of the language model. In example, a random special text string 120 may be provided to the tokenizer for mapping to a token ID 130. To tokenize a string, a tokenizer 110 may perform a longest match first, where text is reduced to a single token. The tokenizer 110 may map one or more texts to corresponding tokens. For example, a special text string 120 may be mapped to a special token ID 130. In another example, a standard text string 122 may be mapped to a standard token ID 132. The standard token ID 132 may be used to reference a standard token 142. The mapping of text to tokens may be updated at will as long as the token number, or the token ID, does not change. The text, such as the special text string 120 that maps to a token ID may be changed on demand and changed repeatedly". Note that that random strings mapped to the token ID may be mapped to the nonce tokens, that is, the tokens with random strings are nonce tokens).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention was effectively filed to modify Aberman to have the method of claim 1, wherein: the target token comprises a nonce token used to train the language model as taught by Harang in order to allow a Language model system to operate more efficiently and securely by referencing special tokens encoded with potentially complex instructions (See Harang: Fig. 1, and [0026], "In various embodiments, the language model system 102 may use a number of special text strings 120, special token identifiers (IDs) 130, and special tokens 140 to ensure the security and integrity of the language model system 102. Language model systems 102 may operate more efficiently and securely by referencing special tokens 140 encoded with potentially complex instructions. Special tokens 140 may be used for a number of security applications and may be used in other relevant contexts. The use of special tokens 140 may also increase the difficultly required for malicious users to detect, extract, and influence elements of language models systems 102"). Aberman teaches a method and system that may generate a synthetic image using generative model based on the text prompts extracted from the text inputs and image tokens (image representations) extracted from the image inputs; while Harang teaches a system and method that may generate text prompt template with random strings inserted to provide security. Therefore, it is obvious to one of ordinary skill in the art to modify Aberman by Harang to use the nonce tokens (text prompts with random strings inserted) for language model training. The motivation to modify Aberman by Harang is "Use of known technique to improve similar devices (methods, or products) in the same way".
Regarding claim 8, Aberman, Lester, and Kowalski teach all the features with respect to claim 1 as outlined above. Further, Harang teaches that the method of claim 1, wherein generating the image generation prompt further comprises: appending the target token to the text generation prompt to obtain an annotated text generation prompt, wherein the image generation prompt is generated based on the annotated text generation prompt (See Harang: Fig. 1, and [0029], "In this example, the language model systems 102 includes a tokenizer 110. The tokenization process for language model systems may be used to prevent unwanted manipulation of language model systems 102 by overcoming the lack of separation between system instructions and user inputs in communication channels. Language model systems 102 use the tokenization process because language models 102 do not operate directly on textual input. Instead, language models 102 convert every word, sentence, or piece of text from a prompt template into a series of token IDs through tokenization, and the token IDs are used as numerical IDs. In one example, standard text strings 122 may be tokenized to standard token ID 132. The token IDs provided from the tokenizer 110 are used by the language model 112 to reference information on tokens in an embedding layer, such as a list of numbers learned by the language model 102. In an example, standard token ID 132 may be used to reference standard token 142. The token IDs reference by the language model 112 may be converted by the tokenizer 110, or detokenized, back into a string of text and may be provided as an output 114. The tokenized input may be provided to language model 112, such as in an auto-regressive manner, where the language model 112 predicts the next token, appending the next token ID to the list, and continues predicting and appending until reaching a stopping criteria. The language model 112 may reference the special tokens 140 based on the provided special token IDs 130 and may reference the standard tokens based on the provided standard token IDs 132").
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights