DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “optimal value” in claims 1 and 11 is a relative term which renders the claim indefinite. The term “optimal value” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term optimal value can make one question what is consider an optimal value? Is it a value that is above a predefined threshold? The specification doesn’t have a clear definition of what is defined as optimal which renders the claim indefinite.
Claims 2-10 are rejected by virtue of dependency on claim 1 and claims 12-20 are rejected by virtue of dependency on claim 11.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-6, 10-16 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Gal et al. ("An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion") in view of Gandelsman et al. US PG-Pub (US 20240161462 A1).
Regarding Claim 1, Gal teaches a system comprising: one or more processors(Page 5, Implementation details, Paragraph 1, “Our experiments were conducted using 2×V100 GPUs with a batch size of 4.”, discloses a GPU used to perform the experiment.); and a memory storing instructions that, when executed by at least one processor in the one or more processors, cause the at least one processor to perform operations for ((Page 5, Implementation details, Paragraph 1, discloses a GPU which inherently would be coupled to a memory that executes a program to perform the experiment.)generating an image having a target visual characteristic(See Figure 1 input images with a text prompt are fed into a pre-trained text to image model to generate an image with the text prompt.), the operations comprising: accessing an image request including a piece of seed data identifying a target visual characteristic (Page 5, Textual inversion, Paragraph 1, “To find these new embeddings, we use a small set of images (typically 3-5), which depicts our target concept across multiple settings such as varied backgrounds or poses. We find v∗ through direct optimization, by minimizing the LDM loss of Equation (1) over images sampled from the small set. To condition the generation, we randomly sample neutral context texts, derived from the CLIP ImageNet templates (Radford et al., 2021). These contain prompts of the form “A photo of S∗”, “A rendition of S∗”, etc.” discloses using prompts to describe a target visual characteristic for the input image.); generating a targeted image dataset based on the piece of seed data, the targeted image dataset including multiple images with the target visual characteristic (Page 6, 4 Qualitative comparisons and application, Paragraph 1-2, “We begin by demonstrating our ability to capture and recreate variations of an object using a single pseudo word. In Figure 3 we compare our method to two baselines: LDM guided by a human caption and DALLE-2 guided by either a human caption or an image prompt. Captions were collected using Mechanical Turk. An notators were provided with four images of a concept and asked to describe it in a manner that could allow an artist to recreate it. We asked for both a short (≤ 12 words) and a long (≤ 30 words) caption. In total, we collected 10 captions per concept — five short and five long. Figure 3 shows multiple results generated with a randomly chosen caption for each setup.”, in this section of the prior art an input dataset with captions is input into the text to image model and output is the image with the target visual characteristic as seen in figure 3. )inputting the targeted image dataset and a text input including a custom token into a constrained text to image model configured to determine a text embedding for the custom token(Page 5, Text embeddings. Paragraph 1-2, “Typical text encoder models, such as BERT, begin with a text processing step (Figure 2, left). First, each word or sub-word in an input string is converted to a token, which is an index in some pre-defined dictionary. Each token is then linked to a unique embedding vector that can be retrieved through an index-based lookup. These embedding vectors are typically learned as part of the text encoder cθ. In our work, we choose this embedding space as the target for inversion. Specifically, we designate a place holder string, S∗, to represent the new concept we wish to learn”, in this section a text embedding of the user input is used to generate a token and shown in Figure 2 shows the text-embedding and inversion process in which the text prompt is transformed into a token and used as an input to the text to image model. ), the custom token including a text representation of the target visual characteristic (Figure 2 shows a prompt “a photo of S” with S being the target visual characteristic.) , the constrained text to image model configured to determine the text embedding using a training process that determines an optimal value for the text embedding while constraining one or more other trainable aspects of the text to image model to one or more pre-trained values (Page 8, First paragraph, “In Figures 1 and 4 we show our ability to compose novel scenes by incorporating the learned pseudo-words into new conditioning texts. For each concept, we show exemplars from our training set, along with an array of generated images and their conditioning texts. As our results demonstrate, the frozen text-to-image model is able to jointly reason over both the new concepts and its large body of prior knowledge, bringing them together in a new creation. Importantly, despite the fact that our training goal was generative in nature, our pseudo-words still encapsulate semantic concepts that the model can then leverage. For example, observe the bowl’s ability (row four) to contain other objects like food, or the ability to preserve the Furby’s bird-like head and crown while adapting his palette to better match a prompt (album cover, row three).”, in this section the pre-trained frozen text-to-image model is able to determine the text embedding and generate a optimal image pertaining to the prompt with the image. );
generating, with the constrained text to image model, a target image based on the custom token, the target image (See Figure 3 where output images from the text to image model are generated with the custom token.)
Gal does not explicitly teach generating, with the constrained text to image model, a target image based on the custom token, the target image including one or more pixels generated based on the target visual characteristic
Gandelsman teaches generating, with the constrained text to image model, a target image based on the custom token, the target image including one or more pixels generated based on the target visual characteristic ([0061] “In some cases, the system converts the text prompt (or other guidance) into a conditional guidance vector or other multi-dimensional representation. For example, text may be converted into a vector or a series of vectors using a transformer model, or a multi-modal encoder. In some cases, the encoder for the conditional guidance is trained independently of the diffusion model.”
[0062] “In some cases, a noise map that includes random noise is initialized. The noise map may be in a pixel space or a latent space. By initializing an image with random noise, a variation of the image including the content described by the conditional guidance can be generated. In some cases, multiple variations of the image can be generated by initializing multiple noise maps.”, ¶[0061]-¶[0062] disclose using the text-to-image model to process the text prompt and generate a target image with the desired style in pixel space using noise maps)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Gal with Gandelsman in order to process the image in pixel space with noise maps. One skilled in the art would have been motivated to modify Gal in this manner in order for image generation using machine learning. (Gandelsman, ¶[0001])
Regarding Claim 2, the combination of Gal and Gandelsman teach the system of claim 1, where Gal further teaches wherein the operations further comprise: determining, with the constrained text to image model, text embeddings for multiple new custom tokens based on multiple new targeted image datasets(Page 5, Text Embeddings, Paragraph 1-2, “In our work, we choose this embedding space as the target for inversion. Specifically, we designate a place holder string, S∗, to represent the new concept we wish to learn. We intervene in the embedding process and replace the vector associated with the tokenized string with a new, learned embedding v∗, in essence “injecting” the concept into our vocabulary. In doing so, we can then compose new sentences containing the concept, just as we would with any other word.”, this section of the prior art discloses generating new tokenized strings related to the embedding to generate a new targeted image data set.); accessing a memory storing a visual vocabulary including the multiple new custom tokens(Page 5, Textual inversion, Paragraph 1, “To find these new embeddings, we use a small set of images (typically 3-5), which depicts our target concept across multiple settings such as varied backgrounds or poses. We find v∗ through direct optimization, by minimizing the LDM loss of Equation (1) over images sampled from the small set. To condition the generation, we randomly sample neutral context texts, derived from the CLIP ImageNet templates (Radford et al., 2021). These contain prompts of the form “A photo of S∗”, “A rendition of S∗”, etc. The full list of templates is provided in the supplementary materials.”, discloses accessing templates of context text from a memory.); and generating, with the constrained image model, a target image based on one or more of the multiple new custom tokens(Fig. 4 shows the generated target image with the custom token using text-guided image synthesis.).
Regarding Claim 3, the combination of Gal and Gandelsman teach the system of claim 2, where Gal further teaches wherein the operations further comprise determining one or more token attributes for each of the new custom tokens based on one or more image attributes of one or more images included in the new targeted image dataset used to determine each of the new custom tokens. (Page 5, Textual inversion, Paragraph 1, “To find these new embeddings, we use a small set of images (typically 3-5), which depicts our target concept across multiple settings such as varied backgrounds or poses. We find v∗ through direct optimization, by minimizing the LDM loss of Equation (1) over images sampled from the small set. To condition the generation, we randomly sample neutral context texts, derived from the CLIP ImageNet templates, discloses generating new embedding which depict the target visual characteristic for the output images and Figure 4 shows Text-guided synthesis with the text prompt generated tokens used to create the new dataset.);
Regarding Claim 4, the combination of Gal and Gandelsman teach the system of claim 3, wherein the operations further comprise identifying a new target visual characteristics of a desired image from a piece of seed data included in a new image request (¶[0030], “In an example process, user 100 provides an input image and a text prompt to the system via a user interface on user device 105. In some cases, the text prompt describes a modification to the image. In one example, the input image is an image of a specific person, the text prompt is “a person with a mustache”.”, ¶[0030] discloses a user can provide a text prompt describing a modification to the image.); matching the new target visual characteristics to a token attribute for at least one of the new custom tokens([0070] According to some embodiments, operation 715 includes an optimization process. The optimization can be run on either the text embedding space of the text encoder or on the weights of the model individually. In some cases, optimization that is run on the text embedding is used to find an image that best matches the given image in the vicinity of the target text embedding. For example, the text embedding is optimized to reconstruct an image that best matches the given image, discloses matching the text embedding such that it matches the visual characteristics of the desired output image.); and inputting a text input into the constrained text to image model to generate a new target image having one or more pixels generated based on the new target visual characteristic that aligns with the token attribute. ([0031] Image processing apparatus 110 receives the input and processes it using a diffusion model. In some cases, the processing synthesizes a variation of the input image in a way described by the text prompt. Then, the system provides the output image to user 100 through the user interface. For example, given an input image of a specific person and text prompt “a person with a mustache” (or wearing glasses, etc.), a generated variation of the input image can be an image that represents the identity of that specific person and depicts the person with a mustache (or wearing glasses)., discloses inputting the text into a model and generating the new desired output image with the text prompt description.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Gal with Gandelsman in order to generate an output image that aligns with the text token. One skilled in the art would have been motivated to modify Gal in this manner in order for image generation using machine learning. (Gandelsman, ¶[0001])
Regarding Claim 5, the combination of Gal and Gandelsman teach the system of claim 1, where Gal further teaches wherein the operations further comprise inputting multiple different text inputs into the constrained image to text model to generate, multiple images including the one or more target visual characteristics, each of the multiple different text inputs including the custom token. (Figure 1, “We find new pseudo-words in the embedding space of a pre-trained text-to-image model which describe specific concepts. (right) These pseudo-words can be composed into new sentences, placing our targets in new scenes, changing their style or composition, or ingraining them into new products.”, Fig. 1 shows an input image with multiple text inputs to sample images with the output image being the text prompt of the desired image.)
Regarding Claim 6, the combination of Gal and Gandelsman teach the system of claim 5, where Gal further teaches wherein each of the multiple images include a different variation of a subject included in the image request. (Fig.4 shows a text-guided synthesis with the text to image model receiving a prompt for sample inputs and generating multiple output images with the input sample.)
Regarding Claim 10, the combination of Gal and Gandelsman teach the system of claim 1, where Gal further teaches wherein the custom token includes a textual representation of the target visual characteristic. (Page 5, Text Embeddings, Paragraph 1-2, “In our work, we choose this embedding space as the target for inversion. Specifically, we designate a place holder string, S∗, to represent the new concept we wish to learn. We intervene in the embedding process and replace the vector associated with the tokenized string with a new, learned embedding v∗, in essence “injecting” the concept into our vocabulary. In doing so, we can then compose new sentences containing the concept, just as we would with any other word.”, this section of the prior art discloses generating new tokenized strings related to the embedding to generate a new targeted image data set.)
Regarding Claim 11, claim 11 is considered a method claim substantially corresponding to claim 1. Please see the discussion of claim 1 above for a discussion of similar limitations
Regarding claim 12, it is substantially similar to claim 2 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding claim 13, it is substantially similar to claim 3 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding claim 14, it is substantially similar to claim 4 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding claim 15, it is substantially similar to claim 5 respectively, and is rejected in the same manner, the same art, and reasoning applying.
subject included in the image request.
Regarding claim 16, it is substantially similar to claim 6 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding claim 20, it is substantially similar to claim 10 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Claims 7-9 and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Gal et al. ("An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion") in view of Gandelsman et al. US PG-Pub(US 20240161462 A1) in view of Kulkarni et al. US PG-Pub(US 20180004847 A1).
Regarding Claim 7, while the combination of Gal and Gandelsman teach the system of claim 5, they do not explicitly teach wherein the operations further comprise inputting each of the multiple images into a performance model of an image evaluator configured to determine a ranking for each of the multiple images based on a predicted performance of each of the multiple images; accessing the ranking for each of the multiple images determined by the one or more performance models of an image evaluator; and recommending at least one of the multiple images based on the ranking.
Kulkarni teaches wherein the operations further comprise inputting each of the multiple images into a performance model of an image evaluator configured to determine a ranking for each of the multiple images based on a predicted performance of each of the multiple images(¶0033] discloses inputting the images into a creative ranking module to generate a ranking score for each image.); accessing the ranking for each of the multiple images determined by the one or more performance models of an image evaluator(¶[0033] discloses the creative ranking module selects an optimal image with the highest score.);and recommending at least one of the multiple images based on the ranking([0034] “The creative assembly module 230 retrieves the selected optimal creatives of the content item, where each selected creative has the highest prediction score among multiple creatives of the same creative type for a target user, and fully assembles the creative into a DCO content item to be shown to the target user.”, discloses retrieving the images that have the highest score.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Gal and Gandelsman with Kulkarni in order to measure the performance evaluation of the model. One skilled in the art would have been motivated to modify Gal and Gandelsman in this manner in order to dynamically creating content items from a set of individual content components for a target audience. (Kulkarni, ¶[0001])
Regarding Claim 8, the combination of Gal, Gandelsman and Kulkarni teach the system of claim 7, where Kulkarni further teaches wherein the operations further comprise generating a piece of content that includes the at least one recommended image(¶[0033]-¶[0034] disclose determining content items that have a high recommendation to a user.); and provide the piece of content to a device configured to display the piece of content at a specific location or domain on a publication network. (¶[0034] discloses displaying the content item to the user through an interface.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Gal and Gandelsman with Kulkarni in order to generate a content item and recommend it to a user. One skilled in the art would have been motivated to modify Gal and Gandelsman in this manner in order to dynamically creating content items from a set of individual content components for a target audience. (Kulkarni, ¶[0001])
Regarding Claim 9, the combination of Gal, Gandelsman and Kulkarni teach the system of claim 7, where Gal further teaches wherein the operations further comprise providing the at least one recommended image to a device configured to display the at least one recommended image in a graphical user interface (GUI) in response to the image request. (See figure 1 where the user inputs a prompt and the model outputs an image related to the prompt from the user.)
Regarding claim 17, it is substantially similar to claim 7 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding claim 18, it is substantially similar to claim 8 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Regarding claim 19, it is substantially similar to claim 9 respectively, and is rejected in the same manner, the same art, and reasoning applying.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HAN D HOANG whose telephone number is (571)272-4344. The examiner can normally be reached Monday-Friday 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JOHN M VILLECCO can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAN HOANG/Primary Examiner, Art Unit 2661
Prosecution Insights