Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 05/13/26 have been fully considered but they are not persuasive.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 7-8, 10-13, 21-22, 24-29 and 31-34 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 7, 21 and 28 recite limitations “terminology related to…a further textual description of the customized image” and “determines the terminology corresponding to…with the set of tokens that define the appearance of the customized image using the further textual description of the customized image and the terminology”. The Examiner has attempted to discern the corresponding support in applicant’s specification, and finds the support to be limited to paragraph [0084]; however, this support does not rise to the level of supporting the claim language. MPEP 2163 II A (b) states “To comply with the written description requirement of 35 U.S.C. 112, para. 1, or to be entitled to an earlier priority date or filing date under 35 U.S.C. 119, 120, or 365(c), each claim limitation must be expressly, implicitly, or inherently supported in the originally filed disclosure. When an explicit limitation in a claim "is not present in the written description whose benefit is sought it must be shown that a person of ordinary skill would have understood, at the time the patent application was filed, that the description requires that limitation.” As it is readily apparent that the recited limitations are far more explicitly detailed and limited in scope than the language of the written description, it is clear that the written description cannot support these limitations. In short, the examiner can find no basis whatsoever for concluding that the Applicant was in possession of the claimed invention including the above mentioned specific limitations at the time of filing.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7-8, 10-13, 21-22, 24-29 and 31-34 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 7, 21 and 28 recite limitations, as discussed above, do not have support within the specification. It is unclear to the examiner what the additional limitations require. For example, “a natural language instruction that includes terminology related to the textual descriptions of the plurality of elements, a plurality of customization images depicting the plurality of elements, and a further textual description of the customized image”. It is unclear what it meant by terminology related to the plurality of customization images depicting the plurality of elements. The claim does not specify whether the terminology describes attributes of the images or otherwise corresponds to metadata associated with the images. As a result, the functional relationship between the textural language and the image-based element is ambiguous. In addition, the phrase “a further textural description of the customized image” is indefinite because it is unclear how this further textual description differs from the natural language instruction or whether it refers to a separate description or a modification of the natural language instruction. Therefore, the scope of this limitation is unclear. In addition, the claim merely recites “natural language instruction” but does not specify how this instruction is used to generate the plurality of customized images. As a result, the function of the instruction is unclear. Because the claim does not recite any functional relationship between the natural language instruction and the customized images, the scope of the limitation is ambiguous. As the examiner is unable to derive a definite interpretation in line with the specification, examination in view of the prior art is precluded.
Furthermore, “a Natural Language Processing (NLP) layer that determines the terminology corresponding to the textual descriptions of each of the plurality of elements with the set of tokens that define the appearance of the customized image using the further textual description of the customized image and the terminology”. It is unclear how the natural language processing layer determines terminology corresponding to the textual description…with the set of tokens. The phrase with the set of tokens fails to specify a clear functional relationship between the terminology and the set of tokens, as a result, it is unclear whether the tokens are used to determine the terminology or the terminology are mapped into the tokens. In addition, the claim limitation does not clearly define the meaning of the terminology in relation to the set of tokens that defines the appearance of the customized image. Because terminology usually refers to specialized words or phrases and tokens usually refer to a structured representation of data. The claim limitation fails to connect how these distinct elements interact with one another within the NLP layer. Additionally, the limitation recites the further textual description of the customized image and the terminology without clearly defining how either the textual description or the terminology is used in determination of tokens. This introduced further ambiguity regarding how data are processed in the NLP layer. As the examiner is unable to derive a definite interpretation in line with the specification, examination in view of the prior art is precluded.
Claim Objections
Claim 28 is objected to because of the following informalities:
Claim 28 3rd limitation recites determines the terminology corresponding to each the textual descriptions of of the plurality of elements; it should be changed to determines the terminology corresponding to each of the plurality of elements.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 7, 10-13, 21 and 24-27 are rejected under 35 U.S.C. 103 as being unpatentable over Green (US 2024/0282130) in view of Endras et al. (US 2019/0294878) in view of Ding et al. (CogView: Mastering Text-to-Image Generation via Transformers, 35th Conference on Neural Information Processing System, 2021) in view of Ruiz et al. (DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, Computer Vision and Pattern Recognition, 2022) in view of Park et al. (US 2022/0374602).
Regarding claim 7, Green discloses a method of generating a customized image with an image- generating artificial intelligence engine (Green, [0006], “a method for image generation”. In addition, in paragraph [0022], “modification of an image using an artificial intelligence (AI) generation model”), the method comprising:
receiving a command from a user for the customized image (Green, [0048], “input request 204 could include an image of the rocket”), the command comprising textual description of a plurality of elements that the user wants to include in the customized image (Green, [0048], “the text can say “make the rocket wider” or “add more flames” or “make it stronger””);
the image-generating artificial intelligence engine (Green, [0022], “modification of an image using an artificial intelligence (AI) generation model”);
submitting the image and text to the image-generating artificial intelligence engine to generate the customized image (Green, [0042], “the generation of an output image, graphics, and/or three-dimensional representation by an image generation AI (IGAI), can include one or more artificial intelligence processing engines and/or models”); and
receiving from the image-generating artificial intelligence engine the customized image that includes a depiction of the plurality of elements (Green, [0031], “a display 131 configured for displaying an interface for purposes of facilitating interaction with a user, and the original image 101 and/or the modified image 102”. In addition, in paragraph [0043], “The IGAI is therefore a custom tool that is engineered to processing specific types of input and render specific types of outputs. When the IGAI is customized, the machine learning and deep learning algorithms are tuned to achieve specific custom outputs”).
Green does not expressly disclose “a natural language instruction”;
Endras et al. (hereinafter Endras) discloses a natural language instruction that includes terms related to textual description of a plurality of elements (Endras, [0161], “an NLP model that is trained to identify specific terms (e.g., to identify terms such as terms related to damage (e.g., “scratch”, “dent”, “chip”, “tear”, etc.) and terms related to location on the vehicle (e.g., “door”, “fender”, “window”, “seat”, “hood”, “engine”, etc.)”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to use Endras’s NLP model to identify terms related to Green’s user text input. The motivation for doing so would have been enabling fast identification of relevant information by analyzing the intent behind user queries.
Green as modified by Endras does not expressly disclose “a fine-tuning mechanism comprising a set of tokens defining an appearance of the person or the object”;
Ding et al. (hereinafter Ding) discloses generating a fine-tuning mechanism comprising a set of tokens generated according to a plurality of customization images defining an appearance of an object (Ding, Figure 3: The framework of CogView illustrates a set of tokens defining an appearance of an object. In addition, in 3.4 Industrial Fashion Design, [0002], “We train a 3B-parameter model on about 10 million fashion-caption pairs, using 50x50 VQGAN image tokens and decodes them into 800 x 800 pixels. Figure 8 shows samples of CogView for fashion design, which has been successfully deployed to Alibaba Rhino fashion production”);
generating the fine-tuning mechanism for the image-generating artificial intelligence engine (Ding, Fig. 3 illustrates text tokens and image tokens are send to Transformer (GPT));
the object tokenized in the fine-tuning mechanism (Ding, Fig. 3 illustrates image tokenizer).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to perform Green’s image modification using Ding’s image tokens defining an appearance of an object. The motivation for doing so would have been providing simplified and compressed way to represent image content.
Green as modified by Endras and Ding does not expressly disclose “utilizes its underlying training set to determine how the customized image should appear based on the command and further utilizes the set of tokens and the NLP layer of the fine-tuning mechanism to generate the customized image”;
Ruiz et al. (hereinafter Ruiz) discloses a plurality of customization images comprising depiction of plurality of elements that user wants to include in a customized image (Ruiz, Figs. 2-3, 5 and 11 illustrate a plurality of input images (customization images) comprising depiction of plurality of elements that user wants to include in a output (customized) image),
terminology related to the plurality of elements (Ruiz, 1 Introduction, [0004], “We first fine-tune the low-resolution text-to image model with the input images and text prompts containing a unique identifier followed by the class name of the subject”. The class name reads on terminology related to the plurality of elements);
an image-generating artificial intelligence engine (Ruiz, 1 Introduction, [0003], “a new approach for “personalization” of text-to-image diffusion models”) utilizes its underlying training set to determine how the customized image should appear based on a command (Ruiz, 4 Method, [0003], “we use the pre-trained Imagen model as the base model [56]”. Fig. 3) and further utilizes the set of tokens (Ruiz, 3 Preliminaries, [0004], “transform a text prompt P into a conditioning embedding c, the text is first tokenized using a tokenizer f using a learned vocabulary”) and the NLP layer of a fine-tuning mechanism (Ruiz, 3 Preliminaries, [0004], “Language models like T5-XXL generate embeddings of a tokenized text prompt, and vocabulary encoding is an important pre-processing step for prompt embedding…Finally, the text-to-image diffusion model is directly conditioned on c”. T5-XXL is considered a natural language processing layer for a fine-tuning mechanism. In addition, Fig. 3 High level method including returns a fine-tuned text-to-image model) to generate the customized image specific to the user (Ruiz, Fig. 5);
submitting the fine-tuning mechanism (Ruiz, 4 Method, [0001], “implant the subject instance into the output domain of the model and to bind the subject with a unique identifier”. In addition, in section 4.1 Representing the subject with a rare-token identifier, [0001], “Our goal is to “implant” a new (key, value) pair into the diffusion model’s “dictionary” such that, given the key for our subject, we are able to generate fully-novel images of this specific subject with meaningful semantic modifications guided by a text prompt”) and the command (Ruiz, Fig. 4 illustrates Given ∼ 3- 5 images of a subject) to the image-generating artificial intelligence (Ruiz, 1 Introduction, [0003], “a new approach for “personalization” of text-to-image diffusion models”) to execute the command and generate the customized image (Ruiz, Figs. 3 and 4);
the customized image that includes the depictions of the plurality of elements (Ruiz, Figs. 2-3, 5 and 11).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to use Ruiz’s fine-tuning text-to-image diffusion models to generate Green’s customized images. The motivation for doing so would have been enabling personalized text-to-image generation using diffusion models.
In addition, Green as modified by Endras, Ding, Ruiz does not expressly disclose “a Natural Language Processing (NLP) layer that determines the terminology corresponding to each of the plurality of elements with the set of tokens”;
Park discloses a natural language processing (NLP) layer that determines terminology corresponding to element with a token (Park, [0051], “the NLP embedding engine 140 processes the tokens of the input 130 from the tokenizer 125 to generate word embeddings 122 and character embeddings 124”. Each generated embedding corresponds to a respective token, and each token corresponds to a respective element in the input. Therefore, generating word embeddings and character embeddings reads on determining terminology corresponding to the tokens).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the natural language processing layer of Park into the customized image generation system of Green as modified by Endras, Ding and Ruiz. The motivation for doing so would have been enabling the AI engine to understand which image elements to modify.
Regarding claim 10, Green discloses submitting to the image-generating artificial intelligence engine via an Application Programming Interface (API) of the image-generating artificial intelligence engine (Green, [0044], “the IGAI can be used online via one or more Application Programming Interface (API) calls”).
Green as modified by Endras and Ding with the same motivation from claim 7 discloses the fine-tunning mechanism (Ding, Figure 3: The framework of CogView illustrates a set of tokens defining an appearance of an object).
Regarding claim 11, Green discloses a textual command entered by a user describing the customized image to be generated (Green, [0048], “the text can say “make the rocket wider” or “add more flames” or “make it stronger””);
Green as modified by Endras, Ding, Ruiz and Park with the same motivation from claim 7 discloses generating the NLP layer based on the terminology used in a textual (Park, [0088], “The natural language content is processed by a tokenizer to break down the natural language content into tokens (step 520). The tokens are provided to an NLP embedding engine to generate word embeddings and/or character embeddings (step 530)”).
Regarding claim 12, Green discloses operating a client application specific to the image-generating artificial intelligence engine (Green, [0031], “a user interface 130 is configured to facilitate implementing changes to an image 101 based on user feedback. In particular, the user interface may include a display 131 configured for displaying an interface for purposes of facilitating interaction with a user, and the original image 101 and/or the modified image 102”. Fig. 1A).
Green as modified by Endras and Ding with the same motivation from claim 7 discloses generate the fine-tuning mechanism (Ding, Figure 3: The framework of CogView illustrates a set of tokens defining an appearance of an object).
Regarding claim 13, Green discloses submitting to the image-generating artificial intelligence engine with a textual command entered by a user describing the customized image to be generated (Green, [0042], “the generation of an output image, graphics, and/or three-dimensional representation by an image generation AI (IGAI), can include one or more artificial intelligence processing engines and/or models”. In addition, in paragraph [0048], “the text can say “make the rocket wider” or “add more flames” or “make it stronger””).
Green as modified by Endras and Ding with the same motivation from claim 7 discloses submitting the fine-tuning mechanism to the image-generating artificial intelligence engine (Ding, Fig. 3 illustrates text tokens and image tokens are send to Transformer (GPT)).
Regarding claim 21, Green discloses a method of generating a customized image with an image-generating artificial intelligence engine (Green, [0006], “a method for image generation”. In addition, in paragraph [0022], “modification of an image using an artificial intelligence (AI) generation model”), the method comprising:
the image-generating artificial intelligence engine retains its training set while adapting to the customized image following generation of the customized image (Green, [0046], “the input 206 is configured to convey the intent of the user that wishes to utilize the IGAI to generate some digital content…the data set used to train the IGAI and input 206 can be used to customized the way artificial intelligence, e.g., deep neural networks process the data to steer and tune the desired output image”. In addition, in paragraph [0051], “the original image may have been generated using an IGAI model implementing latent diffusion in response to user input (e.g., description of a desired image that is encoded into a latent space vector). For example, the image is generated by the IGAI processing model of FIGS. 1 and FIGS. 2A-2C. The extracted features are relevant to generating images using latent diffusion, and in one embodiment, an AI model is configured for extracting those features”. The AI model keeps the training data set, and adjust to create a custom output image based on user input. Moreover, the AI model is designed to extract features from the images generated by the IGAI and continues to adapt to modifications even after image generation is complete).
The remaining limitations recite in claim 21 are similar in scope to the method recited in claim 7 and therefore are rejected under the same rationale.
Regarding claims 24-27, claims 24-27 recite method steps that are similar in scope to the method steps recited in claims 10-13 and therefore are rejected under the same rationale.
Claim 8 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Green (US 2024/0282130) in view of Endras et al. (US 2019/0294878) in view of Ding et al. in view of Ruiz et al. in view of Park et al. (US 2022/0374602), as applied to claims 7 and 21, in further view of Gaither et al. (US 2014/0047429).
Regarding claim 8, Green teaches the image-generating artificial intelligence engine; Green as modified by Endras and Ding with the same motivation from claim 7 teaches the fine-tunning mechanism; Green as modified by Endras, Ding, Ruiz and Park does not expressly disclose “an add-in or plug-in”;
Gaither et al. (hereinafter Gaither) discloses an add-in or plug-in for a program (Gaither, [0013], “Plug-ins are generally used for customizing the functionality of a software application. Applications generally support plug-ins for various reasons. For example, plug-ins can be used to enable third-party developers to, for example, create new features and/or functionality that extends an application”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to perform Green as modified by Endras and Ding’s fine-tunning for the image-generating artificial intelligence engine using the concept of Gaither’s plug-in. The motivation for doing so would have been allowing users to customize and expand existing software capabilities as needed.
Regarding claim 22, claim 22 recites method that is similar in scope to the method recited in claim 8 and therefore is rejected under the same rationale.
Claims 28 and 31-34 are rejected under 35 U.S.C. 103 as being unpatentable over Green (US 2024/0282130) in view of More et al. (US 2019/0303403) in view of Saharia et al. (Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, 36th Conference on Neural Information Processing Systems, 2022) in view of Endras et al. (US 2019/0294878) in view of Ding et al. in view of Ruiz et al. in view of Park et al. (US 2022/0374602).
Regarding claim 28, Green discloses a method of generating a customized image with an image-generating artificial intelligence engine (Green, [0006], “a method for image generation”. In addition, in paragraph [0022], “modification of an image using an artificial intelligence (AI) generation model”), the method comprising:
a processor (Green, [0008], “a processor”);
a memory storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations (Green, [0008], “memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method”);
Green does not expressly “a background image”;
More et al. (hereinafter More) discloses a background image (More, [0126], “a list of visual content elements (e.g., background elements such as landscapes, foreground elements such as people, animals, inanimate objects, etc.)”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to perform Green’s image modification using the concept of More’s foreground and background elements. The motivation for doing so would have been providing distinct foreground and background layers in order to easily modify individual elements without affecting the entire image.
Green as modified by More does not expressly disclose “the customized image that includes a depiction of the product image and the background image”;
Saharia et al. (hereinafter Saharia) discloses customized image that includes a depiction of a product image and a background image (Saharis, Fig. 1 illustrates a high contrast portrait of a very happy fuzzy panda dressed as a chef in a high-end kitchen making dough. There is a painting of flowers on the wall behind him).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the concept of Saharia’s photorealistic Text-to-Image Diffusion Models including foreground and background images in the image generation system, as taught by Green as modified by More. The motivation for doing so would have been allowing more realistic scene composition.
The remaining limitations recite in claim 28 are similar in scope to the method recited in claim 7 and therefore are rejected under the same rationale.
Regarding claims 31-34, claims 31-34 recite method steps that are similar in scope to the method steps recited in claims 10-13 and therefore are rejected under the same rationale.
Claim 29 is rejected under 35 U.S.C. 103 as being unpatentable over Green (US 2024/0282130) in view of More et al. (US 2019/0303403) in view of Saharia et al. in view of Endras et al. (US 2019/0294878) in view of Ding et al. in view of Ruiz et al. in view of Park et al. (US 2022/0374602), as applied to claim 28, in further view of Gaither et al. (US 2014/0047429).
Regarding claim 29, claim 29 recites method that is similar in scope to the method recited in claim 8 and therefore is rejected under the same rationale.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYLE ZHAI whose telephone number is (571)270-3740. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ke Xiao can be reached at (571) 272 - 7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KYLE ZHAI/ Primary Examiner, Art Unit 2612
Prosecution Insights