DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted is considered by the examiner.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 7, 9-12, 13-16 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gadelha et al. (US Publication Number 2025/0061650 A1, hereinafter “Gadelha”) in view of Chiang et al. (US Publication Number 2025/0173519 A1).
(1) regarding claim 1:
As shown in fig. 1, Gadelha disclosed an illustration system (para. [0013], note that the present disclosure describes efficient and user-friendly image processing systems configured to generate accurate (e.g., user intended) images using text information and 3D scene geometry information provided by a use) comprising:
a memory storing instructions that, when executed by a processor (para. [0017], note that the image processing system 100 may include user device 110, server 115, cloud 120, and database 125, which may perform and/or support one or more aspects of the image processing system 100), cause the processor to:
generate a drawing from a drawn line and input text by a machine learning (ML) model that includes (para. [0017], note that user 105 provided inputs including geometry information (e.g., 3D modeling information, such as a 3D model of a castle) and text information (e.g., a text prompt, such as “gingerbread castle”). Also see, para. [0024], note that users 105 may draw sketches of a desired scene (e.g., via a 3D modeling application, etc.), and image processing system 100 may automatically generate semantic labels to describe the various objects and elements in the scene drawn by the user 105);
predict a realistic prompt (see explanation below) and a scaling amount (scale, paragraph 19) about the drawing using a large language model (LLM) and estimate a depth map of the drawing using a depth model according to the scaling amount, (para. [0065], note that at operation 610, the system generates, by the 3D modeling application, a depth map of the 3D model based on the 3D edit input. In some cases, the operations of this step refer to, or may be performed by, a 3D modeling application as described with reference to FIG. 2. At operation 615, the system receives, via a text interface, a text prompt from a user, where the text prompt describes a scene corresponding to the 3D model. In some cases, the operations of this step refer to, or may be performed by, a combined interface as described with reference to FIG. 2.); and
render the drawing within a realistic scene by an outpainting model (para. [0046], note that image processing system 300 may generate aesthetically pleasing RGB images from text prompts 325 that are accurate in accordance with user intention based on user provided 3D scenes (e.g., 3D models 310). Users may create a 3D scene (e.g., 3D model 310) in a canvas equipped with 3D controls, and the image processing system 300 may render the scene (e.g., 3D model 310) as a depth map. Using image generation model 305 (e.g., a conditional image generative model) and a text description (e.g., text prompt 325), image processing system 300 may generate an RGB image following the scene geometry and textual guidance. Also see para. [0067]).
Gadelha disclosed most of the subject matter as described as above except for specifically teaching predict a controlnet using a diffusion model; a realistic prompt, the LLM feeds the scaling amount to the depth model; and an outpainting model that is diffusion based using the realistic prompt fed by the LLM and the depth map fed by the depth model.
However, Chiang disclosed predict a controlnet using a diffusion model (para. [0104], note that machine-learned models 120 may include one or more neural radiance field models, one or more diffusion models, and/or one or more autoregressive language models); a realistic prompt (para. [0144], note that the generation may be prompted based on a user selection and/or may be automatically performed (e.g., automatically performed based on one or more conditions), the LLM feeds the scaling amount to the depth model (para. [0114], note that neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks); and an outpainting model that is diffusion based using the realistic prompt fed by the LLM (para. [0145], note that generative models 90 can include one or more autoregressive models (e.g., a machine-learned model trained to generate predictive values based on previous behavior data) and/or one or more diffusion models (e.g., a machine-learned model trained to generate predicted data based on generating and processing distribution data associated with the input data)) and the depth map fed by the depth model (para. [0125], note that the image processing task can be depth estimation, where the image processing output defines, for each pixel in the one or more images, a respective depth value).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art for Gadelha to teach predict a controlnet using a diffusion model; a realistic prompt, the LLM feeds the scaling amount to the depth model; and an outpainting model that is diffusion based using the realistic prompt fed by the LLM and the depth map fed by the depth model. The suggestion/motivation for doing so would have been in order to leverage a generative model to determine an intent of a query based on a chat session history, which can then be leveraged to generate a contextually aware query (para. [0001]). Therefore, it would have been obvious to combine Gadelha with Chiang to obtain the invention as specified in claim 1.
(2) regarding claim 2:
Gadelha further disclosed the illustration system of claim 1 further including instructions to: approximate a three-dimensional (3D) structure of the drawing using the depth map, and the depth model is a neural network (NN) that identifies depth relationships between pixels and objects within the drawing and feeds the outpainting model with priors of the objects (para. [0076], note that during training, guided latent diffusion model 700 may take an original image 705 in a pixel space 710 as input and apply forward diffusion process 730 to gradually add noise to the original image 705 to obtain noisy images 720 at various noise levels).
(3) regarding claim 3:
Gadelha further disclosed the illustration system of claim 2, wherein the instructions to render the drawing further include instructions to:
add lighting and shadows with the outpainting model according to the depth map and a subject associated with the realistic prompt that is output by the LLM, wherein the outpainting model is a stable diffusion model (para. [0056], note that image processing systems may use text prompts 415 to generate image content such as wrapping 2D images around 3D models 405 and determining how light would affect it in the generated output images 430. Also see, para. [0074], note that diffusion models are a class of generative neural networks which can be trained to generate new data with features similar to features found in training data. In particular, diffusion models can be used to generate novel images. Diffusion models can be used for various image generation tasks including image super-resolution, generation of images with perceptual metrics, conditional generation (e.g., generation based on text guidance), image inpainting, and image manipulation).
(4) regarding claim 4:
Gadelha further disclosed the illustration system of claim 3, wherein the instructions to predict the realistic prompt further include instructions to: process subject and concept inputs about the drawing for placing the subject within the realistic scene and predicting the scaling amount (para. [0047], note that a user may provide geometry information for image generation by creating or editing a 3D scene (e.g., 3D model 310) using a canvas equipped with 3D controls (e.g., where 3D controls may allow users to perform various 3D edits 320, which may include rotations of objects/shapes 315, scaling of objects/shapes 315).
Gadelha disclosed most of the subject matter as described as above except for specifically teaching the scaling amount being a realistic scaling amount output by the LLM.
However, Chiang disclosed the scaling amount being a realistic scaling amount output by the LLM (para. [0122], note that the machine-learned model(s) can process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.)).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art for Gadelha to teach the scaling amount being a realistic scaling amount output by the LLM. The suggestion/motivation for doing so would have been in order to leverage a generative model to determine an intent of a query based on a chat session history, which can then be leveraged to generate a contextually aware query (para. [0001]). Therefore, it would have been obvious to combine Gadelha with Chiang to obtain the invention as specified in claim 4.
(5) regarding claim 7:
Gadelha further disclosed the illustration system of claim 1, wherein the realistic prompt describes a subject associated with the drawing within a natural setting and the scaling amount factors relationships between objects within the drawing (para. [0017], note that as an example shown in FIG. 1, image processing system 100 may generate output images (e.g., a castle made of gingerbread cookie) based on user 105 provided inputs including geometry information (e.g., 3D modeling information, such as a 3D model of a castle) and text information (e.g., a text prompt, such as “gingerbread castle”)).
The proposed rejection of claims 1-4 and 7 renders obvious the steps of the non-transitory computer readable medium claims 9-12 and the method claims 13-16 and 19 because these steps occur in the operation of the proposed rejection as discussed above. Thus, the arguments similar to that presented above for claims 1-4 and 7 are equally applicable to claims 9-16 and 19.
Claim(s) 6, 8, 18 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gadelha and Chiang, and further in view of Ackerman et al. (US Publication Number 2024/0242428 A1, hereinafter “Ackerman”).
(1) regarding claim 6:
Gadelha disclosed most of the subject matter as described as above except for specifically teaching wherein the instructions to generate the drawing from the drawn line further include instructions to: process the text by a language model of the ML model to output ideas, wherein the text includes a subject and a concept associated with the ideas; and form the drawing by a neural network (NN) of the ML model according to one of the ideas selected and the drawn line.
However, Ackerman disclosed wherein the instructions to generate the drawing from the drawn line further include instructions to: process the text by a language model of the ML model to output ideas, wherein the text includes a subject and a concept associated with the ideas (para. [0044], note that referring to FIGS. 2B-2D, the text amplification engine 220 may be configured to generate additional text content based on the text content 214 using one or more AI-based techniques (e.g., AI-based chatbots, large language models (LLMs), or other AI-based algorithms) that can generate text based on a given set of inputs); and form the drawing by a neural network (NN) of the ML model according to one of the ideas selected and the drawn line (para. [0050], note that the media content enrichment engine 250 may detect other elements within the media content 216, such as the number of lines, number of corners, color distribution information, and the like. Also see, para. [0091], note that machine learning models may be trained to verify aspects related to the products of the entity, such as to determine whether a proposed product design is feasible to manufacture, matches or fits within the entity's style, or other product design related functions).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach to wherein the instructions to generate the drawing from the drawn line further include instructions to: process the text by a language model of the ML model to output ideas, wherein the text includes a subject and a concept associated with the ideas; and form the drawing by a neural network (NN) of the ML model according to one of the ideas selected and the drawn line. The suggestion/motivation for doing so would have been in order to efficiently generate media content and more specifically, to techniques for generating media content based on text-based inputs (para. [0002]). Therefore, it would have been obvious to combine Gadelha, Chiang and Ackerman to obtain the invention as specified in claim 6.
(2) regarding claim 8:
Gadelha disclosed most of the subject matter as described as above except for specifically teaching wherein the LLM includes a transformer and the realistic scene is synthetic.
However, Ackerman disclosed wherein the LLM includes a transformer and the realistic scene is synthetic (para. [0044], note that the text amplification engine 220 may be configured to generate additional text content based on the text content 214 using one or more AI-based techniques (e.g., AI-based chatbots, large language models (LLMs), or other AI-based algorithms) that can generate text based on a given set of inputs).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the LLM includes a transformer and the realistic scene is synthetic. The suggestion/motivation for doing so would have been in order to efficiently generate media content and more specifically, to techniques for generating media content based on text-based inputs (para. [0002]). Therefore, it would have been obvious to combine Gadelha, Chiang and Ackerman to obtain the invention as specified in claim 8.
The proposed rejection of claims 6 and 8 renders obvious the steps the method claims 18 and 20 because these steps occur in the operation of the proposed rejection as discussed above. Thus, the arguments similar to that presented above for claims 6 and 8 are equally applicable to claims 18 and 20.
Claim(s) 5 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gadelha and Chiang, and further in view of Johnson et al. (NPL, “Image Generation from Scene Graphs”, 2018).
(1) regrading claim 5:
Gadelha further disclosed the illustration system of claim 4, further including instructions to: segment the drawing to identify boundaries with a segmentation model and extracting edges from the drawing using an edge model (para. [0024], note that some aspects of output image generation by image processing system 100 may include generating (e.g., drawing) segmentation maps, manipulating scenes, labeling segments with labels (e.g., such as sky, sea, sand, snow, etc.), among other processing tasks. Also see para. [0037], training the system may involve supplying values for the inputs and modifying edge weights and activation functions (algorithmically or randomly) until the result closely approximates a set of desired outputs).
Gadelha disclosed most of the subject matter as described as above except for specifically teaching to render an estimated sketch of the drawing by computing an intersection between the boundaries and the edges; and regenerate the drawing with a modified form of the estimated sketch.
However, Johnson teaches to render an estimated sketch of the drawing by computing an intersection between the boundaries and the edges (page 7, 4.4 Object Localization, para. [0001], note that to looking at images, we can also inspect the bounding boxes predicted by our model. One measure of box quality is high agreement between predicted and ground-truth boxes; in Table 2 we show the object recall of our model at two intersection-over-union thresholds); and regenerate the drawing with a modified form of the estimated sketch (page 6, 4. Experiments, para. [0001], note that in our experiments we aim to show that our method generates images of complex scenes which respect the objects and relationships of the input scene graph).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the language model is one of a large language model (LLM) and a language transformer model and the realistic scene is synthetic. The suggestion/motivation for doing so would have been in order to efficiently generating images from scene graphs, enabling explicitly reasoning about objects and their relationships and to ensure realistic output (para. [0002]). Therefore, it would have been obvious to combine Gadelha, Chiang and Johnson to obtain the invention as specified in claim 5.
The proposed rejection of claim 5 renders obvious the steps the method claims 17 because these steps occur in the operation of the proposed rejection as discussed above. Thus, the arguments similar to that presented above for claim 5 is equally applicable to claim 17.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Saharia et al. (US Publication Number 2023/0377226 A1) disclosed an image generation system implemented as computer programs on one or more computers in one or more locations that generates an image from a conditioning input using a text encoder neural network and a sequence of generative neural networks.
Lim et al. (NPL, “LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models”, May 2023) disclosed recent advancements in text-to-image diffusion models have yielded impressive results in generating realistic and diverse images.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Hilina K Demeter whose telephone number is (571) 270-1676.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y. Poon could be reached at (571) 270- 0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about PAIR system, see http://pari-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HILINA K DEMETER/Primary Examiner, Art Unit 2617
Prosecution Insights