DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 10-12, 14, 16, 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. (US 20260220872 A1, hereinafter Guo) in view of Yang et al. (Yang, Zhengyuan, et al. "Idea2img: Iterative self-refinement with gpt-4v for automatic image design and generation." European conference on computer vision. Cham: Springer Nature Switzerland, 2024.; hereinafter Yang).
Regarding claim 1, Guo discloses a method (Title, Abstract, ¶000-0002, ¶0009-0010, fig. 2, ¶0090-0096, claim 1 and dependents) comprising:
at an electronic device (200, fig. 5, ¶0089, ¶0141-0144) including a display (rendering target, ¶0024, ¶0096), an input device (Said computer-implemented system 200 can comprise a user interface, for example a keyboard, a mouse, voice recognition capabilities or other interface permitting the user to access and provide input to computer-implemented system. – ¶0163), a non-transitory memory (¶0030, computer-readable media in ¶0084) and one or more processors (¶0084-0085):
receiving, via a graphical user interface (GUI), a first user input (FIG. 1 illustrates an overview of an embodiment of a method. According to said embodiment, a user provides a prompt 10 to a computer-implemented system 200 (as illustrated in FIG. 5) configured to execute said method 100. Said prompt 10 can be a text, such for example “a tiger doll” or an image such for example a picture of a tiger doll. – ¶0089
Said computer-implemented system 200 can comprise a user interface, for example a keyboard, a mouse, voice recognition capabilities or other interface permitting the user to access and provide input to computer-implemented system. – ¶0163. ) that corresponds to a request to generate a three-dimensional (3D) environment (Then, a 3D object, for example a 3D mesh of a doll, is provided to the computer-implemented system 200. Then, the method 100 according to the present invention, is configured to generate a 3D texture 30, i.e. an UV texture, configured to be structure-aware regarding said 3D mesh 20. – ¶0089);
displaying two-dimensional (2D) previews (According to an embodiment of said invention, First, in each iteration, a texture is rendered into 2D images with sampled camera views, and we also render the 3D geometry into depth maps. The 2D image generation model is conditioned on the rendered depth and the user text prompt. – ¶0024) of the 3D environment prior to generating the 3D environment (ibid, According to an embodiment of said invention, First, in each iteration, a texture is rendered into 2D images with sampled camera views, and we also render the 3D geometry into depth maps. The 2D image generation model is conditioned on the rendered depth and the user text prompt. As supervised by at least one pre-trained 2D image generation model, a loss function is computed, and a gradient is configured to flow back to the rendered image views and finally the UV features is updated in 3D. – ¶0024),
Guo is not found disclosing expressly the limitation of, after receiving the first user input, displaying, within the GUI, suggested inputs that further characterize the first user input, and wherein each 2D preview is associated with a corresponding one of the suggested inputs.
However, Yang receives an “Input: User IDEA”
x
. Fig. 3 expressly begins with the user IDEA and sends it to Prompt Generation. See Fig. 3; §3. Yang's Initial Prompt Generation converts the multimodal user IDEA into multiple T2I text prompts. Its LMM generates
N
prompts corresponding to the original IDEA; later iterations generate revised prompts based on feedback. See Fig. 3; §3, Initial Prompt Generation; Eqs. (1), (3)–(4).
Thus, Yang discloses, after receiving the first user input, displaying, within the GUI, suggested inputs that further characterize the first user input (Yang teaches that after receiving user IDEA
x
, GPT-4V generates
N
descriptive T2I prompts
y
0
0
…
y
0
N
-
1
corresponding to that IDEA. See Yang Fig. 3; §3, Initial Prompt Generation; Eq. (1). Yang explains that a high-level IDEA is converted into descriptive T2I prompts and that subsequent prompts are revised based on feedback. See Eqs. (3)–(4). These constitute suggested/refined inputs further characterizing the initial IDEA.), and wherein each 2D preview is associated with a corresponding one of the suggested inputs (Yang further generates
N
prompts
y
t
n
, and each prompt is separately sent to T2I model
G
, producing
N
respective draft images according to
i
t
n
=
G
y
t
n
. See Yang §3; Eqs. (1)–(2); Fig. 3. Thus prompt
y
t
0
corresponds to image
i
t
0
, prompt
y
t
1
to image
i
t
1
, etc.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify Guo’s text-driven 3D-scene generation method according to Yang's known prompt-generation and candidate-image exploration technique, such that the initial text request for a 3D scene is expanded into multiple more descriptive candidate prompts and respective candidate images before proceeding with the corresponding 3D-scene generation, to obtain, fter receiving the first user input, displaying, within the GUI, suggested inputs that further characterize the first user input, and wherein each 2D preview is associated with a corresponding one of the suggested inputs, because, combining prior art elements ready to be improved according to known method to yield predictable results is obvious (see MPEP §2143.I). Furthermore, the combination would make the overall system more effective text conditioning and visual evaluation of alternative interpretations of the user's requested scene before those characteristics are incorporated into the generated 3D environment.
Regarding claim 2, Guo in view of Yang discloses the method of claim 1, wherein displaying the suggested inputs comprises utilizing a language model to generate the suggested inputs (Photorealistic text-to-image diffusion models with deep language understanding – ¶0104).
Regarding claim 10, Guo in view of Yang discloses the method of claim 1, wherein the first user input includes a first text input (FIG. 1 illustrates an overview of an embodiment of a method. According to said embodiment, a user provides a prompt 10 to a computer-implemented system 200 (as illustrated in FIG. 5) configured to execute said method 100. Said prompt 10 can be a text, such for example “a tiger doll” or an image such for example a picture of a tiger doll. – ¶0089).
Regarding claim 11, Guo in view of Yang discloses the method of claim 10, wherein the first text input is associated with a first set of one or more images (ibid, Then, a 3D object, for example a 3D mesh of a doll, is provided to the computer-implemented system 200. Then, the method 100 according to the present invention, is configured to generate a 3D texture 30, i.e. an UV texture, configured to be structure-aware regarding said 3D mesh 20. – ¶0089).
Regarding claim 12, Guo in view of Yang discloses the method of claim 11, wherein the user provides the first set of one or more images (ibid, ¶0089).
Regarding claim 14, Guo in view of Yang discloses the method of claim 1, wherein the first user input includes a first image (¶0089).
Regarding claim 16, Guo in view of Yang discloses the method of claim 1, wherein each 2D preview is an image (According to an embodiment of said invention, First, in each iteration, a texture is rendered into 2D images with sampled camera views, and we also render the 3D geometry into depth maps. The 2D image generation model is conditioned on the rendered depth and the user text prompt. – ¶0024).
Regarding claim 18, Guo in view of Yang discloses the method of claim 1, wherein the 2D previews are generated using a first model and the 3D environment is generated using a second model that is different from the first model (According to an example, the present invention uses a two-level optimization pipeline to generate at last one 3D texture image, preferably structure-aware, and advantageously with a 4K resolution. – ¶0024
The 2D image generation model is conditioned on the rendered depth and the user text prompt – ¶0024).
Regarding claim 19, Guo discloses a device (fig. 5, ¶0089, ¶0141-0144) comprising:
a display (rendering target, ¶0024, ¶0096);
an input device (Said computer-implemented system 200 can comprise a user interface, for example a keyboard, a mouse, voice recognition capabilities or other interface permitting the user to access and provide input to computer-implemented system. – ¶0163);
one or more processors (¶0084-0085);
a non-transitory memory (¶0030, computer-readable media in ¶0084); and
one or more programs stored in the non-transitory memory, which, when executed by the one or more processors (¶0029-0030, ¶0084-0085), cause the device to:
receive, via a graphical user interface (GUI), a first user input that corresponds to a request to generate a three-dimensional (3D) environment;
after receiving the first user input, display, within the GUI, suggested inputs that further characterize the first user input; and
display two-dimensional (2D) previews of the 3D environment prior to generating the 3D environment, wherein each 2D preview is associated with a corresponding one of the suggested inputs (see substantively similar claim 1 rejection above).
Regarding claim 20, Guo discloses a non-transitory memory (¶0030, computer-readable media in ¶0084) storing one or more programs (¶0029-0030, 0084-085), which, when executed by one or more processors (¶0029-0030, 0084-0085) of a device (200, fig. 5, ¶0089, ¶0141-0144) with an input device (Said computer-implemented system 200 can comprise a user interface, for example a keyboard, a mouse, voice recognition capabilities or other interface permitting the user to access and provide input to computer-implemented system. – ¶0163) and a display (rendering target, ¶0024, ¶0096), cause the device to:
receive, via a graphical user interface (GUI), a first user input that corresponds to a request to generate a three-dimensional (3D) environment;
after receiving the first user input, display, within the GUI, suggested inputs that further characterize the first user input; and
display two-dimensional (2D) previews of the 3D environment prior to generating the 3D environment, wherein each 2D preview is associated with a corresponding one of the suggested inputs (see substantively similar claim 1 rejection above).
Claim(s) 3, 4, 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Yang and further in view of Nguyen et al. (US 20250005293 A1, hereinafter Nguyen).
Regarding claim 3, Guo in view of Yang discloses the method of claim 2, except, wherein the suggested inputs resolve an ambiguity in the first user input by clarifying an ambiguous word in the first user input.
However, Nguyen discloses that potential ambiguity in an initial input is resolved in successive inputs received in a follow-up query generated by VLM prompt (¶0008).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the invention of Guo in view of Yang to further incorporate the teaching of Nguyen of using a VLM prompt based on synthetic follow-up query to accept additional suggested inputs to resolve potential ambiguity in the first user input, because, such combination improves the overall output result by removing ambiguity and enhancing the accuracy and confidence in desired output image.
Regarding claim 4, Guo in view of Yang discloses the method of claim 1, wherein the first user input comprises an image or a selection of the image from a plurality of images (Said prompt 10 can be a text, such for example “a tiger doll” or an image such for example a picture of a tiger doll. – ¶0089); and
Guo in view of Yang is not found disclosing wherein generating the suggested inputs comprises:
generating an image caption for the image; and
generating the suggested inputs based on the image caption.
However, Nguyen leveraging large language model(s) (LLMs) and vision language model(s) (VLMs) to facilitate human-to-computer dialogs (Abstract), where captions are generated for an image or video query that is based on vision data captured by vision component(s) of the client device 110 to facilitate query generation (¶0030), and further suggested inputs are generated based on vision language models VLMs, specifically when an initial input contains potential ambiguity (¶0008).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the invention of Guo in view of Yang to further incorporate the teaching of Nguyen of using a VLM prompt based on synthetic follow-up query to accept additional suggested inputs to resolve potential ambiguity in the first user input, where queries themselves are generation from image captioning generation, because, such combination improves the overall output result by removing ambiguity and enhancing the accuracy and confidence in desired output image.
Regarding claim 9, Guo in view of Yang discloses the method of claim 1, except, wherein the suggested inputs are generated by a large language model (LLM).
However, Nguyen leveraging large language model(s) (LLMs) and vision language model(s) (VLMs) to facilitate human-to-computer dialogs (Abstract).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the invention of Guo in view of Yang to further incorporate the teaching of Nguyen of using a VLM prompt based on synthetic follow-up query to accept additional suggested inputs to resolve potential ambiguity in the first user input, where suggested inputs are generated by a large language model (LLM), because, combining prior art elements ready to be improved according to known method to yield predictable results is obvious (see MPEP §2143.I).
Furthermore, such combination improves the overall output result by removing ambiguity and enhancing the accuracy and confidence in desired output image.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Yang and further in view of LORITZ et al. (US 20130254220 A1, hereinafter LORITZ).
Regarding claim 5, Guo in view of Yang teaches the method of claim 1, except, wherein the suggested inputs are within a divergence threshold of the first user input.
Loritz teaches controlling query expansion based upon semantic similarity, including calculating semantic similarity between an expansion term and a term of the original query and comparing that semantic similarity with a semantic similarity threshold, with the query being expanded using an expansion term satisfying the threshold. (Abstract, ¶0049, claim 16). Loritz further teaches displaying an expansion term to a user and permitting the user to select the displayed expansion term (claims 22, 31).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to apply the semantic-similarity threshold taught by Lortz to Yang's generation of candidate/refined prompts in the Guo-Yang system, in order to retain candidate suggested inputs having sufficient semantic relationship to the initial user input and thereby reduce candidate suggestions that diverge excessively from the user's original requested content. The resulting system would therefore generate/use suggested inputs constrained to remain within an acceptable semantic divergence from the first user input.
Claim(s) 6, 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Yang and further in view of Le Doux et al. (US 20180315238 A1, hereinafter Le Doux).
Regarding claim 6, Guo in view of Yang discloses the e method of claim 1, except, further comprising: receiving, via the GUI, a second user input that selects at least one of the suggested inputs; and generating the 3D environment based on the first user input and the second user input. However, Le Doux discloses an application GUI containing selectable preset buttons 750. A user interacts with a preset button to invoke the corresponding 3D display configuration. See figs. 7-8, particularly application window 700, preset buttons 710, viewport 400, and preview 810. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the Guo-Yang system to present Yang's generated candidate/refined inputs as selectable alternatives in a graphical user interface according to the selectable-alternative GUI technique taught by Le Doux. Yang generates multiple alternative prompts corresponding to an original user IDEA and respective visual candidates for evaluating those alternatives. Le Doux demonstrates the known user-interface technique of presenting alternative configurations through selectable GUI controls and receiving user selection of a desired alternative. Providing Yang's alternatives as selectable GUI elements would predictably permit the user, rather than an automated selection process alone, to indicate which generated refinement represents the desired interpretation before that refinement is supplied to Guo's text-conditioned 3D-scene generation process.
Regarding claim 7, Guo in view of Yang and Le Doux discloses the method of claim 6, wherein generating the 3D environment comprises utilizing a neural radiance field (NeRF) to generate the 3D environment (Guo: ¶0005, ¶0104).
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Yang and further in view of Litany et al. (US 12718488 B2, hereinafter Litany).
Regarding claim 13, Guo in view of Yang discloses the method of claim 11, except, wherein the device identifies the first set of one or more images from a collection of images based on the first text input.
However, Litany discloses systems and methods relating to neural networks for three-dimensional (3D) scene representations and modifying the 3D scene representations (Abstract), wherein an image modifier 212 identifies a plurality of modifier reference view (images) 236 based on one reference view (image, see fig. 2).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the invention of Guo in view of Yang to include the teaching of Litany during synthetic 3D image generation process, where the input image or text of Guo is used to identify a first set of one or more images from a collection of images, because, in that case the system can select at least one of the plurality of modified reference views/images for further processing (e.g., based on user input; based on a performance metric associated with the modification of the reference views 220), and thus improving the accuracy of the desired output image.
Regarding claim 15, Guo in view of Yang discloses the method of claim 1, except, wherein the first user input includes an annotation of a first image.
However, Litany discloses systems and methods relating to neural networks for three-dimensional (3D) scene representations and modifying the 3D scene representations (Abstract), wherein system 100 can condition the configuration of the image modifier 112 on various inputs including, without limitation, text and/or semantic labels.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the invention of Guo in view of Yang to include the teaching of Litany during synthetic 3D image generation process, such that configuration of the incorporated image modifier 112 of Litany can generate text or semantic labels for the input image of Guo, to obtain, wherein the first user input includes an annotation of a first image, because, combining prior art elements ready to be improved according to known method to yield predictable results is obvious (see MPEP §2143.I). Furthermore, such annotation helps the system to produce the output image in a desired manner, wherein desirability is potentially annotated in the input image.
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Yang and further in view of Zion et al. (US 20230032771 A1, hereinafter Zion).
Regarding claim 17, Guo in view of Yang discloses the method of claim 1, except, wherein the 2D previews are generated by the electronic device and the 3D environment is generated by a second electronic device.
However, Zion discloses that 2D representation of a 3D content are generated and displayed in different electronic devices (Abstract).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention (AIA ) to modify the invention Guo in view of Yang with the teaching of Zion of generating and displaying a 2D preview and 3D scene in different electronic devices to obtain, wherein the 2D previews are generated by the electronic device and the 3D environment is generated by a second electronic device, because, generating the preview and 3d synthesis images in different devices enhances the ease and efficiency of the overall 3D image synthesis system by distributing the overall tasks among different electronic devices.
Allowable Subject Matter
Claim 8 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Prior arts of record taken alone or in combination fails to reasonably disclose or suggest
Regarding claim 8, Guo in view of Yang discloses the method of claim 1, further comprising:
receiving a selection of a plurality of the 2D previews; and
generating the 3D environment based on the selection of the plurality of the 2D previews.
Conclusion
The prior and/or pertinent art(s) made of record and not relied upon is considered pertinent to applicant's disclosure, are –
Ding et al. (US 20260141611 A1), discloses approaches to be used to generate three-dimensional (3D) scenes using one or more 3D assets obtained from a layout image. The layout image may be produced from an input prompt, such as a text prompt, and objects at least partially depicted by the layout image may be identified and then used to generate the one or more 3D assets. One or more trained models may be used as part of a pipeline to receive an input, generate the one or more layout images, identify objects at least partially depicted by the one or more layout images, produce one or more 3D assets based on a description of the objects and/or a cropped image of the objects, and then to produce a 3D scene representing the layout image.
Höllein, Lukas, et al. ("Text2room: Extracting textured 3d meshes from 2d text-to-image models." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023), discloses Text2Room1, a method for generating roomscale textured 3D meshes from a given text prompt as input.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NURUN FLORA whose telephone number is (571)272-5742. The examiner can normally be reached M-F 9:30 am -5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571) 272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NURUN FLORA/Primary Examiner, Art Unit 2619
