DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted is considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Neal et al. (US Publication Number 2024/0112394 A1, hereinafter “Neal”) in view of Ranjan et al. (US Publication Number 2025/0148678 A1, hereinafter “Ranjan”).
(1) regarding claim 1:
As shown in fig.1, Neal disclosed a system associated with an immersive experience framework (para. [0027], note that as referring to FIG. 1, the method 100 may be used to generate an immersive image suitable for 6DOF rendering and viewing in VR in response to a text description of the scene from the user), comprising:
an immersive virtual location data store that contains information about a plurality of three-dimensional scenes, each three-dimensional scene being associated with an immersive virtual location (para. [0049], note that the immersive projection and corresponding depthmap, and possibly the background layer, may also be used in various other more complex ways. For example, for a still immersive projection, a simulated camera movement (rotation and or translation) with respect to the surface 206 may be generated 120. The simulated camera movement may include a location and orientation of a virtual camera at each time point in a series of time points. For each time point, a frame of video may be rendered 122 by from the location and orientation corresponding to each time point); and
an immersive virtual location tool, coupled to the immersive virtual location data store (para. [0061], note that a data structure may specify attributes, locations, or other information for entities referenced in the text-to-image prompt), including:
a computer processor, and a computer memory storing instructions that when executed by the computer processor cause the immersive virtual location tool to (para. [0067], note that processor(s) 402 include one or more processors or controllers that execute instructions stored in memory device(s) 404 and/or mass storage device(s) 408):
receive, from a creator, an immersive virtual location request (para. [0029], note that the text prompt may specify the relative locations of any of the above-mentioned entities (in front of, behind, to the side of, in foreground, in background etc.). The text prompt may specify a collective or individual action and an object of such action that is performed by any of the entities referenced in the text prompt. The text prompt may specify attributes of any of the entities referenced in the text prompt),
automatically create a request prompt based on the immersive virtual location request, transmit the request prompt to a text-to-video generative artificial intelligence model (para. [0030], note that the method 100 may include processing 104 the text prompt with a text-to-image artificial image model to obtain a non-immersive image, such as DALL-E 1 or DALLE-2, Midjourney, Stable Diffusion, or other machine learning model, such as a deep neural network (DNN), generative adversarial network (GAN), or other type of machine learning network),
receive, from the text-to-video generative artificial intelligence model, a video of a virtual location, convert the video of the virtual location into a three-dimensional scene using a volume rendering technique (para. [0052], note that receiving 102 the text prompt may include the use of a large language model (LLM) 300 or other generative artificial intelligence model. The use of the LLM 300 will include the use of various types of text prompts. Accordingly, the text prompt used to generate an immersive projection and corresponding depthmap according to the method 100 is referred to in the following description as a text-to-image prompt. Text prompts input to the LLM are referred to herein as text-generation prompts), store information about the three-dimensional scene in the immersive virtual location data store (para. [0048], note that the immersive projection and corresponding depthmap, and possibly the background layer, may be provided to a VR or three-dimensional rendering system in order to render left and right images for displaying 118 in a VR display device (e.g., headset), three-dimensional display viewable).
Neal disclosed most of the subject matter as described as above except for specifically teaching arrange for a user to interact with the three-dimensional scene using a substantially real-time experience interaction engine.
However, Ranjan disclosed arrange for a user to interact with the three-dimensional scene using a substantially real-time experience interaction engine (para. [0032], note that the electronic device 105 works in conjunction with the electronic device 115 to generate a computer-generated reality environment including physical and/or virtual objects that enables different forms of interaction (e.g., visual, auditory, and/or physical or tactile interaction) between the user and the generated computer-generated reality environment in a real-time manner).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach arrange for a user to interact with the three-dimensional scene using a substantially real-time experience interaction engine. The suggestion/motivation for doing so would have been in order to provide benefits such as efficient rasterization and adaptability to unobserved deformations and enhancing the representation and animation of avatars (para. [0022]). Therefore, it would have been obvious to combine Neal with Ranjan to obtain the invention as specified in claim 1.
(2) regarding claim 2:
Neal further disclosed the system of claim 1, wherein the request prompt is based on at least one of: (i) an environment description of the virtual location, and (ii) information inferred from a scenario (para. [0029], note that the text prompt will typically be a description of a scene. The text prompt may specify types of objects, types of people (gender, race, profession, fantasy role, etc.), real or mythical animals, or plants).
(3) regarding claim 3:
Neal further disclosed the system of claim 1, wherein the immersive virtual location request further includes information about at least one of: (i) a room description, (ii) a physics description, (iii) a style suggestion, (iv) a user goal, and (v) a character in the virtual location (para. [0029], note that the text prompt may specify a setting (dessert, jungle, forest, outer space, city, town, fantasy village or city, alien planet or city, etc.), or any other information).
(4) regarding claim 4:
Neal further disclosed the system of claim 1, wherein the immersive virtual location request received from the creator is associated with at least one of: (i) a text request, (ii) an audio request, (iii) an image request, and (iv) a video request (para. [0037], note that a consequence of generating the immersive projection while transforming the depthmap values as described above when warping between projections, is that lines which should be straight in the real 3D world are more likely to be straight in the immersive volumetric scene of the immersive image).
(5) regarding claim 5:
Neal further disclosed the system of claim 1, wherein the text-to-video model comprises a text-to-image model followed by an image-to-video model (para. [0049], note that a frame of video may be rendered 122 by from the location and orientation corresponding to each time point. An audio track corresponding to the rendered video may be generated 124).
(6) regarding claim 6:
Neal further disclosed the system of claim 1, wherein the generative artificial intelligence model comprises a multimodal Large Language Model ("LLM") (para. [0052], note that text prompts input to the LLM are referred to herein as text-generation prompts).
(7) regarding claim 7:
Neal disclosed most of the subject matter as described as above except for specifically teaching wherein the volume rendering technique is associated with Gaussian splatting.
However, Ranjan disclosed wherein the volume rendering technique is associated with Gaussian splatting (para. [0024], note that the subject technology provides for human subject Gaussian splatting using machine learning).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the volume rendering technique is associated with Gaussian splatting. The suggestion/motivation for doing so would have been in order to provide benefits such as efficient rasterization and adaptability to unobserved deformations and enhancing the representation and animation of avatars (para. [0022]). Therefore, it would have been obvious to combine Neal with Ranjan to obtain the invention as specified in claim 7.
(8) regarding claim 8:
Neal disclosed most of the subject matter as described as above except for specifically teaching wherein three-dimensional Gaussians are converted into meshes enabling simulation physics.
However, Ranjan disclosed wherein three-dimensional Gaussians are converted into meshes enabling simulation physics (para. [0021], note that modeling individuals with clothing or intricate hairstyles within this approach may be limited, stemming from the inherent constraints of template meshes, including fixed topologies and surface-like geometries).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein three-dimensional Gaussians are converted into meshes enabling simulation physics. The suggestion/motivation for doing so would have been in order to provide benefits such as efficient rasterization and adaptability to unobserved deformations and enhancing the representation and animation of avatars (para. [0022]). Therefore, it would have been obvious to combine Neal with Ranjan to obtain the invention as specified in claim 8.
(9) regarding claim 9:
Neal further disclosed the system of claim 1, wherein the stored information about the three-dimensional scene includes a Java Script Object Notation ("JSON") file containing at least one of: (i) virtual environment locations, (ii) virtual environment dimensions, and (iii) virtual environment mesh references (para. [0061], note that the LLM 300 receives the text-generation prompt and outputs a text-to-image prompt that will have additional details for concepts referenced in the text-generation prompt. The text-to-image prompt may be in the form of text, one or more data structures (JavaScript Object Notation (JSON), or other type of object).).
(10) regarding claim 10:
Neal further disclosed the system of claim 1, wherein the immersive virtual location tool is associated
with at least one of: (i) a personal soft skill training use case, (ii) a business skill use case, and (iii) an entertainment use case (para. [0024], note that the field of deep learning focuses on models which are represented by graphs that enable efficient calculation of the gradient of a loss function with respect to unknown parameters of the model, thereby enabling learning the unknown parameters (“training the model”) via numerical optimization methods such as stochastic gradient descent).
(11) regarding claim 11:
Neal further disclosed the system of claim 1, wherein the information about the three-dimensional scene in the immersive virtual location data store is sharable with a plurality of creators (para. [0028], note that inputs to the method 100 from a user, including an instruction to invoke the method 100 itself, may be received from a remote user by way of a user device communicating with the computing device 500 by way of a wired and/or wireless network).
(12) regarding claim 12:
Neal further disclosed the system of claim 1, wherein the information about the three-dimensional scene in the immersive virtual location data store is sharable with a plurality of users (para. [0028], note that inputs received from a user and actions performed by a user may be understood as being received from such a user device or input directly by the user to an input device of the computing device 500.).
(13) regarding claim 13:
Neal further disclosed the system of claim 1, wherein the immersive virtual location tool dynamically refines the request prompt via interactions with the creator (para. [0072], note that interface(s) 406 include various interfaces that allow computing device 400 to interact with other systems, devices, or computing environments).
(14) regarding claim 14:
As shown in fig.1, Neal disclosed a computer-implemented method associated with an immersive experience framework (para. [0027], note that as referring to FIG. 1, the method 100 may be used to generate an immersive image suitable for 6DOF rendering and viewing in VR in response to a text description of the scene from the user), comprising:
receiving, by a computer processor of an immersive virtual location tool from a creator, an immersive virtual location request including an environment description of a virtual location (para. [0049], note that the immersive projection and corresponding depthmap, and possibly the background layer, may also be used in various other more complex ways. For example, for a still immersive projection, a simulated camera movement (rotation and or translation) with respect to the surface 206 may be generated 120. The simulated camera movement may include a location and orientation of a virtual camera at each time point in a series of time points. For each time point, a frame of video may be rendered 122 by from the location and orientation corresponding to each time point);
automatically creating a request prompt based on the environment description (para. [0029], note that the method 100 may include receiving 102 a text prompt from the user. The text prompt will typically be a description of a scene);
dynamically refining the request prompt via interactions with the creator (para. [0029], note that the text prompt may specify types of objects, types of people (gender, race, profession, fantasy role, etc.), real or mythical animals, or plants. The text prompt may specify a setting (dessert, jungle, forest, outer space, city, town, fantasy village or city, alien planet or city, etc.), or any other information i.e. refined setting);
transmitting the request prompt to a Large Language Model ("LLM") (para. [0052], note that receiving 102 the text prompt may include the use of a large language model (LLM) 300 or other generative artificial intelligence model);
receiving, from the LLM, a structured scene description for the virtual location (para. [0061], note that the LLM 300 receives the text-generation prompt and outputs a text-to-image prompt that will have additional details for concepts referenced in the text-generation prompt);
storing information about the three-dimensional scene in an immersive virtual location data store, wherein the immersive virtual location data store contains information about a plurality of three-dimensional scenes, each three-dimensional scene being associated with an immersive virtual location (para. [0048], note that the immersive projection and corresponding depthmap, and possibly the background layer, may then be output 116 for use by another process, for storage for later use, or for any other purpose. The immersive projection and corresponding depthmap, and possibly the background layer, may be provided to a VR or three-dimensional rendering system in order to render left and right images for displaying 118 in a VR display device (e.g., headset), three-dimensional display viewable).
Neal disclosed most of the subject matter as described as above except for specifically teaching converting the structured scene description for the virtual location into a three-dimensional scene using a volume rendering technique associated with Gaussian splatting; and arranging for a user to interact with the three-dimensional scene using a substantially real-time experience interaction engine.
However, Ranjan disclosed converting the structured scene description for the virtual location into a three-dimensional scene using a volume rendering technique associated with Gaussian splatting (para. [0081], note that the rendering engine 423 may generate a deformed three-dimensional Gaussian representation of the subject by adapting the three-dimensional Gaussian representation of the subject to the 3D reconstruction of the subject); and arranging for a user to interact with the three-dimensional scene using a substantially real-time experience interaction engine (para. [0032], note that the electronic device 105 works in conjunction with the electronic device 115 to generate a computer-generated reality environment including physical and/or virtual objects that enables different forms of interaction (e.g., visual, auditory, and/or physical or tactile interaction) between the user and the generated computer-generated reality environment in a real-time manner).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach converting the structured scene description for the virtual location into a three-dimensional scene using a volume rendering technique associated with Gaussian splatting; and arranging for a user to interact with the three-dimensional scene using a substantially real-time experience interaction engine. The suggestion/motivation for doing so would have been in order to provide benefits such as efficient rasterization and adaptability to unobserved deformations and enhancing the representation and animation of avatars (para. [0022]). Therefore, it would have been obvious to combine Neal with Ranjan to obtain the invention as specified in claim 14.
The proposed rejection of claims 3-4, 8-9, 12-14, renders obvious the method claims 15-18 (3-4, 8-9 of system claims) and the non-transitory computer-readable media claims 19-21 (12-13 of system and method claim 14) because these steps occur in the operation of the proposed rejection as discussed above. Thus, the arguments similar to that presented above for claims 3-4, 8-9 and 12-14 are equally applicable to claims 15-21.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Roberts et al. (US Publication Number 2022/0004254 A1) disclosed an immersive virtual reality experience is provided. A plurality of areas within a virtual environment are determined. The plurality of areas includes at least a first area and a second area. A plurality of videos associated with the virtual environment are presented. The plurality of videos include a first video associated with the first area and a second video associated with the second area. The first video has a first quality, and the second video has a second quality lower than the first quality. A three-dimensional model of a first object is overlaid in the virtual environment.
Wan et al. (NPL, “Building LLM-based AI Agents in Social Virtual Reality”, 2024) disclosed the design and evaluation of an LLM based AI agent for human-agent interaction in Virtual Reality (VR). Our AI agent system leverages GPT-4, a Large Language Model (LLM) to simulate human behavior. Our LLM-based agent, deployed in VRChat as a Non-playable Character (NPC), exhibits the ability to respond to a player by providing context-relevant responses followed by appropriate facial expressions and body gestures.
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Hilina K Demeter whose telephone number is (571) 270-1676.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y. Poon could be reached at (571) 270- 0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about PAIR system, see http://pari-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HILINA K DEMETER/Primary Examiner, Art Unit 2617
Prosecution Insights