Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 7 is objected to because of the following informalities: Claim 7 recites "said Gaussian splattering algorithm," in which the term "splattering" should read "Splatting". Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5, 6, 8, and 10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 5, it recites high-quality 3D models. The term “high-quality” is a relative term. One having ordinary skill in the art would not be able to determine the metes and bounds of the term high-quality only based on the claim language. The specification does not provide guidance on how to determine the metes and bounds of the term “high-quality,” (specification, para [0020]-[0022]).
Regarding claim 6, it recites high-quality 3D models include people/objects and scenes. The phrase “people/objects” is indefinite because use of the slash makes it unclear as to if the limitation refers to people and other objects, people or objects, people as a type of object, or objects including people. One having ordinary skill in the art would not be able to determine the metes and bounds of the phrase “people/objects” only based on the claim language. The specification does not provide guidance on how to determine the metes and bounds of the phrase “people/objects” (specification, para [0009]). For the purpose of compact prosecution and art rejection, the examiner will treat the phrase “people/objects” to mean people or objects.
Regarding claim 8, it recites 3D modeling and rendering software include Blender b3d, Maya and Unreal Engine. The terms “Blender b3d” and “Maya” are indefinite because it is unclear what software is being referred to. The term “Blender b3d” does not correspond to known software, and it is unclear if the claim intends to refer to Blender, the open-source software maintained by the Blender Foundation. The term “Maya” is unclear because it is unclear if the claim intends to refer to Autodesk Maya, the professional software owned by Autodesk, Inc. One having ordinary skill in the art would not be able to determine the metes and bounds of the terms “Blender b3d” and “Maya” only based on the claim language. The specification does not provide guidance on how to determine the metes and bounds of the of the terms “Blender b3d” and “Maya,” (specification, para [0021]). For the purpose of compact prosecution and art rejection, the examiner will treat the term “Blender b3d” as referring to Blender, the open-source software maintained by the Blender Foundation, and the term “Maya” as referring to Autodesk Maya, the professional software owned by Autodesk, Inc.
Regarding claim 10, it recites …wherein said 3D model reconstruction and rendering device…. The phrase “3D model reconstruction and rendering device” lacks sufficient antecedent basis in the claim. Claim 1 introduces a “three-dimensional (3D) model reconstruction and rendering apparatus;” however, the claims do not previously introduce a “device.” Therefore, it is unclear whether the recited device refers to the previously recited apparatus or to a different element. For the purpose of compact prosecution and art rejection, the examiner will treat the phrase “3D model reconstruction and rendering device” as referring to the previously recited “three-dimensional (3D) model reconstruction and rendering apparatus.”
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Step 1:
Claims 1-12 are directed to a machine. As such, claims 1-12 are drawn to one of the statutory categories of invention (Step 1: Yes).
Regarding Step 2A Prong 1:
Claims 1 and 10 recite:
obtaining required data from two-dimensional (2D) images
training a 3D modeling model and learn the ability of reconstructing and rendering 3D models and scenes
converting said required data into 3D models and scenes
processing and post-production on the generated 3D models and scenes
Limitation (b) recites training a model. Training a model using gathered data consists of mathematical relationships, formulas, and calculations. Thus, limitation (b) recites a mathematical concept.
Limitations (c) - (d) can practically be performed in the human mind with or without the use of a physical aid such as pen and paper; therefore, the limitation falls within the
mental processes grouping, and the claims recite an abstract idea. (Step 2A – Prong 1: Yes).
Regarding Step 2A Prong 2:
Limitation (a) recites the additional element of obtaining required data. This additional
element represents mere data gathering that is necessary for use of the recited judicial
exception and is recited at a high level of generality. Limitation (a) in the claim is thus
insignificant extra-solution activity. Thus, the claims as a whole do not integrate the exception into a practical application. (Step 2A –Prong 2: No).
Regarding Step 2B:
The claims do not include additional elements that are sufficient to amount to
significantly more than the judicial exception because the processor(s), memory,
network interface, and storage are at best the equivalent of merely adding the words
"apply it" to the judicial exception. Mere instructions to apply an exception cannot
provide an inventive concept. The claims lack affirmative recitation a specific,
unconventional technical means. The additional element, considered both individually and in combination, does not amount to significantly more than the judicial exception itself. A court is likely to view the steps as routine computer implementation. (Step 2B: No).
Regarding claim 2, it recites a high-resolution reconstruction module configured to produce high-resolution reconstruction from generated 3D models and scenes. The limitation can practically be performed in the human mind with or without the use of a physical aid such as pen and paper; therefore, the limitation falls within the
mental processes grouping, and the claim recites an abstract idea. Further, the claim does not recite any additional element.
Regarding claims 3, 6, 8, 9, and 12, the claims recite clarifications of reconstructing a 3D model with a resolution of up to 4K, including people/objects and scenes in 3D models, including Blender b3d, Maya and Unreal Engine as 3D modeling and rendering software, including a multi-core central processing unit (CPU), a graphics processor unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or their combinations, and providing high-resolution reconstruction. The limitations are further instructions for applying the judicial exceptions with a generic computing device/interface acting as an intermediary for performing the abstract ideas of converting said required data into 3D models and scenes, processing and post-production on the generated 3D models and scenes, see MPEP 2106.05(f). Therefore, the limitations fail to provide any teaching that integrates the judicial exceptions into a practical application or amount to significantly more than the judicial exception. For this reason, the analysis performed on the independent claim is also applicable on these claims.
Regarding claims 4 and 7, the claims recite clarifications of including a neural radiation field (NeRF) algorithm and a Gaussian Splatting algorithm, and using the Gaussian Splatting algorithm to improve rendering effect. The limitations are further instructions for applying the judicial exceptions with a generic computing device/interface acting as an intermediary for performing the abstract ideas of training a 3D modeling model and learn the ability of reconstructing and rendering 3D models and scenes, see MPEP 2106.05(f). Therefore, the limitations fail to provide any teaching that integrates the judicial exceptions into a practical application or amount to significantly more than the judicial exception. For this reason, the analysis performed on the independent claim is also applicable on these claims.
Regarding claims 5 and 11, the claims recite reconstructing high-quality 3D models from multi-view 2D images and obtaining required data from 2D images. Such clarifications, under their broadest reasonable interpretation, are merely defining/selecting a type of data to be manipulated which, per MPEP 2106.05(g), is insignificant extra-solution activity. Therefore, the limitations fail to provide any teaching that integrates the judicial exceptions into a practical application or amount to significantly more than the judicial exception. For this reason, the analysis performed on the independent claim is also applicable on these claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 4-7 are rejected under 35 U.S.C. 103 as being unpatentable over Alimo et al. (US 20240135623 A1; hereinafter Alimo) in view of A M Avramescu, "Special effects used in creating 3D animated scenes part 1," IOP Conference Series: Materials Science and Engineering, Volume 95, 2015; hereinafter Avramescu.
Regarding claim 1, Alimo teaches a three-dimensional (3D) model reconstruction and rendering apparatus, comprising: a processor; a storage device couple to said processor ("an apparatus is disclosed that includes a non-transitory memory and a processor configured to execute instructions stored in the non-transitory memory. The processor may be configured to executed the instructions to... generate, based upon the differentiable radiance field, a three-dimensional representation of the environment," (pages 1-2, para [0013]). The non-transitory memory reads on storage device coupled to said processor.);
a data collection module, stored in said storage device and accessible through said processor, configured to obtain required data from two-dimensional (2D) images captured by different photographic equipment ("In an initial stage, images of an environment are captured at operation 410. The images may be captured by the image capture device 104 or one or more additional image capture device. For example, the image capture device 104 may be contained within a personal device of the user 102 such that the user 102 may interface with the image capture device 104 through the user interface 310 to capture the images. In certain situations, the images may have been captured by one or more image capture devices (e.g., cameras) at different times and stored for use by the process 400. For example, the images may be captured and stored, such as within a database on the server 306, for later access by the process 400. As such, the images are not limited to those captured by one image capture device. That is, the process 400 may also be implemented using images of (e.g., received from or captured by) more than one (e.g., two or more) image capture device. The images may be extracted at operation 420. The images may be extracted directly from an image capture device, such as the image capture device 104 or the images may be extracted from one or more databases, such as a database stored on the server 306. Extraction of the images may require one or more operations, procedures, or steps. For example, the computing device 308 may execute a query or retrieval procedure to direct the extraction process to a particular set of images. That is, the query or retrieval procedure may be configured to only retrieve images pertaining to the desired scene or object and omit retrieval of irrelevant images. Additionally, during extraction, any data associated with the images, such as various metrics associated with the images, may also be retrieved," (page 5, para [0050] - [0051]). Extraction of the images reads on obtaining required data from two-dimensional images.
"To prepare for 3D generation of the environment 100, the user 102 may capture one or more images (e.g., 2D images and/or videos) using an image capture device 104. The image capture device 104 may be a personal electronic device carried by the user 102, such as a mobile phone, tablet, computer, or a combination thereof. As discussed in further detail below, the image capture device 104 may be connected, such as via a wireless network connection, to a system configured to generate the 3D representation of the environment 100. It should also be noted that additional image capture devices may be used to capture the images. For example, the image capture device 104 may be a first image capture device configured to capture a first set of images of the environment 100. A second image capture device may be configured to capture a second set of images of the environment 100. The first set of images and the second set of images may both be used for the 3D generation of the environment 100," (page 3, para [0034]). A first image capture device and a second image capture device which may be a mobile phone, tablet, or computer read on different photographic equipment.
The software components comprising instructions that implement the algorithm as disclosed is mapped to the corresponding module, (pages 1-2, para [0013]). The same mapping is applied to all the software modules in the claims.);
a model training module, stored in said storage device and accessible through said processor, configured to obtain said required data from said data collection module to train a 3D modeling model and learn the ability of reconstructing and rendering 3D models and scenes ("As mentioned above, the 3D reconstruction, such as the 3D reconstruction 200, may be generated based on or using a differentiable radiance field. The differentiable radiance field may be a neural network trained to generate the 3D reconstruction of an environment, such as the environment 100. The differentiable radiance field may be created based on the images captured at the operation 410 and/or additional data corresponding to the images captured. The neural network may be trained to take the images captured (i.e., a trained neural network) and create the differentiable radiance field for rendering the 3D reconstruction (e.g., the reconstruction 200) of the desired environment (e.g., the environment 100) by interpolating between the images captured to render one complete (e.g., substantially continuous) 3D reconstruction of the environment...," (page 5 -6, para [0053] - [0056]).
The model of the differentiable radiance field reads on a 3D modeling model. The model of the differentiable radiance field may be created based on the images captured at the operation 410 and/or additional data corresponding to the images captured reads on obtain said required data from said data collection module. The model of the differentiable radiance field is trained to generate the 3D reconstruction of an environment and render the 3D reconstruction of the desired environment which reads on learn the ability of reconstructing and rendering 3D models and scenes.);
a 3D model and scene rendering module, stored in said storage device and accessible through said processor, utilizing said 3D modeling model that has been trained to convert said required data into 3D models and scenes ("At operation 480, a view may be synthesized from the trained differentiable radiance field. To synthesize a view, the differentiable radiance field (e.g., a NeRF or Gaussian Splatting) may or may be caused to generate the 3D reconstruction, such as the 3D reconstruction 200 of the environment 100. The 3D reconstruction may include one or more portions of the real-world scene captured in the images and one or more novel portions of the real-world scene that are synthesized in the 3D reconstruction and were not captured in the images used to create the differentiable radiance field. As a result, the resultant 3D reconstruction may provide a complete and accurate representation of the real-world scene in its entirety without requiring all portions of the scene to be captured in the images used when creating the differentiable radiance field.
The 3D reconstruction, including synthesizing novel portions of the scene, may be provided to the user 202 via any device, such as the computing device 308 or any other interaction device. The 3D reconstruction may be scaled and visually provided to the user 202 for further review and analysis, thereby providing the user 202 a means to visually survey a representation of the real-world scene without physically visiting the real-world scene. For example, the 3D reconstruction may be provided to the user 202 via a drone or a projector. Alternatively, the 3D reconstruction may be provided using smart glasses, in which the user 202 may look through the glasses into an augmented or virtual reality that contains the 3D reconstruction," (page 7, para [0069] - [0070]).
The differentiable radiance field (reads on 3D modeling model) is clearly utilized.);
Alimo fails to explicitly teach but Avramescu teaches and a rendering and 3D effect module, stored in said storage device and accessible through said processor, configured to performed processing and post-production on the generated 3D models and scenes through a 3D modeling and rendering software ("3D representations are obtained following a process of three-dimensional graphical creation using computers and special software programs. These creations can be represented as static images or video clips. The steps in obtaining a 3D project involve: modelling objects from the scene, scene illumination, texturing and rendering. Further, a game was created using modelling, texturing and animation techniques, specific tools for video games. Considering that games must run in real time, limitations are quite harsh at least comparing to off-line rendering limits (as some rendering engines: Mental Ray, V-Ray, Brazil etc.). Although many times the optimization can be difficult, techniques used can be regarded as standard. That’s why, an object will be formed using a more advanced technique called Sub-D modeling. Included in the final scene, this object is the tank. As for the animation, from a technical standpoint a relatively simple infrastructure was kept. The turret of the tank was animated and the camera was moving in a three-dimensional way. In the post production stage, the explosion of the projectile destroying the track of the tank was added and synchronized to the scene," (page 4, para 1-2; page 5, figure 3, figure 4).
The animation of the turret of the tank and the explosion of the projectile destroying the track of the tank added and synchronized to the scene in the post production stage reads on performed processing and post-production. After combination, the processing and post production are done on the generated composite 3D model (3D models and scenes) from Alimo.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Avramescu to Alimo. The motivation would have been to integrate visual effects in complex scenes.
Regarding claim 4, Alimo in view of Avramescu teaches the three-dimensional (3D) model reconstruction and rendering apparatus of claim 1, wherein said 3D modeling model is built in said model training module including a neural radiation field (NeRF) algorithm and a Gaussian Splatting algorithm (Alimo; "The differentiable radiance field may be, for example, a neural radiance field (NeRF) or a 3D Gaussian Splatting. The differentiable radiance field may be created using one or more other techniques for 3D reconstruction and novel view synthesis. The differentiable radiance field may be created and trained in a centralized manner in which the data (e.g., images) is centralized in a single location and/or on a single device (e.g., the computing device 308) to train a central model. The differentiable radiance field may also be created and trained using a distributed learning framework in which the data may be centralized but training of the differentiable radiance field may be completed using different training nodes of a training model. Additionally, the differentiable radiance field may be a federated learning framework in which the data may be decentralized to train a central training model.
For purposes of the process 400 described herein, the differentiable radiance field may be referred to as a NeRF or a Gaussian Splatting. However, it is not intended to limit the teachings herein, and any of the above differentiable radiance fields may also be created and trained based on the process 400," (page 6, para [0054]- [0055]).
The differentiable radiance field reads on 3D modeling model. Alimo discloses that the differentiable radiance field can include a neural radiation field (NeRF), and that the differentiable radiance field can include a Gaussian Splatting algorithm.).
Regarding claim 5, Alimo in view of Avramescu teaches the three-dimensional (3D) model reconstruction and rendering apparatus of claim 4, wherein said NeRF algorithm is used to reconstruct high-quality 3D models from multi-view 2D images (Alimo; "As mentioned above, the 3D reconstruction, such as the 3D reconstruction 200, may be generated based on or using a differentiable radiance field. The differentiable radiance field may be a neural network trained to generate the 3D reconstruction of an environment, such as the environment 100. The differentiable radiance field may be created based on the images captured at the operation 410 and/or additional data corresponding to the images captured. The neural network may be trained to take the images captured (i.e., a trained neural network) and create the differentiable radiance field for rendering the 3D reconstruction (e.g., the reconstruction 200) of the desired environment (e.g., the environment 100) by interpolating between the images captured to render one complete (e.g., substantially continuous) 3D reconstruction of the environment. In addition to rendering a 3D reconstruction that contains portions of the environment 100 captured in the images, the trained differentiable radiance field may synthesize one or more portions of the environment 100 that are not captured in the images. That is, the differentiable radiance field may generate novel portions of the environment not seen in the images of the environment.
The differentiable radiance field may be, for example, a neural radiance field (NeRF) or a 3D Gaussian Splatting," (pages 5-6, para [0053]-[0054]).
The 3D reconstruction reads on 3D model. Alimo discloses that the 3D reconstruction, may be generated using a neural radiance field (NeRF) which reads on NeRF algorithm is used to reconstruct 3D models.
"Additionally, a 3D representation of a real-world scene or object can be generated based upon incomplete information with respect to the real-world scene or object. That is, the 3D scene or object may be generated based upon partial views of the real-world scene or object, such as 2D images or videos captured of only a portion of the real-world scene or object. Additionally, portions of a real-world scene or object unrepresented by (e.g., not captured in) the captured images may be accurately generated (e.g., synthesized). Furthermore, a 3D scene or object may be generated based upon images obtained by two or more different image capture devices that include different intrinsic parameters and/or extrinsic parameters, as discussed in further detail below. That images are obtained by two or more different image capture devices includes that the images may be obtained at different points, from different points of view (e.g., perspectives), using different camera configurations, and the like," (Alimo; page 3, para [0031]).
Obtaining the images from different points of view (e.g., perspectives) reads on mutli-view. The images used to reconstruct 3D models are multi-view 2D images.
"As a result, the resultant 3D reconstruction may provide a complete and accurate representation of the real-world scene in its entirety without requiring all portions of the scene to be captured in the images used when creating the differentiable radiance field," (Alimo; page 7, para [0069]). The complete and accurate representation of the real-world scene in its entirety reads on high-quality.).
Regarding claim 6, Alimo in view of Avramescu teaches the three-dimensional (3D) model reconstruction and rendering apparatus of claim 5, wherein said high-quality 3D models include people/objects and scenes (Alimo; “For illustrative purposes, the differentiable radiance field created at the operation 450 may be a neural radiance field (NeRF). At the operation 450, the creation of the NeRF may be optimized to ensure that the resultant generated 3D reconstruction accurately reflects the real-world scene or object," (page 7, para [0064]).).
Regarding claim 7, Alimo in view of Avramescu teaches the three-dimensional (3D) model reconstruction and rendering apparatus of claim 4, wherein said Gaussian splattering algorithm is used to improve rendering effect of said 3D models and scenes (Alimo; "The differentiable radiance field may be, for example, a neural radiance field (NeRF) or a 3D Gaussian Splatting. The differentiable radiance field may be created using one or more other techniques for 3D reconstruction and novel view synthesis. The differentiable radiance field may be created and trained in a centralized manner in which the data (e.g., images) is centralized in a single location and/or on a single device (e.g., the computing device 308) to train a central model. The differentiable radiance field may also be created and trained using a distributed learning framework in which the data may be centralized but training of the differentiable radiance field may be completed using different training nodes of a training model. Additionally, the differentiable radiance field may be a federated learning framework in which the data may be decentralized to train a central training model," (page 6, para [0054]).
"At operation 480, a view may be synthesized from the trained differentiable radiance field. To synthesize a view, the differentiable radiance field (e.g., a NeRF or Gaussian Splatting) may or may be caused to generate the 3D reconstruction, such as the 3D reconstruction 200 of the environment 100. The 3D reconstruction may include one or more portions of the real-world scene captured in the images and one or more novel portions of the real-world scene that are synthesized in the 3D reconstruction and were not captured in the images used to create the differentiable radiance field. As a result, the resultant 3D reconstruction may provide a complete and accurate representation of the real-world scene in its entirety without requiring all portions of the scene to be captured in the images used when creating the differentiable radiance field," (Alimo; page 7, para [0069]).
Novel view synthesis reads on improve rendering effect. When the differentiable radiance field is a 3D Gaussian Splatting it is used to improve the rendering effect of 3D models and scenes.).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Alimo in view Avramescu in further view of Son (US 20150109415 A1; hereinafter Son).
Regarding claim 2, Alimo in view of Avramescu fail to explicitly teach but Son teaches the three-dimensional (3D) model reconstruction and rendering apparatus of claim 1, further including a high-resolution reconstruction module, stored in said storage device and accessible through said processor, configured to produce high-resolution reconstruction from said generated 3D models and scenes (Son; “The low resolution reconstruction unit 30, the high resolution reconstruction unit 50, or both, may be connected to various systems that use a 3D model, and the high resolution 3D model may be used in a wide variety of application systems that use animation movies, games, and characters.
The 3D model reconstruction system 1 according to the present embodiment is a hybrid 3D model reconstruction system that separates reconstruction of a low resolution 3D model from reconstruction of a high resolution 3D model. The reconstruction of a high resolution 3D model requires a large number of computations and a long computation time, thereby making real-time monitoring difficult. Furthermore, a high performance system is required to generate a high resolution 3D model in real-time.
The 3D model reconstruction system 1 according to the present embodiment reconstructs a low resolution 3D model in real-time by using a low resolution depth map, thereby allowing real-time monitoring. The 3D model reconstruction system 1 also reconstructs a high resolution 3D model by using a high resolution depth map and the result of reconstruction of a low resolution 3D model, thereby enabling the use for an embedded system instead of a high performance system," (page 3, para [0054]-[0056]).
The reconstruction of the high resolution 3D model is produced from a low resolution 3D model. After combination, the reconstruction of the generated composite 3D model (3D models and scenes) from Alimo is reconstructed according to the method disclosed by Son.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Son to Alimo in view of Avramescu. The motivation would have been to improve the accuracy and detail of the 3D models and scenes.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Alimo in view of Avramescu in further view of Son in further view of Araki (US 20220084282 A1; hereinafter Araki).
Regarding claim 3, Alimo in view of Avramescu in further view of Son fail to explicitly teach but Araki teaches the three-dimensional (3D) model reconstruction and rendering apparatus of claim 2, wherein said high-resolution reconstruction module reconstructs a 3D model with a resolution of up to 4K to meet the needs of high-resolution visual effects ("The 3D-data generating device 21 of the third modification generates a 3D model of the object for each resolution at the time of imaging. Specifically, the 3D-data generating device 21 generates a 3D model of the object by using texture images supplied from the imaging devices 53H having the HD resolution and camera parameters corresponding to the imaging devices 53H. Furthermore, the 3D-data generating device 21 generates a 3D model of the object by using the texture images supplied from the imaging devices 53K having the 4K resolution and the camera parameters corresponding to the imaging devices 53K. Then, the 3D-data generating device 21 supplies 3D-model data representing the 3D models that have been generated to the image generating device 22," (page 14, para [0283]). Araki teaches the 3D model has a resolution of up to 4K.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Araki to Alimo in view of Avramescu in further view of Son. The motivation would have been to improve handling of complex lighting/textures and enable higher-quality rendering.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Alimo in view of Avramescu in further view of Hussey (WO 2021191017 A1; hereinafter Hussey).
Regarding claim 8, Alimo in view of Avramescu fails to explicitly teach but Hussey teaches the three-dimensional (3D) model reconstruction and rendering apparatus of claim 1, wherein said 3D modeling and rendering software include Blender b3d, Maya and Unreal Engine (Hussey; "A 57th embodiment of the invention is a computer program comprising instructions which, when the program is executed by a computer, causes the computer to carry out a computer implemented method, according to the invention, for producing a visualisation, preferably the computer implemented method according to of any of the 1st to 53rd embodiments. Examples of computer programs in the 57th embodiment are: a program that runs locally on the device of the user, e.g., a program that runs in a web browser; a program that runs on a server, e.g., a website, or a combination thereof. In a preferred embodiment of the computer program, the computer program can be integrated with, or comprises, at least one or all of the following:
a. a game engine, e.g., the Unity game engine or the Unreal game engine;
b. machine learning software, e.g., TensorFlow;
c. artificial intelligence software, e.g., IBM Watson;
d. design software, e.g., Blender, Autodesk Maya;
e. file compression software, e.g., WinZip;
f. a database, e.g., PostgreSQL or MongoDB;
g. location software, e.g., GPS software;
h. messaging software, preferably instant messaging software, e.g., Slack;
i. application programming interface;
j. workflow software, e.g., SAP;
k. a sales platform, e.g., Ebay or Amazon," (page 25, lines 22-30; page 26, lines 1 - 15)).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Hussey to Alimo in view of Avramescu. The motivation would have been for the convenience of the user to choose whichever software they deem most appropriate.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Alimo in view of Avramescu in further view of Krishnan et al. (US 20180253894 A1; hereinafter Krishnan).
Regarding claim 9, Alimo in view of Avramescu fails to explicitly teach but Krishnan teaches the three-dimensional (3D) model reconstruction and rendering apparatus of claim 1, wherein said processor includes a multi-core central processing unit (CPU), a graphics processor unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or their combinations (Krishnan; "Processor 620 can be any suitable processor, and may include one or more coprocessors or controllers, such as an audio processor or a graphics processing unit, to assist in control and processing operations associated with system 600. In some embodiments, the processor 620 may be implemented as any number of processor cores. The processor (or processor cores) may be any type of processor, such as, for example, a micro-processor, an embedded processor, a digital signal processor (DSP), a graphics processor (GPU), a network processor, a field programmable gate array or other device configured to execute code. The processors may be multithreaded cores in that they may include more than one hardware thread context (or “logical processor”) per core. Processor 620 may be implemented as a complex instruction set computer (CISC) or a reduced instruction set computer (RISC) processor. In some embodiments, processor 620 may be configured as an x86 instruction set compatible processor.
In some embodiments, the disclosed techniques for hybrid 3D model reconstruction can be implemented in a parallel fashion, where tasks may be distributed across multiple CPU/GPU cores or other cloud based resources to enable real-time processing from image capture to display," (page 3, para [0036]-[0037])).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Krishnan to Alimo in view of Avramescu. The motivation would have been to improve system/ computational performance.
Claims 10-12 are rejected under 35 U.S.C. 103 as being unpatentable over Alimo in view of Avramescu in further view of Son in further view of Shen et al. (CN 116977525 A; hereinafter Shen).
Regarding claim 10, Alimo in view of Avramescu in further view of Son teaches the three-dimensional (3D) model reconstruction and rendering apparatus of claim 2, wherein said 3D model reconstruction and rendering device configured to realize processes for reconstructing and rendering said 3D models and scenes includes performing the following steps through said processor: obtaining said required data from said 2D images captured by said different photographic equipment by said data collection module (Alimo; "In an initial stage, images of an environment are captured at operation 410. The images may be captured by the image capture device 104 or one or more additional image capture device. For example, the image capture device 104 may be contained within a personal device of the user 102 such that the user 102 may interface with the image capture device 104 through the user interface 310 to capture the images. In certain situations, the images may have been captured by one or more image capture devices (e.g., cameras) at different times and stored for use by the process 400. For example, the images may be captured and stored, such as within a database on the server 306, for later access by the process 400. As such, the images are not limited to those captured by one image capture device. That is, the process 400 may also be implemented using images of (e.g., received from or captured by) more than one (e.g., two or more) image capture device. The images may be extracted at operation 420. The images may be extracted directly from an image capture device, such as the image capture device 104 or the images may be extracted from one or more databases, such as a database stored on the server 306. Extraction of the images may require one or more operations, procedures, or steps. For example, the computing device 308 may execute a query or retrieval procedure to direct the extraction process to a particular set of images. That is, the query or retrieval procedure may be configured to only retrieve images pertaining to the desired scene or object and omit retrieval of irrelevant images. Additionally, during extraction, any data associated with the images, such as various metrics associated with the images, may also be retrieved," (page 5, para [0050] - [0051]). Extraction of the images reads on obtaining required data from two-dimensional images.
"To prepare for 3D generation of the environment 100, the user 102 may capture one or more images (e.g., 2D images and/or videos) using an image capture device 104. The image capture device 104 may be a personal electronic device carried by the user 102, such as a mobile phone, tablet, computer, or a combination thereof. As discussed in further detail below, the image capture device 104 may be connected, such as via a wireless network connection, to a system configured to generate the 3D representation of the environment 100. It should also be noted that additional image capture devices may be used to capture the images. For example, the image capture device 104 may be a first image capture device configured to capture a first set of images of the environment 100. A second image capture device may be configured to capture a second set of images of the environment 100. The first set of images and the second set of images may both be used for the 3D generation of the environment 100," (Alimo; page 3, para [0034]). A first image capture device and a second image capture device which may be a mobile phone, tablet, or computer read on different photographic equipment.);
learning said ability of reconstructing and rendering 3D models and scenes from said required data by utilizing neural architecture Alimo; "As mentioned above, the 3D reconstruction, such as the 3D reconstruction 200, may be generated based on or using a differentiable radiance field. The differentiable radiance field may be a neural network trained to generate the 3D reconstruction of an environment, such as the environment 100. The differentiable radiance field may be created based on the images captured at the operation 410 and/or additional data corresponding to the images captured. The neural network may be trained to take the images captured (i.e., a trained neural network) and create the differentiable radiance field for rendering the 3D reconstruction (e.g., the reconstruction 200) of the desired environment (e.g., the environment 100) by interpolating between the images captured to render one complete (e.g., substantially continuous) 3D reconstruction of the environment...," (page 5 -6, para [0053] - [0056]).
The model of the differentiable radiance field reads on a 3D modeling model. The model of the differentiable radiance field is trained to generate the 3D reconstruction of an environment and render the 3D reconstruction of the desired environment which reads on learn the ability of reconstructing and rendering 3D models and scenes. The neural network of the differentiable radiance field reads on neural architecture of said model.);
converting said required data into said 3D models and scenes using said 3D modeling model that has been trained (Alimo; "At operation 480, a view may be synthesized from the trained differentiable radiance field. To synthesize a view, the differentiable radiance field (e.g., a NeRF or Gaussian Splatting) may or may be caused to generate the 3D reconstruction, such as the 3D reconstruction 200 of the environment 100. The 3D reconstruction may include one or more portions of the real-world scene captured in the images and one or more novel portions of the real-world scene that are synthesized in the 3D reconstruction and were not captured in the images used to create the differentiable radiance field. As a result, the resultant 3D reconstruction may provide a complete and accurate representation of the real-world scene in its entirety without requiring all portions of the scene to be captured in the images used when creating the differentiable radiance field.
The 3D reconstruction, including synthesizing novel portions of the scene, may be provided to the user 202 via any device, such as the computing device 308 or any other interaction device. The 3D reconstruction may be scaled and visually provided to the user 202 for further review and analysis, thereby providing the user 202 a means to visually survey a representation of the real-world scene without physically visiting the real-world scene. For example, the 3D reconstruction may be provided to the user 202 via a drone or a projector. Alternatively, the 3D reconstruction may be provided using smart glasses, in which the user 202 may look through the glasses into an augmented or virtual reality that contains the 3D reconstruction," (page 7, para [0069] - [0070]).); and
producing 3D effects by performing rendering and post-production on said generated 3D models and scenes through said rendering and 3D effect module (Avramescu; "3D representations are obtained following a process of three-dimensional graphical creation using computers and special software programs. These creations can be represented as static images or video clips. The steps in obtaining a 3D project involve: modelling objects from the scene, scene illumination, texturing and rendering. Further, a game was created using modelling, texturing and animation techniques, specific tools for video games. Considering that games must run in real time, limitations are quite harsh at least comparing to off-line rendering limits (as some rendering engines: Mental Ray, V-Ray, Brazil etc.). Although many times the optimization can be difficult, techniques used can be regarded as standard. That’s why, an object will be formed using a more advanced technique called Sub-D modeling. Included in the final scene, this object is the tank. As for the animation, from a technical standpoint a relatively simple infrastructure was kept. The turret of the tank was animated and the camera was moving in a three-dimensional way. In the post production stage, the explosion of the projectile destroying the track of the tank was added and synchronized to the scene," (page 4, para 1-2; page 5, figure 3, figure 4).
The animation of the turret of the tank and the explosion of the projectile destroying the track of the tank added and synchronized to the scene in the post production stage reads on performed processing and post-production. After combination, the processing and post production are done on the generated composite 3D model (3D models and scenes) from Alimo.).
Alimo in view of Avramescu in view of Son does not explicitly teach that the neural architecture is a deep neural architecture.
Shen teaches that the neural architecture is a deep neural architecture (Shen; “Neural radiation field (Neural Radiance Fields, NeRF) is a 3D scene reconstruction technology based on deep learning, the technology uses deep neural network to learn the color and depth information of each pixel in a certain scene, Therefore, the 3D model with high quality can be generated, and the 2D image corresponding to the 3D model which can be collected under the camera pose can be generated according to the input camera pose,” (page 2, para 1). The deep neural network reads on deep neural architecture.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Shen to Alimo in view of Avramescu in further view of Son. The motivation would have been to enable high quality 3D model and scene reconstruction.
Regarding claim 11, Alimo in view of Avramescu in further view of Son in further view of Shen teaches the three-dimensional (3D) model reconstruction and rendering apparatus of claim 10, wherein said required data is obtained from said 2D images captured by said different photographic equipment, including videos captured by ordinary mobile phones, multiple consecutive photos or a continuous video from different angles captured by professional photography equipment (Alimo; “To prepare for 3D generation of the environment 100, the user 102 may capture one or more images (e.g., 2D images and/or videos) using an image capture device 104. The image capture device 104 may be a personal electronic device carried by the user 102, such as a mobile phone, tablet, computer, or a combination thereof. As discussed in further detail below, the image capture device 104 may be connected, such as via a wireless network connection, to a system configured to generate the 3D representation of the environment 100. It should also be noted that additional image capture devices may be used to capture the images. For example, the image capture device 104 may be a first image capture device configured to capture a first set of images of the environment 100. A second image capture device may be configured to capture a second set of images of the environment 100. The first set of images and the second set of images may both be used for the 3D generation of the environment 100.
The user 102 may capture the one or more images using the image capture device 104 from various viewpoints and locations within or surrounding the environment 100. The user 102 need not capture every aspect or portion of the environment 100 to successfully generate a 3D representation of the environment 100. For example, the user 102 may capture images of one or more portions of the environment 100 using the image capture device 104 so that, during generation of the 3D representation of the environment 100, one or more novel portions of the environment 100 may be created and displayed even though such novel portions were not captured in the images," (page 3, para [0034] - [0035]; page 4, para [0042]).
Alimo does not differentiate between one or more images and videos, and one or more images includes videos. To prepare for 3D generation of the environment, the user may capture one or more images (e.g., 2D images and/or videos) using an image capture device reads on including videos. The image capture device 104 may be a personal electronic device carried by the user 102, such as a mobile phone, tablet, computer, or a combination thereof reads on captured by ordinary mobile phones. The user 102 may capture the one or more images using the image capture device 104 from various viewpoints and locations within or surrounding the environment 100 reads on multiple consecutive photos or a continuous video from different angles.).
Regarding claim 12, Alimo in view of Avramescu in further view of Son in further view of Shen teaches the three-dimensional (3D) model reconstruction and rendering apparatus of claim 11, further including providing high-resolution reconstruction of said generated 3D models and scenes by said high-resolution reconstruction module (Son; “The low resolution reconstruction unit 30, the high resolution reconstruction unit 50, or both, may be connected to various systems that use a 3D model, and the high resolution 3D model may be used in a wide variety of application systems that use animation movies, games, and characters.
The 3D model reconstruction system 1 according to the present embodiment is a hybrid 3D model reconstruction system that separates reconstruction of a low resolution 3D model from reconstruction of a high resolution 3D model. The reconstruction of a high resolution 3D model requires a large number of computations and a long computation time, thereby making real-time monitoring difficult. Furthermore, a high performance system is required to generate a high resolution 3D model in real-time.
The 3D model reconstruction system 1 according to the present embodiment reconstructs a low resolution 3D model in real-time by using a low resolution depth map, thereby allowing real-time monitoring. The 3D model reconstruction system 1 also reconstructs a high resolution 3D model by using a high resolution depth map and the result of reconstruction of a low resolution 3D model, thereby enabling the use for an embedded system instead of a high performance system," (page 3, para [0054]-[0056]).
The reconstruction of the high resolution 3D model reads on providing high-resolution reconstruction. After combination, the reconstruction of the generated composite 3D model (3D models and scenes) from Alimo is reconstructed according to the method disclosed by Son.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Son to Alimo in view of Avramescu in further view of Shen. The motivation would have been to improve the accuracy and detail of the 3D models and scenes.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERICA G THERKORN whose telephone number is (571)272-2939. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ERICA G THERKORN/Examiner, Art Unit 2618
/DEVONA E FAULK/Supervisory Patent Examiner, Art Unit 2618