DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are pending under this Office action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 10-15, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ackerson, etc. (US 20230281955 A1) in view of Metzler, etc. (US 20220005332 A1).
Regarding claim 1, Ackerson teaches that an image generation method (See Ackerson: Figs. 3A-G, and [0070], “With reference to FIG. 3A, some embodiments of the invention are operable to reconstruct a plenoptic field, including using incremental processes, where the plenoptic field may represent an entire scene, a portion of a scene, or a particular object or region of interest in a scene. In some embodiments, the system may first determine settings for the reconstruction of the scene 301. For example, the system may access or set a working resolution, initial size, target accuracy, relightability characteristics, or other characteristics. In some embodiments of the invention, the system may give an initial size to the scene. The size of the scene could be, for example, on the scale of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, user, or other factor that may be determined to be acceptable or advantageous. In some embodiments, including the exemplary embodiment depicted in FIG. 1B, the first camera 105 or set of image data may define the origin of the scene, and subsequent camera images, either captured by camera 105, a second camera or image sensing device 106, or otherwise, may be added to the scene and processed”), comprising:
obtaining density information by a target model corresponding to a target scene space (See Ackerson: Figs. 1A-E, and [0138], “In some embodiments, the higher-resolution information may be processed immediately and/or stored for later processing. The system may then construct a 3D model using all or a subset of the data available. In some embodiments, the highest resolution of the spatial model would roughly correspond to the projected sizes of the pixels”; and [0187], “In certain embodiments, the present invention may be used in conjunction with, in parallel with, be supplemented by, or otherwise implemented using in whole or in part artificial intelligence (AI), machine learning (ML), and neural networks, including neural radiance networks, such as Neural Radiance Fields, or NeRFs, volumetric scene methods such as PlenOctrees or Plenoxels, Deep Signed Distance Functions (SDF), and Neural Volumes. or in other technology. Such methods may be used for scene reconstruction, novel view synthesis (NVS) and other uses to model radiance, density, or other information as a continuous function in a 3D space such as a volumetric space using a multilayer perceptron (MLP) or voxels such as voxel arrays. For example, given a location in space with 3D coordinates (x, y, and z) and a viewing direction, the representation will return a color (red, green, and blue) and density at that location. Deep SDF systems may be configured to learn a signed distance function in 3D space whose zero level-set represents a 2D surface. Neural Volumes systems may be configured as neural graphics primitives that may be parameterized by fully connected neural networks. NeRF systems may be configured to model the color and density of a scene. Other embodiments operate with alternative input and return information. In certain embodiments, the returned density may be a differential opacity value which includes, in part or in whole, an estimate of the radiance and other information such as color that could be accumulated by a ray in the specified direction through the specified point”. Note that the scene reconstruction system builds a scene model representing the 3D space using the density information (volumetric) for points/voxels in the scene, the scene model is mapped to the target model, and the modeling 3D scene space is mapped to the target scene space), wherein
the target model is trained to output a viewpoint image corresponding to any viewpoint after inputting the any viewpoint in the target scene space (See Ackerson: Fig. 16, and [0169], “The residual layer 1604 may be applied to the predicted solution. The residual layer is optionally designed to ensure that the predicted solution satisfies the governing physics equations of the system. The residual layer may take partial derivatives of the predicted solution from the fully connected neural network with respect to the input coordinates and time, and enforce the physics equations governing the predicted solution 1603. The output of the residual layer 1605 may then be combined with a loss function that may include one or both of data constraints (such as known boundary conditions or initial conditions) and physics constraints (such as conservation laws or other governing equations). The loss function may be used to train the neural network to minimize the difference between the predicted solution and the observed data while still satisfying the underlying physics”); and
the density information is a rendering parameter required for image rendering by the target model, and the density information represents transparency of a point in the target scene space (See Ackerson: Figs. 1A-E, and [0187], “In certain embodiments, the present invention may be used in conjunction with, in parallel with, be supplemented by, or otherwise implemented using in whole or in part artificial intelligence (AI), machine learning (ML), and neural networks, including neural radiance networks, such as Neural Radiance Fields, or NeRFs, volumetric scene methods such as PlenOctrees or Plenoxels, Deep Signed Distance Functions (SDF), and Neural Volumes. or in other technology. Such methods may be used for scene reconstruction, novel view synthesis (NVS) and other uses to model radiance, density, or other information as a continuous function in a 3D space such as a volumetric space using a multilayer perceptron (MLP) or voxels such as voxel arrays. For example, given a location in space with 3D coordinates (x, y, and z) and a viewing direction, the representation will return a color (red, green, and blue) and density at that location. Deep SDF systems may be configured to learn a signed distance function in 3D space whose zero level-set represents a 2D surface. Neural Volumes systems may be configured as neural graphics primitives that may be parameterized by fully connected neural networks. NeRF systems may be configured to model the color and density of a scene. Other embodiments operate with alternative input and return information. In certain embodiments, the returned density may be a differential opacity value which includes, in part or in whole, an estimate of the radiance and other information such as color that could be accumulated by a ray in the specified direction through the specified point”; and [0189], “In some embodiments, the foregoing process may generate novel viewpoints with a high degree of realism after some level of “learning.” For example, this may be through AI, such as converging on estimated color values within a scene. In certain processes known in the art, the use of neural radiance networks or volumetric representations to generate novel-viewpoint images can require significant processing and/or time. Certain queries may require perhaps 500,000 to 1,000,000 multiplication and/or other operations for each point on the ray. Certain prior systems may require 30 seconds or more to generate a single 800-pixel by 800-pixel image on a powerful graphics processing unit (“GPU”), such as an Nvidia V100”. Note that density/opacity is a core rendering parameter in the volumetric model, it directly represents transparency/opacity of the points that is used for accurate image synthesis, and this teaching is mapped to the current cited limitation);
determining a first constraint condition based on the density information (See Ackerson: Figs. 3A-G, and [0018], “In some embodiments, the scene reconstruction may comprise processing the image data by postulating the orientation of the digital scene data. The processing of the image data may include (i) postulating that media exists in a voxel; (ii) postulating one or more of a surface normal, a light interaction property, an exitant radiance vector, an incident light field of the media, among other properties; (iii) calculating a cost for the existence of the media in the voxel based on the postulated one or more of a surface normal, a light interaction property (e.g., a refractive index, roughness, polarized diffuse coefficient, unpolarized diffuse coefficient, or extinction coefficient), an exitant radiance vector, and an incident light field of the media; (iv) comparing the cost to a cost threshold; and (iv) accepting media as existing at a voxel when the cost is below the cost threshold”; [0086], “Such operation may optionally use an actual value and/or change in value of a radiance, other radiel characteristic(s), and/or a confidence (consistency) of radiel characteristics to decide when to terminate that sequence of ops. For example, the system may also be configured to include a specific termination criteria, computation budget, or other threshold 342, including with regard to a light transport depth reflecting an iterative and/or recursive set of calculations. In such embodiments, the system may determine if the termination criteria, computation budget, or other threshold has been exceeded as discussed elsewhere herein. If the threshold has not been exceeded, the system may be configured to repeat the process, for example beginning at step 341. If the threshold has been exceeded, the system may complete the process”; and Figs. 14A-C, and [0108], “In some embodiments, the invention may perform the foregoing calculations for other mediels (or all mediels) within the region or scene of interest. The system may then compare the results for one or more of the mediels to the confidence threshold or other metric, for example based on predicted radiometric characteristic minus observed characteristics associated with exitant radiels of the mediel. For mediels where the confidence threshold or other metric is not achieved, the system may be configured to perform further processing related to such mediels. For example, FIG. 14B depicts a circumstances where the system has determined the bottom right mogel 1403 of mediel 1401 did not meet the appropriate threshold. In some embodiments, the system may subdivide such mediels not meeting the threshold or other metric into two or more child mediels, such as dividing a cube-shaped mediel into eight child cube-shaped mediels. In FIG. 14B, the system has subdivided mogel 1403 into four sub-mediels 1406, each of which having an associated content postulation 1407 and confidence 1408. In the embodiment depicted in FIG. 14B, the system has now postulated sub-mediel 1409 to contain a surface, as denoted at 1413, for example, an opaque dielectric surface that may be represented by a surfel, denoted by “S” with a confidence of 50. The remaining sub mediels remain postulated as containing air with varying degrees of confidence”. Note that the density information (such as a refractive index, roughness, polarized diffuse coefficient, unpolarized diffuse coefficient, or extinction coefficient, opacity, transparency, etc.) are used to determine if the mediels exitant in voxel by comparing the values to a threshold, which is mapped to the density related constraints);
determining a target viewpoint from the target scene space (See Ackerson: Fig. 3A-G, and [0111], “With reference to FIG. 3D, the system may be configured to incorporate new image data. In some embodiments, the system may initialize one or more new camera poses 331, which may be accomplished, for example, as described with reference to FIGS. 3B and 3F. Some embodiments of the invention may then place one or more new radiels into the scene at voxels containing one or more new viewpoints 332”; and [0272], “The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint”) based on the first constraint condition; and
rendering a viewpoint image corresponding to the target viewpoint by the target model (See Ackerson: Fig. 21, and [0136], “In some circumstances, there may be multiple related pixels in a region of the image. If the number of related pixels is sufficient, certain embodiments of the invention may perform a statistical analysis of the texture. Such a statistical analysis may involve the application of a set of one or more filters to the region, and preferably would include clusters of the responses to the one or more filters assembled into a texture signature. In this example, a calculated texture signature may then be added as a property to the scene model and later used to insert synthetically generated textures into renderings to provide for realistic views”; and [0241], “In certain embodiments of this invention, the human-computer interface is capable of rendering and displaying a finished reconstruction. In some embodiments, displays may include analytic visualizations in addition to realistic views”).
However, Ackerson fails to explicitly disclose that (a target viewpoint is determined) based on the first constraint condition.
However, Metzler teaches that (a target viewpoint is determined) based on the first constraint condition (See Metzler: Fig. 9, and [0344], “The acquisition position and/or orientation for generation of such state verification information is optionally determined analogue to a next-best-view method (NBV) known in the art. The goal of the next best view planning according to the invention is to recognize state relevant features with high certainty (wherefore it is not necessary to recognize or survey the door 108 as a whole). In the example, a relevant feature might be the lock of door 108 and/or edges of the door panel 123 or the adjacent wall 121 or presence or form of a shadow or light cone. Preferably, the computing unit of robot 100 provides a database with event relevant features of a plurality or multitude of objects of the property to be under surveillance”. Note that NBV is a well-known algorithm used to determine the optimal views).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Ackerson to have (a target viewpoint is determined) based on the first constraint condition as taught by Metzler in order to improve the overall detection rate respectively reduce the false alarm rate, in particular in view of contradicting surveillance sensor results (See Metzler: Fig. 1, and [0176], “Thereby, the invention is more than a simple independent side by side usage or a hierarchical surveillance-sensor approach of prior art, which can overcome many prior art drawbacks and can improve the overall detection rate respectively reduce the false alarm rate, in particular in view of at least partially contradicting surveillance sensor results.”). Ackerson teaches a method and system that may perform a scene reconstruction using machine learning algorithm based on various scene information, such as density information, and generate a user selected viewpoint image for the user; while Metzler teaches a system and method that may use the next-best-view algorithm to determine the best view from plurality views for the user in the surveillance system. Therefore, it is obvious to one of ordinary skill in the art to modify Ackerson by Metzler to use the next-best-view algorithm to determine the best view from multiple views for the user. The motivation to modify Ackerson by Metzler is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 2, Ackerson and Metzler teach all the features with respect to claim 1 as outlined above. Further, Ackerson teaches that the method according to claim 1, wherein determining the first constraint condition based on the density information comprises:
determining a first restriction space in the target scene space based on the density information (See Ackerson: Figs. 1A-E, and [0054], “Various exemplary embodiments of scene models are depicted in FIGS. 1C-1E. A scene model 110 may comprise a matter field 120 and a light field 130, either in a single model, as depicted in FIG. 1C, or separate, as depicted in FIG. 1D (matter field) and FIG. 1E (light field). A scene may have external illumination 112 flowing into the scene and providing a source of light in the scene. A scene may also be a unitary scene, wherein there is no light flowing into the scene 112. A scene may have a boundary 115, which may optionally be defined by the system during reconstruction of the scene, by a physical boundary in the scene, by a user or other input, by some combination of the foregoing, or otherwise. Information beyond the boundary of a scene may be considered a frontier 117, and may not be represented in the scene. However, in some embodiments, the boundary 115 may comprise a fenestral boundary 111 in whole or in part. A fenestral boundary 111 may be a portion of a scene boundary 115 through which incident light 112 may flow into the scene and exitant light 116 may flow out of the scene. In some embodiments, portions of the frontier 117 may be represented, at least in part, at the fenestral boundary 111. By way of example, a fenestral boundary 111 may be defined based on a physical feature in the scene (e.g., a window or skylight in a wall or ceiling through which light can enter the scene), based on a scene parallax (e.g., a boundary based on a distance or lack of resolution of image data, such as for an outdoor night scene looking at the sky where there is very long range in the field of view), some combination of the two, or some other factor. The scene may include one or more objects, including responsive objects 113 and emissive objects 114. Emissive objects 114 may emit light independent of light incident to the object, whereas responsive objects may interact with incident light without emitting light themselves”; and [0059], “A scene may contain one or more voxels, each of which may be the same size and shape, or may be selected from a range of sizes and/or shapes as determined by the user or the system. A voxel may contain a mediel, or media element, that may represent all or a portion of media sampled in the voxel. Media is a volumetric region that includes some or no matter in which light flows. Media can be homogeneous or heterogeneous. Examples of homogeneous media include: empty space, air and water. Examples of heterogeneous media include volumetric regions including the surface of a mirror (part air and part slivered glass), the surface of a pane of glass (part air and part transmissive glass) and the branch of a pine tree (part air and part organic material). Light flows in media by phenomena including absorption, reflection, transmission and scattering. Examples of media that is partially transmissive includes the branch of a pine tree and a pane of glass”; and Figs. 14A-C, and [0104], “In some embodiments, such as depicted in FIG. 14A, a scene of interest may contain both homogenous transmissive media and opaque media. In embodiments where the region of interest includes empty space (e.g., air or otherwise homogenous space not containing media substantially interacting with light), the system may specify within the data structure that the scene is comprised of mediels comprising empty space (e.g., air). Empty mediels or mediels comprising air and other homogenous elements may be referred to as mogels. In some embodiments, even if a mediel 1401 contains an opaque surface 1402, the system may initially stipulate that the mediel 1401 is comprised of one or more mogels 1403 comprising empty space or air (or air mogel), with such initialization would allow the system to let light flow through the mediel 1401 and mogels 1403 rather than being postulated to be blocked by interacting media, such as 1402. Some embodiments of the invention may specify a low confidence 1405 associated with each of the air mogels, which can facilitate the system later determining the presence of other media within each air mogel. In FIG. 14A, the postulated contents 1404 and confidence 1405 are depicted, with contents “A” 1404 representing an initial postulation of the mogel 1403 containing air and confidence “10” 1405 representing a hypothetical confidence value associated with that postulation”. Note that the target model (neural/volumetric) builds a scene representation (e.g., voxel, mediels, radiels, surfels,, etc.; and high density regions (the region associated with the object of interest may be the restricted regions) are identified as “restricted” or occupied regions),
wherein a density value of a point in the first restriction space is greater than a first density threshold (See Ackerson: Figs. 3A-G, and [0070], “With reference to FIG. 3A, some embodiments of the invention are operable to reconstruct a plenoptic field, including using incremental processes, where the plenoptic field may represent an entire scene, a portion of a scene, or a particular object or region of interest in a scene. In some embodiments, the system may first determine settings for the reconstruction of the scene 301. For example, the system may access or set a working resolution, initial size, target accuracy, relightability characteristics, or other characteristics. In some embodiments of the invention, the system may give an initial size to the scene. The size of the scene could be, for example, on the scale of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, user, or other factor that may be determined to be acceptable or advantageous. In some embodiments, including the exemplary embodiment depicted in FIG. 1B, the first camera 105 or set of image data may define the origin of the scene, and subsequent camera images, either captured by camera 105, a second camera or image sensing device 106, or otherwise, may be added to the scene and processed”; [0086], “Such operation may optionally use an actual value and/or change in value of a radiance, other radiel characteristic(s), and/or a confidence (consistency) of radiel characteristics to decide when to terminate that sequence of ops. For example, the system may also be configured to include a specific termination criteria, computation budget, or other threshold 342, including with regard to a light transport depth reflecting an iterative and/or recursive set of calculations. In such embodiments, the system may determine if the termination criteria, computation budget, or other threshold has been exceeded as discussed elsewhere herein. If the threshold has not been exceeded, the system may be configured to repeat the process, for example beginning at step 341. If the threshold has been exceeded, the system may complete the process”; and Fig. 5, and [0066], “As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable”. Note that the scene regions may contain high density regions, such as water, objects, glasses, etc., and the regions with objects have density higher than the empty/air regions); and
determining the first constraint condition based on the first restriction space (See Ackerson: Figs. 1A-E, and [0050], “As described in various embodiments herein, one aim of the present invention is to provide systems and methods for performing scene reconstruction, and particularly performing generalized scene reconstruction (GSR). In some embodiments, the result of GSR processes or systems may result in a reconstruction of a light field, a matter field (including a relightable matter field), characterization of camera poses, or any combination of the foregoing. The result of GSR processes may result in a model representing a scene based upon the reconstructed light field or matter field (including the relightable matter field) individually and separately, or of the two together, as may be desirable under the circumstances. As used herein, a scene may refer to the entire scope of the light and/or matter field represented in an image, any portion thereof, or any media therein. Although the terms subscene, portion of a scene, region of interest, object of interest, and other similar terminology may be used to refer to a portion of a larger scene, each of the foregoing is itself a scene”; and [0272], “The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint”. Note that the high-density regions, such as objects of interest, regions of interest, etc. define the constraint for viewpoint selections, sampling and or rendering (to avoid empty space), and this is mapped to the viewpoint determinations).
Regarding claim 3, Ackerson and Metzler teach all the features with respect to claim 2 as outlined above. Further, Ackerson teaches that the method according to claim 2, wherein determining the target viewpoint from the target scene space based on the first constraint condition comprises:
determining a target point from a target space (See Ackerson: Figs. 1A-E, and [0143], “As a third example, the embodiments described herein may be used to reconstruct a scene including water, objects submerged or partially submerged in water, one or more water drops entering a body of water, such as a swimming pool, or objects submerged in water or another liquid. In one such example, multiple water droplets and a nearby body of water may be reconstructed. In certain embodiments, the droplets may be modelled moving to and entering the water body according to the laws of physics or other characteristics that may be provided to or known by the system. In some embodiments, the droplets may be represented volumetrically, which provides a basis for the system to calculate the mass properties of each drop using known mass properties of water. The system then may, based in whole, in part, or otherwise, on the mass and/or center-of-mass of a drop, model the trajectory of each such drop to the water. In some embodiments, the system may optionally include an advanced modeling system, which may support deformations of one or more of the drops or of the swimming pool”),
wherein the target space is a space in the target scene space excluding the first restriction space (See Ackerson: Figs. 1A-E, and [0059], “A scene may contain one or more voxels, each of which may be the same size and shape, or may be selected from a range of sizes and/or shapes as determined by the user or the system. A voxel may contain a mediel, or media element, that may represent all or a portion of media sampled in the voxel. Media is a volumetric region that includes some or no matter in which light flows. Media can be homogeneous or heterogeneous. Examples of homogeneous media include: empty space, air and water. Examples of heterogeneous media include volumetric regions including the surface of a mirror (part air and part slivered glass), the surface of a pane of glass (part air and part transmissive glass) and the branch of a pine tree (part air and part organic material). Light flows in media by phenomena including absorption, reflection, transmission and scattering. Examples of media that is partially transmissive includes the branch of a pine tree and a pane of glass”. Note that when the regions of objects with high density are mapped to the “restricted” region, the water drops can be mapped to the “target point” in the scene, and is it excluded from the restricted regions); and
determining the target viewpoint based on the target point (See Ackerson: Fig. 3A-G, and [0111], “With reference to FIG. 3D, the system may be configured to incorporate new image data. In some embodiments, the system may initialize one or more new camera poses 331, which may be accomplished, for example, as described with reference to FIGS. 3B and 3F. Some embodiments of the invention may then place one or more new radiels into the scene at voxels containing one or more new viewpoints 332”; and [0272], “The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint”).
Regarding claim 4, Ackerson and Metzler teach all the features with respect to claim 2 as outlined above. Further, Ackerson teaches that the method according to claim 2, wherein the density value of the point in the first restriction space is less than a second density threshold, and the second density threshold is greater than the first density threshold (See Ackerson: Fig. 5, and [0066], “As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable”; and [0194], “Certain embodiments of the invention may use ML to reconstruct the light field in a scene, including, in some circumstances, constructing a physics model of the interactions of light and a matter field in the scene. In such embodiments, the system may decouple components that may contribute to the light sensed by camera pixels or another imaging device. This data may be used to determine the characteristics of the matter and objects in the scene, including non-Lambertian surfaces (e.g., human skin, cloth, mirrors, glass, and water). In some embodiments, certain surface information may be represented in a Bidirectional Light Interaction Function (BLIF) for one or more sensed locations on an object, and optionally all sensed locations on an object. The sensed locations may include locations captured by individual pixels of a camera or imaging device. The present invention may use the BLIF, and modeling based on BLIFs, to extend concepts such as a Bidirectional Reflectance Distribution Function (BDRF) and/or cosine lobe reflectance models to develop a greater level of sophistication by include light/matter interactions involving color, material, roughness, polarization, and so on”. Note that air, water, and solid objects like glass are inherently of different density: the first density threshold is used to distinguish the air (lowest density) and the water (intermediate density), and the second density threshold is used to distinguish the water and the solid objects like glass (high density), then, the claimed limitation is obvious).
Regarding claim 5, Ackerson and Metzler teach all the features with respect to claim 2 as outlined above. Further, Ackerson teaches that the method according to claim 2, wherein determining the first constraint condition based on the first restriction space comprises:
determining structural information in the target scene space (See Ackerson: Fig. 21, and [0116], “In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures”);
determining a second restriction space in the target scene space based on the structural information (See Ackerson: Fig. 21, and [0049, “FIG. 21 is an illustration of a combination of a reconstruction performed with the methods described herein and a reconstruction created with another method”; and [0116], “In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures”. Note that 2102 is mapped to the second restriction space); and
determining the first constraint condition based on the first restriction space and the second restriction space (See Ackerson: Fig. 21, and [0116], “In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures”. Note that 2101 is mapped to the first restriction space).
Regarding claim 6, Ackerson and Metzler teach all the features with respect to claim 5 as outlined above. Further, Ackerson teaches that the method according to claim 5, wherein the second restriction space comprises a structured information area in the target scene space and an unstructured information area within a specific distance from the structured information area (See Ackerson: Fig. 21, and [0116], “In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures”. Note that 2102 is a bone with complex shape structure, and 2101 is almost a homogeneous region mapped to “unstructured information area”).
Regarding claim 7, Ackerson and Metzler teach all the features with respect to claim 1 as outlined above. Further, Ackerson teaches that the method according to claim 1, wherein
the first constraint condition is that a density value of a point satisfies a density threshold limitation (See Ackerson: Figs. 14A-C, and [0103], “Some embodiments of the invention may then use a confidence threshold or other metric to guide processing, and calculate an associated confidence or other metric associated with each mediel or other volumetric element within the scene. If a confidence threshold is used, the system may examine one or more mediels where the confidence is below the confidence threshold. In some embodiments, if the confidence is below the threshold, the system may then compare the characteristics of the mediel with various known light interaction characteristics, such as a bidirectional light interaction function (or BLIF) associated with different types of media. For example, in the example depicted in FIG. 14A, if the confidence threshold is 75, the system may be configured to perform further calculations on each of the four depicted mogels 1403 because the associated confidence is below 75. Some embodiments may use a waterfall, or sequential, order of comparison based upon what the system has calculated to be the most likely candidate characteristics for the particular mediel (e.g., most likely candidate BLIF). For example, for a particular mediel, the system may first test the mediel for containing air, then a general dielectric media, then a general metallic media, and so on”); and
determining the target viewpoint from the target scene space based on the first constraint condition (See Ackerson: Figs. 3A-G, and [0080], “The system may calculate or refine information regarding one or more poses associated with image data 322, as described with reference to FIG. 3F and elsewhere. In some embodiments, the system may determine if one or more camera or image data viewpoints containing voxel's light field has changed 322, which may optionally be determined based on some threshold of significance which could be preset or calculated by the system. This determination may be based, in part, on the system postulating or having other information indicating a camera image or set of image data exists at a voxel 201 in the data structure, as depicted in FIG. 2. In such embodiments, for each postulated position, the system may postulate an orientation in a coarse orientation space”) comprises:
determining a preselected point from the target scene space (See Ackerson: Figs. 1A-E, and [0143], “As a third example, the embodiments described herein may be used to reconstruct a scene including water, objects submerged or partially submerged in water, one or more water drops entering a body of water, such as a swimming pool, or objects submerged in water or another liquid. In one such example, multiple water droplets and a nearby body of water may be reconstructed. In certain embodiments, the droplets may be modelled moving to and entering the water body according to the laws of physics or other characteristics that may be provided to or known by the system. In some embodiments, the droplets may be represented volumetrically, which provides a basis for the system to calculate the mass properties of each drop using known mass properties of water. The system then may, based in whole, in part, or otherwise, on the mass and/or center-of-mass of a drop, model the trajectory of each such drop to the water. In some embodiments, the system may optionally include an advanced modeling system, which may support deformations of one or more of the drops or of the swimming pool”);
determining the preselected point as a target point when a density value of the preselected point satisfies the density threshold limitation (See Ackerson: Figs. 1A-E, and [0144], “In some embodiments, the movement of a droplet may be modeled at discrete instances in time. At a point in time where a drop may first enter the larger segment representing the water body, an operation may be performed to determine the volume of water that is common between the swimming pool and the droplet. The system may then use the results of such an operation to compensate for a volume increase in the larger segment, which may optionally be accomplished using a morphological dilation operation. Upon such an operation, one or more volume elements on the larger segment surface (the swimming pool) that interface with movable material (a drop) may be extended incrementally to compensate for the displaced water volume and may be further modified to account for the dynamic reaction of the segment surface to the interaction with the movable material. The system may use such tools and similar tools to implement a more advanced displacement model. In some embodiments, the overall process may continue for additional water displacements until the droplet has become fully incorporated into the body of water”. Note that the density of the water drop is the same as the density of the water, and this is mapped to the current cited limitation, “the preselected point” (the water drop) satisfies the density threshold limitation); and
determining the target viewpoint based on the target point (See Ackerson: Fig. 3A-G, and [0111], “With reference to FIG. 3D, the system may be configured to incorporate new image data. In some embodiments, the system may initialize one or more new camera poses 331, which may be accomplished, for example, as described with reference to FIGS. 3B and 3F. Some embodiments of the invention may then place one or more new radiels into the scene at voxels containing one or more new viewpoints 332”; and [0272], “The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint”).
Regarding claim 8, Ackerson and Metzler teach all the features with respect to claim 7 as outlined above. Further, Ackerson teaches that the method according to claim 7, wherein determining the first constraint condition based on the density information comprises:
obtaining a density value variation range from the density information (See Ackerson: Fig. 5, and [0066], “As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable”; and [0194], “Certain embodiments of the invention may use ML to reconstruct the light field in a scene, including, in some circumstances, constructing a physics model of the interactions of light and a matter field in the scene. In such embodiments, the system may decouple components that may contribute to the light sensed by camera pixels or another imaging device. This data may be used to determine the characteristics of the matter and objects in the scene, including non-Lambertian surfaces (e.g., human skin, cloth, mirrors, glass, and water). In some embodiments, certain surface information may be represented in a Bidirectional Light Interaction Function (BLIF) for one or more sensed locations on an object, and optionally all sensed locations on an object. The sensed locations may include locations captured by individual pixels of a camera or imaging device. The present invention may use the BLIF, and modeling based on BLIFs, to extend concepts such as a Bidirectional Reflectance Distribution Function (BDRF) and/or cosine lobe reflectance models to develop a greater level of sophistication by include light/matter interactions involving color, material, roughness, polarization, and so on”. Note that air, water, and solid objects like glass are inherently of different density: in Physics, the density of water is defined as 1, vacuum density is zero (or almost zero, air density is less than 1 (depending on the temperature and pressure), while most solid objects have density more than 1 (except ice); to identifying air, water, and glass by their density has inherently (by Physics) a density range); and
determining the first constraint condition based on the density value variation range (See Ackerson: Fig. 16, and [0169], “The residual layer 1604 may be applied to the predicted solution. The residual layer is optionally designed to ensure that the predicted solution satisfies the governing physics equations of the system. The residual layer may take partial derivatives of the predicted solution from the fully connected neural network with respect to the input coordinates and time, and enforce the physics equations governing the predicted solution 1603. The output of the residual layer 1605 may then be combined with a loss function that may include one or both of data constraints (such as known boundary conditions or initial conditions) and physics constraints (such as conservation laws or other governing equations). The loss function may be used to train the neural network to minimize the difference between the predicted solution and the observed data while still satisfying the underlying physics”).
Regarding claim 10, Ackerson and Metzler teach all the features with respect to claim 1 as outlined above. Further, Ackerson and Metzler teach that an electronic device, comprising one or more processors and a memory containing a computer program that, when being executed, causes the one or more processors (See Ackerson: Figs. 3A-G, and [0070], “With reference to FIG. 3A, some embodiments of the invention are operable to reconstruct a plenoptic field, including using incremental processes, where the plenoptic field may represent an entire scene, a portion of a scene, or a particular object or region of interest in a scene. In some embodiments, the system may first determine settings for the reconstruction of the scene 301. For example, the system may access or set a working resolution, initial size, target accuracy, relightability characteristics, or other characteristics. In some embodiments of the invention, the system may give an initial size to the scene. The size of the scene could be, for example, on the scale of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, user, or other factor that may be determined to be acceptable or advantageous. In some embodiments, including the exemplary embodiment depicted in FIG. 1B, the first camera 105 or set of image data may define the origin of the scene, and subsequent camera images, either captured by camera 105, a second camera or image sensing device 106, or otherwise, may be added to the scene and processed”) to:
obtain density information by a target model corresponding to a target scene space (See Ackerson: Figs. 1A-E, and [0138], “In some embodiments, the higher-resolution information may be processed immediately and/or stored for later processing. The system may then construct a 3D model using all or a subset of the data available. In some embodiments, the highest resolution of the spatial model would roughly correspond to the projected sizes of the pixels”; and [0187], “In certain embodiments, the present invention may be used in conjunction with, in parallel with, be supplemented by, or otherwise implemented using in whole or in part artificial intelligence (AI), machine learning (ML), and neural networks, including neural radiance networks, such as Neural Radiance Fields, or NeRFs, volumetric scene methods such as PlenOctrees or Plenoxels, Deep Signed Distance Functions (SDF), and Neural Volumes. or in other technology. Such methods may be used for scene reconstruction, novel view synthesis (NVS) and other uses to model radiance, density, or other information as a continuous function in a 3D space such as a volumetric space using a multilayer perceptron (MLP) or voxels such as voxel arrays. For example, given a location in space with 3D coordinates (x, y, and z) and a viewing direction, the representation will return a color (red, green, and blue) and density at that location. Deep SDF systems may be configured to learn a signed distance function in 3D space whose zero level-set represents a 2D surface. Neural Volumes systems may be configured as neural graphics primitives that may be parameterized by fully connected neural networks. NeRF systems may be configured to model the color and density of a scene. Other embodiments operate with alternative input and return information. In certain embodiments, the returned density may be a differential opacity value which includes, in part or in whole, an estimate of the radiance and other information such as color that could be accumulated by a ray in the specified direction through the specified point”. Note that the scene reconstruction system builds a scene model representing the 3D space using the density information (volumetric) for points/voxels in the scene, the scene model is mapped to the target model, and the modeling 3D scene space is mapped to the target scene space),
wherein the target model is trained to output a viewpoint image corresponding to any viewpoint after inputting the any viewpoint in the target scene space (See Ackerson: Fig. 16, and [0169], “The residual layer 1604 may be applied to the predicted solution. The residual layer is optionally designed to ensure that the predicted solution satisfies the governing physics equations of the system. The residual layer may take partial derivatives of the predicted solution from the fully connected neural network with respect to the input coordinates and time, and enforce the physics equations governing the predicted solution 1603. The output of the residual layer 1605 may then be combined with a loss function that may include one or both of data constraints (such as known boundary conditions or initial conditions) and physics constraints (such as conservation laws or other governing equations). The loss function may be used to train the neural network to minimize the difference between the predicted solution and the observed data while still satisfying the underlying physics”), and
the density information is a rendering parameter required for image rendering by the target model, and the density information represents transparency of a point in the target scene space (See Ackerson: Figs. 1A-E, and [0187], “In certain embodiments, the present invention may be used in conjunction with, in parallel with, be supplemented by, or otherwise implemented using in whole or in part artificial intelligence (AI), machine learning (ML), and neural networks, including neural radiance networks, such as Neural Radiance Fields, or NeRFs, volumetric scene methods such as PlenOctrees or Plenoxels, Deep Signed Distance Functions (SDF), and Neural Volumes. or in other technology. Such methods may be used for scene reconstruction, novel view synthesis (NVS) and other uses to model radiance, density, or other information as a continuous function in a 3D space such as a volumetric space using a multilayer perceptron (MLP) or voxels such as voxel arrays. For example, given a location in space with 3D coordinates (x, y, and z) and a viewing direction, the representation will return a color (red, green, and blue) and density at that location. Deep SDF systems may be configured to learn a signed distance function in 3D space whose zero level-set represents a 2D surface. Neural Volumes systems may be configured as neural graphics primitives that may be parameterized by fully connected neural networks. NeRF systems may be configured to model the color and density of a scene. Other embodiments operate with alternative input and return information. In certain embodiments, the returned density may be a differential opacity value which includes, in part or in whole, an estimate of the radiance and other information such as color that could be accumulated by a ray in the specified direction through the specified point”; and [0189], “In some embodiments, the foregoing process may generate novel viewpoints with a high degree of realism after some level of “learning.” For example, this may be through AI, such as converging on estimated color values within a scene. In certain processes known in the art, the use of neural radiance networks or volumetric representations to generate novel-viewpoint images can require significant processing and/or time. Certain queries may require perhaps 500,000 to 1,000,000 multiplication and/or other operations for each point on the ray. Certain prior systems may require 30 seconds or more to generate a single 800-pixel by 800-pixel image on a powerful graphics processing unit (“GPU”), such as an Nvidia V100”. Note that density/opacity is a core rendering parameter in the volumetric model, it directly represents transparency/opacity of the points that is used for accurate image synthesis, and this teaching is mapped to the current cited limitation);
determine a first constraint condition based on the density information (See Ackerson: Figs. 3A-G, and [0018], “In some embodiments, the scene reconstruction may comprise processing the image data by postulating the orientation of the digital scene data. The processing of the image data may include (i) postulating that media exists in a voxel; (ii) postulating one or more of a surface normal, a light interaction property, an exitant radiance vector, an incident light field of the media, among other properties; (iii) calculating a cost for the existence of the media in the voxel based on the postulated one or more of a surface normal, a light interaction property (e.g., a refractive index, roughness, polarized diffuse coefficient, unpolarized diffuse coefficient, or extinction coefficient), an exitant radiance vector, and an incident light field of the media; (iv) comparing the cost to a cost threshold; and (iv) accepting media as existing at a voxel when the cost is below the cost threshold”; [0086], “Such operation may optionally use an actual value and/or change in value of a radiance, other radiel characteristic(s), and/or a confidence (consistency) of radiel characteristics to decide when to terminate that sequence of ops. For example, the system may also be configured to include a specific termination criteria, computation budget, or other threshold 342, including with regard to a light transport depth reflecting an iterative and/or recursive set of calculations. In such embodiments, the system may determine if the termination criteria, computation budget, or other threshold has been exceeded as discussed elsewhere herein. If the threshold has not been exceeded, the system may be configured to repeat the process, for example beginning at step 341. If the threshold has been exceeded, the system may complete the process”; and Figs. 14A-C, and [0108], “In some embodiments, the invention may perform the foregoing calculations for other mediels (or all mediels) within the region or scene of interest. The system may then compare the results for one or more of the mediels to the confidence threshold or other metric, for example based on predicted radiometric characteristic minus observed characteristics associated with exitant radiels of the mediel. For mediels where the confidence threshold or other metric is not achieved, the system may be configured to perform further processing related to such mediels. For example, FIG. 14B depicts a circumstances where the system has determined the bottom right mogel 1403 of mediel 1401 did not meet the appropriate threshold. In some embodiments, the system may subdivide such mediels not meeting the threshold or other metric into two or more child mediels, such as dividing a cube-shaped mediel into eight child cube-shaped mediels. In FIG. 14B, the system has subdivided mogel 1403 into four sub-mediels 1406, each of which having an associated content postulation 1407 and confidence 1408. In the embodiment depicted in FIG. 14B, the system has now postulated sub-mediel 1409 to contain a surface, as denoted at 1413, for example, an opaque dielectric surface that may be represented by a surfel, denoted by “S” with a confidence of 50. The remaining sub mediels remain postulated as containing air with varying degrees of confidence”. Note that the density information (such as a refractive index, roughness, polarized diffuse coefficient, unpolarized diffuse coefficient, or extinction coefficient, opacity, transparency, etc.) are used to determine if the mediels exitant in voxel by comparing the values to a threshold, which is mapped to the density related constraints);
determine a target viewpoint from the target scene space (See Ackerson: Fig. 3A-G, and [0111], “With reference to FIG. 3D, the system may be configured to incorporate new image data. In some embodiments, the system may initialize one or more new camera poses 331, which may be accomplished, for example, as described with reference to FIGS. 3B and 3F. Some embodiments of the invention may then place one or more new radiels into the scene at voxels containing one or more new viewpoints 332”; and [0272], “The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint”) based on the first constraint condition (See Metzler: Fig. 9, and [0344], “The acquisition position and/or orientation for generation of such state verification information is optionally determined analogue to a next-best-view method (NBV) known in the art. The goal of the next best view planning according to the invention is to recognize state relevant features with high certainty (wherefore it is not necessary to recognize or survey the door 108 as a whole). In the example, a relevant feature might be the lock of door 108 and/or edges of the door panel 123 or the adjacent wall 121 or presence or form of a shadow or light cone. Preferably, the computing unit of robot 100 provides a database with event relevant features of a plurality or multitude of objects of the property to be under surveillance”. Note that NBV is a well-known algorithm used to determine the optimal views); and
render a viewpoint image corresponding to the target viewpoint by the target model (See Ackerson: Fig. 21, and [0136], “In some circumstances, there may be multiple related pixels in a region of the image. If the number of related pixels is sufficient, certain embodiments of the invention may perform a statistical analysis of the texture. Such a statistical analysis may involve the application of a set of one or more filters to the region, and preferably would include clusters of the responses to the one or more filters assembled into a texture signature. In this example, a calculated texture signature may then be added as a property to the scene model and later used to insert synthetically generated textures into renderings to provide for realistic views”; and [0241], “In certain embodiments of this invention, the human-computer interface is capable of rendering and displaying a finished reconstruction. In some embodiments, displays may include analytic visualizations in addition to realistic views”).
Regarding claim 11, Ackerson and Metzler teach all the features with respect to claim 10 as outlined above. Further, Ackerson teaches that the device according to claim 10, wherein the one or more processors are further configured to:
determine a first restricted space in the target scene space based on the density information (See Ackerson: Figs. 1A-E, and [0054], “Various exemplary embodiments of scene models are depicted in FIGS. 1C-1E. A scene model 110 may comprise a matter field 120 and a light field 130, either in a single model, as depicted in FIG. 1C, or separate, as depicted in FIG. 1D (matter field) and FIG. 1E (light field). A scene may have external illumination 112 flowing into the scene and providing a source of light in the scene. A scene may also be a unitary scene, wherein there is no light flowing into the scene 112. A scene may have a boundary 115, which may optionally be defined by the system during reconstruction of the scene, by a physical boundary in the scene, by a user or other input, by some combination of the foregoing, or otherwise. Information beyond the boundary of a scene may be considered a frontier 117, and may not be represented in the scene. However, in some embodiments, the boundary 115 may comprise a fenestral boundary 111 in whole or in part. A fenestral boundary 111 may be a portion of a scene boundary 115 through which incident light 112 may flow into the scene and exitant light 116 may flow out of the scene. In some embodiments, portions of the frontier 117 may be represented, at least in part, at the fenestral boundary 111. By way of example, a fenestral boundary 111 may be defined based on a physical feature in the scene (e.g., a window or skylight in a wall or ceiling through which light can enter the scene), based on a scene parallax (e.g., a boundary based on a distance or lack of resolution of image data, such as for an outdoor night scene looking at the sky where there is very long range in the field of view), some combination of the two, or some other factor. The scene may include one or more objects, including responsive objects 113 and emissive objects 114. Emissive objects 114 may emit light independent of light incident to the object, whereas responsive objects may interact with incident light without emitting light themselves”; and [0059], “A scene may contain one or more voxels, each of which may be the same size and shape, or may be selected from a range of sizes and/or shapes as determined by the user or the system. A voxel may contain a mediel, or media element, that may represent all or a portion of media sampled in the voxel. Media is a volumetric region that includes some or no matter in which light flows. Media can be homogeneous or heterogeneous. Examples of homogeneous media include: empty space, air and water. Examples of heterogeneous media include volumetric regions including the surface of a mirror (part air and part slivered glass), the surface of a pane of glass (part air and part transmissive glass) and the branch of a pine tree (part air and part organic material). Light flows in media by phenomena including absorption, reflection, transmission and scattering. Examples of media that is partially transmissive includes the branch of a pine tree and a pane of glass”; and Figs. 14A-C, and [0104], “In some embodiments, such as depicted in FIG. 14A, a scene of interest may contain both homogenous transmissive media and opaque media. In embodiments where the region of interest includes empty space (e.g., air or otherwise homogenous space not containing media substantially interacting with light), the system may specify within the data structure that the scene is comprised of mediels comprising empty space (e.g., air). Empty mediels or mediels comprising air and other homogenous elements may be referred to as mogels. In some embodiments, even if a mediel 1401 contains an opaque surface 1402, the system may initially stipulate that the mediel 1401 is comprised of one or more mogels 1403 comprising empty space or air (or air mogel), with such initialization would allow the system to let light flow through the mediel 1401 and mogels 1403 rather than being postulated to be blocked by interacting media, such as 1402. Some embodiments of the invention may specify a low confidence 1405 associated with each of the air mogels, which can facilitate the system later determining the presence of other media within each air mogel. In FIG. 14A, the postulated contents 1404 and confidence 1405 are depicted, with contents “A” 1404 representing an initial postulation of the mogel 1403 containing air and confidence “10” 1405 representing a hypothetical confidence value associated with that postulation”. Note that the target model (neural/volumetric) builds a scene representation (e.g., voxel, mediels, radiels, surfels,, etc.; and high density regions (the region associated with the object of interest may be the restricted regions) are identified as “restricted” or occupied regions),
wherein a density value of a point in the first restricted space is greater than a first density threshold (See Ackerson: Figs. 3A-G, and [0070], “With reference to FIG. 3A, some embodiments of the invention are operable to reconstruct a plenoptic field, including using incremental processes, where the plenoptic field may represent an entire scene, a portion of a scene, or a particular object or region of interest in a scene. In some embodiments, the system may first determine settings for the reconstruction of the scene 301. For example, the system may access or set a working resolution, initial size, target accuracy, relightability characteristics, or other characteristics. In some embodiments of the invention, the system may give an initial size to the scene. The size of the scene could be, for example, on the scale of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, user, or other factor that may be determined to be acceptable or advantageous. In some embodiments, including the exemplary embodiment depicted in FIG. 1B, the first camera 105 or set of image data may define the origin of the scene, and subsequent camera images, either captured by camera 105, a second camera or image sensing device 106, or otherwise, may be added to the scene and processed”; [0086], “Such operation may optionally use an actual value and/or change in value of a radiance, other radiel characteristic(s), and/or a confidence (consistency) of radiel characteristics to decide when to terminate that sequence of ops. For example, the system may also be configured to include a specific termination criteria, computation budget, or other threshold 342, including with regard to a light transport depth reflecting an iterative and/or recursive set of calculations. In such embodiments, the system may determine if the termination criteria, computation budget, or other threshold has been exceeded as discussed elsewhere herein. If the threshold has not been exceeded, the system may be configured to repeat the process, for example beginning at step 341. If the threshold has been exceeded, the system may complete the process”; and Fig. 5, and [0066], “As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable”. Note that the scene regions may contain high density regions, such as water, objects, glasses, etc., and the regions with objects have density higher than the empty/air regions); and
determine the first constraint condition based on the first restricted space (See Ackerson: Figs. 1A-E, and [0050], “As described in various embodiments herein, one aim of the present invention is to provide systems and methods for performing scene reconstruction, and particularly performing generalized scene reconstruction (GSR). In some embodiments, the result of GSR processes or systems may result in a reconstruction of a light field, a matter field (including a relightable matter field), characterization of camera poses, or any combination of the foregoing. The result of GSR processes may result in a model representing a scene based upon the reconstructed light field or matter field (including the relightable matter field) individually and separately, or of the two together, as may be desirable under the circumstances. As used herein, a scene may refer to the entire scope of the light and/or matter field represented in an image, any portion thereof, or any media therein. Although the terms subscene, portion of a scene, region of interest, object of interest, and other similar terminology may be used to refer to a portion of a larger scene, each of the foregoing is itself a scene”; and [0272], “The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint”. Note that the high-density regions, such as objects of interest, regions of interest, etc. define the constraint for viewpoint selections, sampling and or rendering (to avoid empty space), and this is mapped to the viewpoint determinations).
Regarding claim 12, Ackerson and Metzler teach all the features with respect to claim 11 as outlined above. Further, Ackerson teaches that the device according to claim 11, wherein the one or more processors are further configured to:
determine structural information in the target scene space (See Ackerson: Fig. 21, and [0116], “In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures”);
determine a second restricted space in the target scene space based on the structural information (See Ackerson: Fig. 21, and [0049, “FIG. 21 is an illustration of a combination of a reconstruction performed with the methods described herein and a reconstruction created with another method”; and [0116], “In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures”. Note that 2102 is mapped to the second restriction space); and
determine the first constraint condition based on the first restricted space and the second restricted space (See Ackerson: Fig. 21, and [0116], “In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures”. Note that 2101 is mapped to the first restriction space),
wherein the second restricted space comprises a structured information area in the target scene space and an unstructured information area within a specific distance from the structured information area (See Ackerson: Fig. 21, and [0116], “In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures”. Note that 2102 is a bone with complex shape structure, and 2101 is almost a homogeneous region mapped to “unstructured information area”).
Regarding claim 13, Ackerson and Metzler teach all the features with respect to claim 10 as outlined above. Further, Ackerson teaches that the device according to claim 10, wherein the one or more processors are further configured to:
obtain a density value variation range from the density information (See Ackerson: Fig. 5, and [0066], “As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable”; and [0194], “Certain embodiments of the invention may use ML to reconstruct the light field in a scene, including, in some circumstances, constructing a physics model of the interactions of light and a matter field in the scene. In such embodiments, the system may decouple components that may contribute to the light sensed by camera pixels or another imaging device. This data may be used to determine the characteristics of the matter and objects in the scene, including non-Lambertian surfaces (e.g., human skin, cloth, mirrors, glass, and water). In some embodiments, certain surface information may be represented in a Bidirectional Light Interaction Function (BLIF) for one or more sensed locations on an object, and optionally all sensed locations on an object. The sensed locations may include locations captured by individual pixels of a camera or imaging device. The present invention may use the BLIF, and modeling based on BLIFs, to extend concepts such as a Bidirectional Reflectance Distribution Function (BDRF) and/or cosine lobe reflectance models to develop a greater level of sophistication by include light/matter interactions involving color, material, roughness, polarization, and so on”. Note that air, water, and solid objects like glass are inherently of different density: in Physics, the density of water is defined as 1, vacuum density is zero (or almost zero, air density is less than 1 (depending on the temperature and pressure), while most solid objects have density more than 1 (except ice); to identifying air, water, and glass by their density has inherently (by Physics) a density range); and
determine the first constraint condition based on the density value variation range (See Ackerson: Fig. 16, and [0169], “The residual layer 1604 may be applied to the predicted solution. The residual layer is optionally designed to ensure that the predicted solution satisfies the governing physics equations of the system. The residual layer may take partial derivatives of the predicted solution from the fully connected neural network with respect to the input coordinates and time, and enforce the physics equations governing the predicted solution 1603. The output of the residual layer 1605 may then be combined with a loss function that may include one or both of data constraints (such as known boundary conditions or initial conditions) and physics constraints (such as conservation laws or other governing equations). The loss function may be used to train the neural network to minimize the difference between the predicted solution and the observed data while still satisfying the underlying physics”).
Regarding claim 14, Ackerson and Metzler teach all the features with respect to claim 10 as outlined above. Further, Ackerson teaches that the device according to claim 10, wherein the one or more processors are further configured to:
determine a target point from a target space (See Ackerson: Figs. 1A-E, and [0143], “As a third example, the embodiments described herein may be used to reconstruct a scene including water, objects submerged or partially submerged in water, one or more water drops entering a body of water, such as a swimming pool, or objects submerged in water or another liquid. In one such example, multiple water droplets and a nearby body of water may be reconstructed. In certain embodiments, the droplets may be modelled moving to and entering the water body according to the laws of physics or other characteristics that may be provided to or known by the system. In some embodiments, the droplets may be represented volumetrically, which provides a basis for the system to calculate the mass properties of each drop using known mass properties of water. The system then may, based in whole, in part, or otherwise, on the mass and/or center-of-mass of a drop, model the trajectory of each such drop to the water. In some embodiments, the system may optionally include an advanced modeling system, which may support deformations of one or more of the drops or of the swimming pool”),
wherein the target space is a space in the target scene space excluding a first restricted space (See Ackerson: Figs. 1A-E, and [0059], “A scene may contain one or more voxels, each of which may be the same size and shape, or may be selected from a range of sizes and/or shapes as determined by the user or the system. A voxel may contain a mediel, or media element, that may represent all or a portion of media sampled in the voxel. Media is a volumetric region that includes some or no matter in which light flows. Media can be homogeneous or heterogeneous. Examples of homogeneous media include: empty space, air and water. Examples of heterogeneous media include volumetric regions including the surface of a mirror (part air and part slivered glass), the surface of a pane of glass (part air and part transmissive glass) and the branch of a pine tree (part air and part organic material). Light flows in media by phenomena including absorption, reflection, transmission and scattering. Examples of media that is partially transmissive includes the branch of a pine tree and a pane of glass”. Note that when the regions of objects with high density are mapped to the “restricted” region, the water drops can be mapped to the “target point” in the scene, and is it excluded from the restricted regions), and
a density value of a point in the first restricted space is greater than a first density threshold (See Ackerson: Fig. 5, and [0066], “As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable”; and [0194], “Certain embodiments of the invention may use ML to reconstruct the light field in a scene, including, in some circumstances, constructing a physics model of the interactions of light and a matter field in the scene. In such embodiments, the system may decouple components that may contribute to the light sensed by camera pixels or another imaging device. This data may be used to determine the characteristics of the matter and objects in the scene, including non-Lambertian surfaces (e.g., human skin, cloth, mirrors, glass, and water). In some embodiments, certain surface information may be represented in a Bidirectional Light Interaction Function (BLIF) for one or more sensed locations on an object, and optionally all sensed locations on an object. The sensed locations may include locations captured by individual pixels of a camera or imaging device. The present invention may use the BLIF, and modeling based on BLIFs, to extend concepts such as a Bidirectional Reflectance Distribution Function (BDRF) and/or cosine lobe reflectance models to develop a greater level of sophistication by include light/matter interactions involving color, material, roughness, polarization, and so on”. Note that air, water, and solid objects like glass are inherently of different density: the first density threshold is used to distinguish the air (lowest density) and the water (intermediate density), and the second density threshold is used to distinguish the water and the solid objects like glass (high density), then, the claimed limitation is obvious); and
determine the target viewpoint based on the target point (See Ackerson: Fig. 3A-G, and [0111], “With reference to FIG. 3D, the system may be configured to incorporate new image data. In some embodiments, the system may initialize one or more new camera poses 331, which may be accomplished, for example, as described with reference to FIGS. 3B and 3F. Some embodiments of the invention may then place one or more new radiels into the scene at voxels containing one or more new viewpoints 332”; and [0272], “The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint”).
Regarding claim 15, Ackerson and Metzler teach all the features with respect to claim 10 as outlined above. Further, Ackerson teaches that the device according to claim 10, wherein the one or more processors are further configured to:
determine a preselected point from the target scene space (See Ackerson: Figs. 1A-E, and [0143], “As a third example, the embodiments described herein may be used to reconstruct a scene including water, objects submerged or partially submerged in water, one or more water drops entering a body of water, such as a swimming pool, or objects submerged in water or another liquid. In one such example, multiple water droplets and a nearby body of water may be reconstructed. In certain embodiments, the droplets may be modelled moving to and entering the water body according to the laws of physics or other characteristics that may be provided to or known by the system. In some embodiments, the droplets may be represented volumetrically, which provides a basis for the system to calculate the mass properties of each drop using known mass properties of water. The system then may, based in whole, in part, or otherwise, on the mass and/or center-of-mass of a drop, model the trajectory of each such drop to the water. In some embodiments, the system may optionally include an advanced modeling system, which may support deformations of one or more of the drops or of the swimming pool”);
determine the preselected point as a target point when a density value of the preselected point satisfies a density threshold limitation (See Ackerson: Figs. 1A-E, and [0059], “A scene may contain one or more voxels, each of which may be the same size and shape, or may be selected from a range of sizes and/or shapes as determined by the user or the system. A voxel may contain a mediel, or media element, that may represent all or a portion of media sampled in the voxel. Media is a volumetric region that includes some or no matter in which light flows. Media can be homogeneous or heterogeneous. Examples of homogeneous media include: empty space, air and water. Examples of heterogeneous media include volumetric regions including the surface of a mirror (part air and part slivered glass), the surface of a pane of glass (part air and part transmissive glass) and the branch of a pine tree (part air and part organic material). Light flows in media by phenomena including absorption, reflection, transmission and scattering. Examples of media that is partially transmissive includes the branch of a pine tree and a pane of glass”. Note that when the regions of objects with high density are mapped to the “restricted” region, the water drops can be mapped to the “target point” in the scene, and is it excluded from the restricted regions); and
determine the target viewpoint based on the target point (See Ackerson: Fig. 3A-G, and [0111], “With reference to FIG. 3D, the system may be configured to incorporate new image data. In some embodiments, the system may initialize one or more new camera poses 331, which may be accomplished, for example, as described with reference to FIGS. 3B and 3F. Some embodiments of the invention may then place one or more new radiels into the scene at voxels containing one or more new viewpoints 332”; and [0272], “The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint”).
Regarding claim 17, Ackerson and Metzler teach all the features with respect to claim 1 as outlined above. Further, Ackerson and Metzler teach that a non-transitory computer-readable storage medium storing a computer program that, when being executed, causes at least one processor to perform (See Ackerson: Figs. 3A-G, and [0070], “With reference to FIG. 3A, some embodiments of the invention are operable to reconstruct a plenoptic field, including using incremental processes, where the plenoptic field may represent an entire scene, a portion of a scene, or a particular object or region of interest in a scene. In some embodiments, the system may first determine settings for the reconstruction of the scene 301. For example, the system may access or set a working resolution, initial size, target accuracy, relightability characteristics, or other characteristics. In some embodiments of the invention, the system may give an initial size to the scene. The size of the scene could be, for example, on the scale of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, user, or other factor that may be determined to be acceptable or advantageous. In some embodiments, including the exemplary embodiment depicted in FIG. 1B, the first camera 105 or set of image data may define the origin of the scene, and subsequent camera images, either captured by camera 105, a second camera or image sensing device 106, or otherwise, may be added to the scene and processed”):
obtaining density information by a target model corresponding to a target scene space (See Ackerson: Figs. 1A-E, and [0138], “In some embodiments, the higher-resolution information may be processed immediately and/or stored for later processing. The system may then construct a 3D model using all or a subset of the data available. In some embodiments, the highest resolution of the spatial model would roughly correspond to the projected sizes of the pixels”; and [0187], “In certain embodiments, the present invention may be used in conjunction with, in parallel with, be supplemented by, or otherwise implemented using in whole or in part artificial intelligence (AI), machine learning (ML), and neural networks, including neural radiance networks, such as Neural Radiance Fields, or NeRFs, volumetric scene methods such as PlenOctrees or Plenoxels, Deep Signed Distance Functions (SDF), and Neural Volumes. or in other technology. Such methods may be used for scene reconstruction, novel view synthesis (NVS) and other uses to model radiance, density, or other information as a continuous function in a 3D space such as a volumetric space using a multilayer perceptron (MLP) or voxels such as voxel arrays. For example, given a location in space with 3D coordinates (x, y, and z) and a viewing direction, the representation will return a color (red, green, and blue) and density at that location. Deep SDF systems may be configured to learn a signed distance function in 3D space whose zero level-set represents a 2D surface. Neural Volumes systems may be configured as neural graphics primitives that may be parameterized by fully connected neural networks. NeRF systems may be configured to model the color and density of a scene. Other embodiments operate with alternative input and return information. In certain embodiments, the returned density may be a differential opacity value which includes, in part or in whole, an estimate of the radiance and other information such as color that could be accumulated by a ray in the specified direction through the specified point”. Note that the scene reconstruction system builds a scene model representing the 3D space using the density information (volumetric) for points/voxels in the scene, the scene model is mapped to the target model, and the modeling 3D scene space is mapped to the target scene space), wherein
the target model is trained to output a viewpoint image corresponding to any viewpoint after inputting the any viewpoint in the target scene space (See Ackerson: Fig. 16, and [0169], “The residual layer 1604 may be applied to the predicted solution. The residual layer is optionally designed to ensure that the predicted solution satisfies the governing physics equations of the system. The residual layer may take partial derivatives of the predicted solution from the fully connected neural network with respect to the input coordinates and time, and enforce the physics equations governing the predicted solution 1603. The output of the residual layer 1605 may then be combined with a loss function that may include one or both of data constraints (such as known boundary conditions or initial conditions) and physics constraints (such as conservation laws or other governing equations). The loss function may be used to train the neural network to minimize the difference between the predicted solution and the observed data while still satisfying the underlying physics”); and
the density information is a rendering parameter required for image rendering by the target model, and the density information represents transparency of a point in the target scene space (See Ackerson: Figs. 1A-E, and [0187], “In certain embodiments, the present invention may be used in conjunction with, in parallel with, be supplemented by, or otherwise implemented using in whole or in part artificial intelligence (AI), machine learning (ML), and neural networks, including neural radiance networks, such as Neural Radiance Fields, or NeRFs, volumetric scene methods such as PlenOctrees or Plenoxels, Deep Signed Distance Functions (SDF), and Neural Volumes. or in other technology. Such methods may be used for scene reconstruction, novel view synthesis (NVS) and other uses to model radiance, density, or other information as a continuous function in a 3D space such as a volumetric space using a multilayer perceptron (MLP) or voxels such as voxel arrays. For example, given a location in space with 3D coordinates (x, y, and z) and a viewing direction, the representation will return a color (red, green, and blue) and density at that location. Deep SDF systems may be configured to learn a signed distance function in 3D space whose zero level-set represents a 2D surface. Neural Volumes systems may be configured as neural graphics primitives that may be parameterized by fully connected neural networks. NeRF systems may be configured to model the color and density of a scene. Other embodiments operate with alternative input and return information. In certain embodiments, the returned density may be a differential opacity value which includes, in part or in whole, an estimate of the radiance and other information such as color that could be accumulated by a ray in the specified direction through the specified point”; and [0189], “In some embodiments, the foregoing process may generate novel viewpoints with a high degree of realism after some level of “learning.” For example, this may be through AI, such as converging on estimated color values within a scene. In certain processes known in the art, the use of neural radiance networks or volumetric representations to generate novel-viewpoint images can require significant processing and/or time. Certain queries may require perhaps 500,000 to 1,000,000 multiplication and/or other operations for each point on the ray. Certain prior systems may require 30 seconds or more to generate a single 800-pixel by 800-pixel image on a powerful graphics processing unit (“GPU”), such as an Nvidia V100”. Note that density/opacity is a core rendering parameter in the volumetric model, it directly represents transparency/opacity of the points that is used for accurate image synthesis, and this teaching is mapped to the current cited limitation);
determining a first constraint condition based on the density information (See Ackerson: Figs. 3A-G, and [0018], “In some embodiments, the scene reconstruction may comprise processing the image data by postulating the orientation of the digital scene data. The processing of the image data may include (i) postulating that media exists in a voxel; (ii) postulating one or more of a surface normal, a light interaction property, an exitant radiance vector, an incident light field of the media, among other properties; (iii) calculating a cost for the existence of the media in the voxel based on the postulated one or more of a surface normal, a light interaction property (e.g., a refractive index, roughness, polarized diffuse coefficient, unpolarized diffuse coefficient, or extinction coefficient), an exitant radiance vector, and an incident light field of the media; (iv) comparing the cost to a cost threshold; and (iv) accepting media as existing at a voxel when the cost is below the cost threshold”; [0086], “Such operation may optionally use an actual value and/or change in value of a radiance, other radiel characteristic(s), and/or a confidence (consistency) of radiel characteristics to decide when to terminate that sequence of ops. For example, the system may also be configured to include a specific termination criteria, computation budget, or other threshold 342, including with regard to a light transport depth reflecting an iterative and/or recursive set of calculations. In such embodiments, the system may determine if the termination criteria, computation budget, or other threshold has been exceeded as discussed elsewhere herein. If the threshold has not been exceeded, the system may be configured to repeat the process, for example beginning at step 341. If the threshold has been exceeded, the system may complete the process”; and Figs. 14A-C, and [0108], “In some embodiments, the invention may perform the foregoing calculations for other mediels (or all mediels) within the region or scene of interest. The system may then compare the results for one or more of the mediels to the confidence threshold or other metric, for example based on predicted radiometric characteristic minus observed characteristics associated with exitant radiels of the mediel. For mediels where the confidence threshold or other metric is not achieved, the system may be configured to perform further processing related to such mediels. For example, FIG. 14B depicts a circumstances where the system has determined the bottom right mogel 1403 of mediel 1401 did not meet the appropriate threshold. In some embodiments, the system may subdivide such mediels not meeting the threshold or other metric into two or more child mediels, such as dividing a cube-shaped mediel into eight child cube-shaped mediels. In FIG. 14B, the system has subdivided mogel 1403 into four sub-mediels 1406, each of which having an associated content postulation 1407 and confidence 1408. In the embodiment depicted in FIG. 14B, the system has now postulated sub-mediel 1409 to contain a surface, as denoted at 1413, for example, an opaque dielectric surface that may be represented by a surfel, denoted by “S” with a confidence of 50. The remaining sub mediels remain postulated as containing air with varying degrees of confidence”. Note that the density information (such as a refractive index, roughness, polarized diffuse coefficient, unpolarized diffuse coefficient, or extinction coefficient, opacity, transparency, etc.) are used to determine if the mediels exitant in voxel by comparing the values to a threshold, which is mapped to the density related constraints);
determining a target viewpoint from the target scene space (See Ackerson: Fig. 3A-G, and [0111], “With reference to FIG. 3D, the system may be configured to incorporate new image data. In some embodiments, the system may initialize one or more new camera poses 331, which may be accomplished, for example, as described with reference to FIGS. 3B and 3F. Some embodiments of the invention may then place one or more new radiels into the scene at voxels containing one or more new viewpoints 332”; and [0272], “The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint”) based on the first constraint condition (See Metzler: Fig. 9, and [0344], “The acquisition position and/or orientation for generation of such state verification information is optionally determined analogue to a next-best-view method (NBV) known in the art. The goal of the next best view planning according to the invention is to recognize state relevant features with high certainty (wherefore it is not necessary to recognize or survey the door 108 as a whole). In the example, a relevant feature might be the lock of door 108 and/or edges of the door panel 123 or the adjacent wall 121 or presence or form of a shadow or light cone. Preferably, the computing unit of robot 100 provides a database with event relevant features of a plurality or multitude of objects of the property to be under surveillance”. Note that NBV is a well-known algorithm used to determine the optimal views); and
rendering the viewpoint image corresponding to the target viewpoint by the target model (See Ackerson: Fig. 21, and [0136], “In some circumstances, there may be multiple related pixels in a region of the image. If the number of related pixels is sufficient, certain embodiments of the invention may perform a statistical analysis of the texture. Such a statistical analysis may involve the application of a set of one or more filters to the region, and preferably would include clusters of the responses to the one or more filters assembled into a texture signature. In this example, a calculated texture signature may then be added as a property to the scene model and later used to insert synthetically generated textures into renderings to provide for realistic views”; and [0241], “In certain embodiments of this invention, the human-computer interface is capable of rendering and displaying a finished reconstruction. In some embodiments, displays may include analytic visualizations in addition to realistic views”).
Regarding claim 18, Ackerson and Metzler teach all the features with respect to claim 17 as outlined above. Further, Ackerson teaches that the non-transitory computer-readable storage medium according to claim 17, wherein the at least one processor is further configured to perform:
determining a first restriction space in the target scene space based on the density information (See Ackerson: Figs. 1A-E, and [0054], “Various exemplary embodiments of scene models are depicted in FIGS. 1C-1E. A scene model 110 may comprise a matter field 120 and a light field 130, either in a single model, as depicted in FIG. 1C, or separate, as depicted in FIG. 1D (matter field) and FIG. 1E (light field). A scene may have external illumination 112 flowing into the scene and providing a source of light in the scene. A scene may also be a unitary scene, wherein there is no light flowing into the scene 112. A scene may have a boundary 115, which may optionally be defined by the system during reconstruction of the scene, by a physical boundary in the scene, by a user or other input, by some combination of the foregoing, or otherwise. Information beyond the boundary of a scene may be considered a frontier 117, and may not be represented in the scene. However, in some embodiments, the boundary 115 may comprise a fenestral boundary 111 in whole or in part. A fenestral boundary 111 may be a portion of a scene boundary 115 through which incident light 112 may flow into the scene and exitant light 116 may flow out of the scene. In some embodiments, portions of the frontier 117 may be represented, at least in part, at the fenestral boundary 111. By way of example, a fenestral boundary 111 may be defined based on a physical feature in the scene (e.g., a window or skylight in a wall or ceiling through which light can enter the scene), based on a scene parallax (e.g., a boundary based on a distance or lack of resolution of image data, such as for an outdoor night scene looking at the sky where there is very long range in the field of view), some combination of the two, or some other factor. The scene may include one or more objects, including responsive objects 113 and emissive objects 114. Emissive objects 114 may emit light independent of light incident to the object, whereas responsive objects may interact with incident light without emitting light themselves”; and [0059], “A scene may contain one or more voxels, each of which may be the same size and shape, or may be selected from a range of sizes and/or shapes as determined by the user or the system. A voxel may contain a mediel, or media element, that may represent all or a portion of media sampled in the voxel. Media is a volumetric region that includes some or no matter in which light flows. Media can be homogeneous or heterogeneous. Examples of homogeneous media include: empty space, air and water. Examples of heterogeneous media include volumetric regions including the surface of a mirror (part air and part slivered glass), the surface of a pane of glass (part air and part transmissive glass) and the branch of a pine tree (part air and part organic material). Light flows in media by phenomena including absorption, reflection, transmission and scattering. Examples of media that is partially transmissive includes the branch of a pine tree and a pane of glass”; and Figs. 14A-C, and [0104], “In some embodiments, such as depicted in FIG. 14A, a scene of interest may contain both homogenous transmissive media and opaque media. In embodiments where the region of interest includes empty space (e.g., air or otherwise homogenous space not containing media substantially interacting with light), the system may specify within the data structure that the scene is comprised of mediels comprising empty space (e.g., air). Empty mediels or mediels comprising air and other homogenous elements may be referred to as mogels. In some embodiments, even if a mediel 1401 contains an opaque surface 1402, the system may initially stipulate that the mediel 1401 is comprised of one or more mogels 1403 comprising empty space or air (or air mogel), with such initialization would allow the system to let light flow through the mediel 1401 and mogels 1403 rather than being postulated to be blocked by interacting media, such as 1402. Some embodiments of the invention may specify a low confidence 1405 associated with each of the air mogels, which can facilitate the system later determining the presence of other media within each air mogel. In FIG. 14A, the postulated contents 1404 and confidence 1405 are depicted, with contents “A” 1404 representing an initial postulation of the mogel 1403 containing air and confidence “10” 1405 representing a hypothetical confidence value associated with that postulation”. Note that the target model (neural/volumetric) builds a scene representation (e.g., voxel, mediels, radiels, surfels,, etc.; and high density regions (the region associated with the object of interest may be the restricted regions) are identified as “restricted” or occupied regions),
wherein a density value of a point in the first restriction space is greater than a first density threshold (See Ackerson: Figs. 3A-G, and [0070], “With reference to FIG. 3A, some embodiments of the invention are operable to reconstruct a plenoptic field, including using incremental processes, where the plenoptic field may represent an entire scene, a portion of a scene, or a particular object or region of interest in a scene. In some embodiments, the system may first determine settings for the reconstruction of the scene 301. For example, the system may access or set a working resolution, initial size, target accuracy, relightability characteristics, or other characteristics. In some embodiments of the invention, the system may give an initial size to the scene. The size of the scene could be, for example, on the scale of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, user, or other factor that may be determined to be acceptable or advantageous. In some embodiments, including the exemplary embodiment depicted in FIG. 1B, the first camera 105 or set of image data may define the origin of the scene, and subsequent camera images, either captured by camera 105, a second camera or image sensing device 106, or otherwise, may be added to the scene and processed”; [0086], “Such operation may optionally use an actual value and/or change in value of a radiance, other radiel characteristic(s), and/or a confidence (consistency) of radiel characteristics to decide when to terminate that sequence of ops. For example, the system may also be configured to include a specific termination criteria, computation budget, or other threshold 342, including with regard to a light transport depth reflecting an iterative and/or recursive set of calculations. In such embodiments, the system may determine if the termination criteria, computation budget, or other threshold has been exceeded as discussed elsewhere herein. If the threshold has not been exceeded, the system may be configured to repeat the process, for example beginning at step 341. If the threshold has been exceeded, the system may complete the process”; and Fig. 5, and [0066], “As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable”. Note that the scene regions may contain high density regions, such as water, objects, glasses, etc., and the regions with objects have density higher than the empty/air regions); and
determining the first constraint condition based on the first restriction space (See Ackerson: Figs. 1A-E, and [0050], “As described in various embodiments herein, one aim of the present invention is to provide systems and methods for performing scene reconstruction, and particularly performing generalized scene reconstruction (GSR). In some embodiments, the result of GSR processes or systems may result in a reconstruction of a light field, a matter field (including a relightable matter field), characterization of camera poses, or any combination of the foregoing. The result of GSR processes may result in a model representing a scene based upon the reconstructed light field or matter field (including the relightable matter field) individually and separately, or of the two together, as may be desirable under the circumstances. As used herein, a scene may refer to the entire scope of the light and/or matter field represented in an image, any portion thereof, or any media therein. Although the terms subscene, portion of a scene, region of interest, object of interest, and other similar terminology may be used to refer to a portion of a larger scene, each of the foregoing is itself a scene”; and [0272], “The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint”. Note that the high-density regions, such as objects of interest, regions of interest, etc. define the constraint for viewpoint selections, sampling and or rendering (to avoid empty space), and this is mapped to the viewpoint determinations).
Regarding claim 19, Ackerson and Metzler teach all the features with respect to claim 18 as outlined above. Further, Ackerson teaches that the non-transitory computer-readable storage medium according to claim 18, wherein the at least one processor is further configured to perform:
determining a target point from a target space (See Ackerson: Figs. 1A-E, and [0143], “As a third example, the embodiments described herein may be used to reconstruct a scene including water, objects submerged or partially submerged in water, one or more water drops entering a body of water, such as a swimming pool, or objects submerged in water or another liquid. In one such example, multiple water droplets and a nearby body of water may be reconstructed. In certain embodiments, the droplets may be modelled moving to and entering the water body according to the laws of physics or other characteristics that may be provided to or known by the system. In some embodiments, the droplets may be represented volumetrically, which provides a basis for the system to calculate the mass properties of each drop using known mass properties of water. The system then may, based in whole, in part, or otherwise, on the mass and/or center-of-mass of a drop, model the trajectory of each such drop to the water. In some embodiments, the system may optionally include an advanced modeling system, which may support deformations of one or more of the drops or of the swimming pool”),
wherein the target space is a space in the target scene space excluding the first restriction space (See Ackerson: Figs. 1A-E, and [0059], “A scene may contain one or more voxels, each of which may be the same size and shape, or may be selected from a range of sizes and/or shapes as determined by the user or the system. A voxel may contain a mediel, or media element, that may represent all or a portion of media sampled in the voxel. Media is a volumetric region that includes some or no matter in which light flows. Media can be homogeneous or heterogeneous. Examples of homogeneous media include: empty space, air and water. Examples of heterogeneous media include volumetric regions including the surface of a mirror (part air and part slivered glass), the surface of a pane of glass (part air and part transmissive glass) and the branch of a pine tree (part air and part organic material). Light flows in media by phenomena including absorption, reflection, transmission and scattering. Examples of media that is partially transmissive includes the branch of a pine tree and a pane of glass”. Note that when the regions of objects with high density are mapped to the “restricted” region, the water drops can be mapped to the “target point” in the scene, and is it excluded from the restricted regions); and
determining the target viewpoint based on the target point (See Ackerson: Fig. 3A-G, and [0111], “With reference to FIG. 3D, the system may be configured to incorporate new image data. In some embodiments, the system may initialize one or more new camera poses 331, which may be accomplished, for example, as described with reference to FIGS. 3B and 3F. Some embodiments of the invention may then place one or more new radiels into the scene at voxels containing one or more new viewpoints 332”; and [0272], “The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint”).
Regarding claim 20, Ackerson and Metzler teach all the features with respect to claim 18 as outlined above. Further, Ackerson teaches that the non-transitory computer-readable storage medium according to claim 18, wherein the density value of the point in the first restriction space is less than a second density threshold, and the second density threshold is greater than the first density threshold (See Ackerson: Fig. 5, and [0066], “As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable”; and [0194], “Certain embodiments of the invention may use ML to reconstruct the light field in a scene, including, in some circumstances, constructing a physics model of the interactions of light and a matter field in the scene. In such embodiments, the system may decouple components that may contribute to the light sensed by camera pixels or another imaging device. This data may be used to determine the characteristics of the matter and objects in the scene, including non-Lambertian surfaces (e.g., human skin, cloth, mirrors, glass, and water). In some embodiments, certain surface information may be represented in a Bidirectional Light Interaction Function (BLIF) for one or more sensed locations on an object, and optionally all sensed locations on an object. The sensed locations may include locations captured by individual pixels of a camera or imaging device. The present invention may use the BLIF, and modeling based on BLIFs, to extend concepts such as a Bidirectional Reflectance Distribution Function (BDRF) and/or cosine lobe reflectance models to develop a greater level of sophistication by include light/matter interactions involving color, material, roughness, polarization, and so on”. Note that air, water, and solid objects like glass are inherently of different density: the first density threshold is used to distinguish the air (lowest density) and the water (intermediate density), and the second density threshold is used to distinguish the water and the solid objects like glass (high density), then, the claimed limitation is obvious).
Allowable Subject Matter
Claims 9 and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The best arts searched do not teach the limitations of “the method according to claim 1, wherein determining the target viewpoint from the target scene space based on the first constraint condition comprises: determining a target point from the target scene space based on the first constraint condition; determining a plurality of candidate viewpoints based on the target point; obtaining depth values for the plurality of candidate viewpoints, and the depth values are used to represent distances between the plurality of candidate viewpoints and an object in the target scene space; and determining the target viewpoint from the plurality of candidate viewpoints based on the depth values for the plurality of candidate viewpoints, wherein a depth value for the target viewpoint satisfies a depth value requirement.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights