DETAILED ACTION
This Office Action is in response to the Applicants' communication filed on August 26, 2026, which amends the independent claims 1, 10 and 17, amends the dependent claims 2. 9, 11-12, 15-16 and 18, cancels the dependent claim, 3, 14 and 19, adds a new independent claimx21, and presents arguments, is hereby acknowledged. Claims 1-2, 4-13, 15-18 and 20-21 are currently pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s arguments filed on August 26, 2026, have been fully considered.
Applicant argues that by this response, the independent claims 1, 10, and 17 are hereby amended to add a new limitation “determining a first restriction space in the target scene space based on the density information; and determining the first constraint condition based on the first restriction space; determining a target viewpoint from the target scene space based on the first constraint condition, comprising: determining a target point from a target space, wherein the target space is a space in the target scene space excluding the first restriction space; and determining the target viewpoint based on the target point;” in order to overcome the 35 U.S.C. §103 rejection.
Examiner replies that the amended claims with new limitation may overcome the cited portions of the prior arts. However, a newly found art, Aluru, etc. (US 20230388470 A1) teaches that determining a first restriction space in the target scene space based on the density information (See Aluru: Fig. 1, and [0025], “This disclosure describes example techniques to reduce the number of samples that need to be input for training the neural network, while ensuring that the trained model is robust and accurately generates image content. For instance, in some cases, many samples along the ray are empty (e.g., provide no color or opacity information) and do not contribute to the training, but are still sampled to ensure that samples that do contribute to the training are not skipped”; and [0057], “The following describes examples of dealing with free space sampling/computing. In volumetric rendering, to render an image, multiple points are sampled on the ray. While these samples span from the near and far point of the scene, one or more servers 102 or possibly a graphics processing unit (GPU) on personal computing device 106 (e.g., as part of volumetric rendering) may restrain the sampling to occur only within occupied voxels, i.e., voxels where the occupancy grid is one. To do so, one or more servers 102 or personal computing device 106 may scale the points to the minimum and maximum coordinates of the occupancy grid and interpolate values using nearest neighbor interpolation. In this way, in some cases, it may be possible to eliminate more than 95% of sampling that would have otherwise happened in a naive linear sampling. The elimination of 95% of sampling is provided as an example, and should not be considered as a requirement. The example techniques may be applicable to almost all inverse volumetric rendering pipelines”. Note that the free/empty regions (samples that provide no color or opacity information) is mapped to the first restriction space); and
determining the first constraint condition based on the first restriction space (See Aluru: Figs. 1-2, and [0046], “That is, as described above, for training to generate trained model 108, the amount of time needed to generate trained model 108 may be based on the number of samples on a ray that are used as input. In this disclosure, one or more servers 102 may use point clouds to filter out samples on a ray that do not contribute to the training, while selectively keeping samples on the ray that do contribute to the training for generation of trained model 108”; and [0070], “Neural network training unit 206 may be configured to train a neural network based on the determined samples on the ray to generate a trained model 216. The trained model 216 may be configured to generate image content of the scene from a viewpoint different than the plurality of viewpoints (e.g., from a viewpoint for which there is not one of images 210). For instance, processing circuitry 200 may receive a request to generate image content for the scene from a second viewpoint other than the plurality of viewpoints. For instance, personal computing device 106 may transmit the request. That is, there may be no one of images 210 that provides all of the image content from the second viewpoint of the scene.”);
determining a target viewpoint from the target scene space based on the first constraint condition (See Aluru: Figs. 1-2, and [0030], “In one or more examples, the trained model may be a volumetric scene function that can directly generate an appearance of the scene. For instance, as described above, the trained model may be used in implicit rendering in which an object is defined by mathematical functions and equations (e.g., continuous mathematical functions and equations). To render the image content of a viewpoint, other than the plurality of viewpoints for which images are available, the trained model may use volumetric rendering techniques. For instance, for training, the processing circuitry may use rays for determining which samples to input. Then, once trained, the processing circuitry may again generate rays, but from the viewpoint for which image content needs to be generated”), comprising:
determining a target point from a target space, wherein the target space is a space in the target scene space excluding the first restriction space(See Aluru: Fig. 9, and (51), “At a block 1002, the module 182 polls current user interaction data or stored user map interaction data, such as data on past user interactions with map data displayed on the interface 134, including data such as locations highlighted by the user, map points placed on a map display by the user, and geographic regions most displayed on a map display, for example”, and “For example, the module 182 may collect data on client device locations during workweek, Monday-Friday, and use that data for pre-fetching map data develop travel patterns of the client device”); and
determining the target viewpoint based on the target point (See Aluru: Figs. 1-2, and [0027], “The processing circuitry may determine samples on a ray from a viewpoint of the plurality of viewpoints based on the point cloud. For instance, the processing circuitry may use the point cloud as a way to determine occupancy of samples on a ray. The occupancy of samples on the ray may mean information that indicates whether a sample on the ray intersects the one or more objects in the scene. In some examples, the occupancy may indicate that the sample is close to the one or more objects in the scene. The processing circuitry may use as input, for training the neural network, samples on the ray that intersect the one or more objects in the scene or are relatively close to the one or more objects in the scene. In this manner, the processing circuitry may reduce the number of samples that are input for training”; and [0076], “Sample selection unit 204 may assign the determined voxels a value indicating whether the ray intersects the determined voxels. For instance, the value indicating whether the ray intersects the determined voxels may be considered as an occupancy grid that indicates whether the voxel is on or near a point in point cloud 212 and intersects the ray of the viewpoint.”). The remaining arguments of the applicant are mooted in view of the newly found art.
Examiner respectfully further replies that the Applicant's arguments have been fully considered and a new ground of rejections has been made. Accordingly, new grounds of rejection are set forth below. Since the new grounds of rejection are necessitated by Applicant's amendments to the claims, the present action is made final.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-8, 10-13, 15, and 17-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ackerson, etc. (US 20230281955 A1) in view of Metzler, etc. (US 20220005332 A1), further in view of Aluru, etc. (US 20230388470 A1).
Regarding claim 1, Ackerson teaches that an image generation method (See Ackerson: Figs. 3A-G, and [0070], ''With reference to FIG. 3A, some embodiments of the invention are operable to reconstruct a plenoptic field, including using incremental processes, where the plenoptic field may represent an entire scene, a portion of a scene, or a particular object or region of interest in a scene. In some embodiments, the system may first determine settings for the reconstruction of the scene 301. For example, the system may access or set a working resolution, initial size, target accuracy, relightability characteristics, or other characteristics. In some embodiments of the invention, the system may give an initial size to the scene. The size of the scene could be, for example, on the scale of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, user, or other factor that may be determined to be acceptable or advantageous. In some embodiments, including the exemplary embodiment depicted in FIG. 1B, the first camera 105 or set of image data may define the origin of the scene, and subsequent camera images, either captured by camera 105, a second camera or image sensing device 106, or otherwise, may be added to the scene and processed''), comprising:
obtaining density information by a target model corresponding to a target scene space (See Ackerson: Figs. lA-E, and [0138], ''In some embodiments, the higher-resolution information may be processed immediately and/or stored for later processing. The system may then construct a 3D model using all or a subset of the data available. In some embodiments, the highest resolution of the spatial model would roughly correspond to the projected sizes of the pixels''; and [0187], ''In certain embodiments, the present invention may be used in conjunction with, in parallel with, be supplemented by, or otherwise implemented using in whole or in part artificial intelligence (Al), machine learning (ML), and neural networks, including neural radiance networks, such as Neural Radiance Fields, or NeRFs, volumetric scene methods such as PlenOctrees or Plenoxels, Deep Signed Distance Functions (SDF), and Neural Volumes. or in other technology. Such methods may be used for scene reconstruction, novel view synthesis (NVS) and other uses to model radiance, density, or other information as a continuous function in a 3D space such as a volumetric space using a multilayer perceptron (MLP) or voxels such as voxel arrays. For example, given a location in space with 3D coordinates (x, y, and z) and a viewing direction, the representation will return a color (red, green, and blue) and density at that location. Deep SDF systems may be configured to learn a signed distance function in 3D space whose zero level-set represents a 2D surface. Neural Volumes systems may be configured as neural graphics primitives that may be parameterized by fully connected neural networks. NeRF systems may be configured to model the color and density of a scene. Other embodiments operate with alternative input and return information. In certain embodiments, the returned density may be a differential opacity value which includes, in part or in whole, an estimate of the radiance and other information such as color that could be accumulated by a ray in the specified direction through the specified point''. Note that the scene reconstruction system builds a scene model representing the 3D space using the density information (volumetric) for points/voxels in the scene, the scene model is mapped to the target model, and the modeling 3D scene space is mapped to the target scene space), wherein
the target model is trained to output a viewpoint image corresponding to any viewpoint after inputting the any viewpoint in the target scene space (See Ackerson: Fig. 16, and [0169], ''The residual layer 1604 may be applied to the predicted solution. The residual layer is optionally designed to ensure that the predicted solution satisfies the governing physics equations of the system. The residual layer may take partial derivatives of the predicted solution from the fully connected neural network with respect to the input coordinates and time, and enforce the physics equations governing the predicted solution 1603. The output of the residual layer 1605 may then be combined with a loss function that may include one or both of data constraints (such as known boundary conditions or initial conditions) and physics constraints (such as conservation laws or other governing equations). The loss function may be used to train the neural network to minimize the difference between the predicted solution and the observed data while still satisfying the underlying physics''); and
the density information is a rendering parameter required for image rendering by the target model, and the density information represents transparency of a point in the target scene space (See Ackerson: Figs. lA-E, and [0187], ''In certain embodiments, the present invention may be used in conjunction with, in parallel with, be supplemented by, or otherwise implemented using in whole or in part artificial intelligence (Al), machine learning (ML), and neural networks, including neural radiance networks, such as Neural Radiance Fields, or NeRFs, volumetric scene methods such as PlenOctrees or Plenoxels, Deep Signed Distance Functions (SDF), and Neural Volumes. or in other technology. Such methods may be used for scene reconstruction, novel view synthesis (NVS) and other uses to model radiance, density, or other information as a continuous function in a 3D space such as a volumetric space using a multilayer perceptron (MLP) or voxels such as voxel arrays. For example, given a location in space with 3D coordinates (x, y, and z) and a viewing direction, the representation will return a color (red, green, and blue) and density at that location. Deep SDF systems may be configured to learn a signed distance function in 3D space whose zero level-set represents a 2D surface. Neural Volumes systems may be configured as neural graphics primitives that may be parameterized by fully connected neural networks. NeRF systems may be configured to model the color and density of a scene. Other embodiments operate with alternative input and return information. In certain embodiments, the returned density may be a differential opacity value which includes, in part or in whole, an estimate of the radiance and other information such as color that could be accumulated by a ray in the specified direction through the specified point''; and [0189], ''In some embodiments, the foregoing process may generate novel viewpoints with a high degree of realism after some level of ''learning.'' For example, this may be through Al, such as converging on estimated color values within a scene. In certain processes known in the art, the use of neural radiance networks or volumetric representations to generate novel-viewpoint images can require significant processing and/or time. Certain queries may require perhaps 500,000 to 1,000,000 multiplication and/or other operations for each point on the ray. Certain prior systems may require 30 seconds or more to generate a single 800-pixel by 800-pixel image on a powerful graphics processing unit (''GPU''), such as an Nvidia VlOO''. Note that density/opacity is a core rendering parameter in the volumetric model, it directly represents transparency/opacity of the points that is used for accurate image synthesis, and this teaching is mapped to the current cited limitation);
determining a first constraint condition based on the density information (See Ackerson: Figs. 3A-G, and [0018], ''In some embodiments, the scene reconstruction may comprise processing the image data by postulating the orientation of the digital scene data. The processing of the image data may include (i) postulating that media exists in a voxel; (ii) postulating one or more of a surface normal, a light interaction property, an exitant radiance vector, an incident light field of the media, among other properties; (iii) calculating a cost for the existence of the media in the voxel based on the postulated one or more of a surface normal, a light interaction property (e.g., a refractive index, roughness, polarized diffuse coefficient, unpolarized diffuse coefficient, or extinction coefficient), an exitant radiance vector, and an incident light field of the media; (iv) comparing the cost to a cost threshold; and (iv) accepting media as existing at a voxel when the cost is below the cost threshold''; [0086], ''Such operation may optionally use an actual value and/or change in value of a radiance, other radiel characteristic(s), and/or a confidence (consistency) of radiel characteristics to decide when to terminate that sequence of ops. For example, the system may also be configured to include a specific termination criteria, computation budget, or other threshold 342, including with regard to a light transport depth reflecting an iterative and/or recursive set of calculations. In such embodiments, the system may determine if the termination criteria, computation budget, or other threshold has been exceeded as discussed elsewhere herein. If the threshold has not been exceeded, the system may be configured to repeat the process, for example beginning at step 341. If the threshold has been exceeded, the system may complete the process''; and Figs. 14A-C, and [0108], ''In some embodiments, the invention may perform the foregoing calculations for other mediels (or all mediels) within the region or scene of interest. The system may then compare the results for one or more of the mediels to the confidence threshold or other metric, for example based on predicted radiometric characteristic minus observed characteristics associated with exitant radiels of the mediel. For mediels where the confidence threshold or other metric is not achieved, the system may be configured to perform further processing related to such mediels. For example, FIG. 14B depicts a circumstances where the system has determined the bottom right mogel 1403 of mediel 1401 did not meet the appropriate threshold. In some embodiments, the system may subdivide such mediels not meeting the threshold or other metric into two or more child mediels, such as dividing a cube-shaped mediel into eight child cube-shaped mediels. In FIG. 14B, the system has subdivided mogel 1403 into four sub-mediels 1406, each of which having an associated content postulation 1407 and confidence 1408. In the embodiment depicted in FIG. 14B, the system has now postulated sub-mediel 1409 to contain a surface, as denoted at 1413, for example, an opaque dielectric surface that may be represented by a surfel, denoted by ''S'' with a confidence of 50. The remaining sub mediels remain postulated as containing air with varying degrees of confidence''. Note that the density information (such as a refractive index, roughness, polarized diffuse coefficient, unpolarized diffuse coefficient, or extinction coefficient, opacity, transparency, etc.) are used to determine if the mediels exitant in voxel by comparing the values to a threshold, which is mapped to the density related constraints), comprising:
determining a first restriction space in the target scene space based on the density information; and
determining the first constraint condition based on the first restriction space;
determining a target viewpoint from the target scene space (See Ackerson: Fig. 3A-G, and [0111], ''With reference to FIG. 3D, the system may be configured to incorporate new image data. In some embodiments, the system may initialize one or more new camera poses 331, which may be accomplished, for example, as described with reference to FIGS. 3B and 3F. Some embodiments of the invention may then place one or more new radiels into the scene at voxels containing one or more new viewpoints 332''; and [0272], ''The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint'') based on the first constraint condition, comprising:
determining a target point from a target space, wherein the target space is a space in the target scene space excluding the first restriction space; and
determining the target viewpoint based on the target point; and
rendering a viewpoint image corresponding to the target viewpoint by the target model (See Ackerson: Fig. 21, and [0136], ''In some circumstances, there may be multiple related pixels in a region of the image. If the number of related pixels is sufficient, certain embodiments of the invention may perform a statistical analysis of the texture. Such a statistical analysis may involve the application of a set of one or more filters to the region, and preferably would include clusters of the responses to the one or more filters assembled into a texture signature. In this example, a calculated texture signature may then be added as a property to the scene model and later used to insert synthetically generated textures into renderings to provide for realistic views''; and [0241], ''In certain embodiments of this invention, the human-computer interface is capable of rendering and displaying a finished reconstruction. In some embodiments, displays may include analytic visualizations in addition to realistic views'').
However, Ackerson fails to explicitly disclose that determining a first restriction space in the target scene space based on the density information; and determining the first constraint condition based on the first restriction space; (a target viewpoint is determined) based on the first constraint condition, comprising: determining a target point from a target space, wherein the target space is a space in the target scene space excluding the first restriction space; and determining the target viewpoint based on the target point.
However, Metzler teaches that (a target viewpoint is determined) based on the first constraint condition (See Metzler: Fig. 9, and [0344], ''The acquisition position and/or orientation for generation of such state verification information is optionally determined analogue to a next-best-view method (NBV) known in the art. The goal of the next best view planning according to the invention is to recognize state relevant features with high certainty (wherefore it is not necessary to recognize or survey the door 108 as a whole). In the example, a relevant feature might be the lock of door 108 and/or edges of the door panel 123 or the adjacent wall 121 or presence or form of a shadow or light cone. Preferably, the computing unit of robot 100 provides a database with event relevant features of a plurality or multitude of objects of the property to be under surveillance''. Note that NBV is a well-known algorithm used to determine the optimal views).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Ackerson to have (a target viewpoint is determined) based on the first constraint condition as taught by Metzler in order to improve the overall detection rate respectively reduce the false alarm rate, in particular in view of contradicting surveillance sensor results (See Metzler: Fig. 1, and [0176], ''Thereby, the invention is more than a simple independent side by side usage or a hierarchical surveillance-sensor approach of prior art, which can overcome many prior art drawbacks and can improve the overall detection rate respectively reduce the false alarm rate, in particular in view of at least partially contradicting surveillance sensor results."). Ackerson teaches a method and system that may perform a scene reconstruction using machine learning algorithm based on various scene information, such as density information, and generate a user selected viewpoint image for the user; while Metzler teaches a system and method that may use the next-best-view algorithm to determine the best view from plurality views for the user in the surveillance system. Therefore, it is obvious to one of ordinary skill in the art to modify Ackerson by Metzler to use the next-best-view algorithm to determine the best view from multiple views for the user. The motivation to modify Ackerson by Metzler is ''Use of known technique to improve similar devices (methods, or products) in the same way''.
However, Ackerson, modified by Metzler, fails to explicitly disclose that determining a first restriction space in the target scene space based on the density information; and determining the first constraint condition based on the first restriction space; determining a target point from a target space, wherein the target space is a space in the target scene space excluding the first restriction space; and determining the target viewpoint based on the target point.
However, Aluru teaches that determining a first restriction space in the target scene space based on the density information (See Aluru: Fig. 1, and [0025], “This disclosure describes example techniques to reduce the number of samples that need to be input for training the neural network, while ensuring that the trained model is robust and accurately generates image content. For instance, in some cases, many samples along the ray are empty (e.g., provide no color or opacity information) and do not contribute to the training, but are still sampled to ensure that samples that do contribute to the training are not skipped”; and [0057], “The following describes examples of dealing with free space sampling/computing. In volumetric rendering, to render an image, multiple points are sampled on the ray. While these samples span from the near and far point of the scene, one or more servers 102 or possibly a graphics processing unit (GPU) on personal computing device 106 (e.g., as part of volumetric rendering) may restrain the sampling to occur only within occupied voxels, i.e., voxels where the occupancy grid is one. To do so, one or more servers 102 or personal computing device 106 may scale the points to the minimum and maximum coordinates of the occupancy grid and interpolate values using nearest neighbor interpolation. In this way, in some cases, it may be possible to eliminate more than 95% of sampling that would have otherwise happened in a naive linear sampling. The elimination of 95% of sampling is provided as an example, and should not be considered as a requirement. The example techniques may be applicable to almost all inverse volumetric rendering pipelines”. Note that the free/empty regions (samples that provide no color or opacity information) is mapped to the first restriction space); and
determining the first constraint condition based on the first restriction space (See Aluru: Figs. 1-2, and [0046], “That is, as described above, for training to generate trained model 108, the amount of time needed to generate trained model 108 may be based on the number of samples on a ray that are used as input. In this disclosure, one or more servers 102 may use point clouds to filter out samples on a ray that do not contribute to the training, while selectively keeping samples on the ray that do contribute to the training for generation of trained model 108”; and [0070], “Neural network training unit 206 may be configured to train a neural network based on the determined samples on the ray to generate a trained model 216. The trained model 216 may be configured to generate image content of the scene from a viewpoint different than the plurality of viewpoints (e.g., from a viewpoint for which there is not one of images 210). For instance, processing circuitry 200 may receive a request to generate image content for the scene from a second viewpoint other than the plurality of viewpoints. For instance, personal computing device 106 may transmit the request. That is, there may be no one of images 210 that provides all of the image content from the second viewpoint of the scene.”);
determining a target viewpoint from the target scene space based on the first constraint condition (See Aluru: Figs. 1-2, and [0030], “In one or more examples, the trained model may be a volumetric scene function that can directly generate an appearance of the scene. For instance, as described above, the trained model may be used in implicit rendering in which an object is defined by mathematical functions and equations (e.g., continuous mathematical functions and equations). To render the image content of a viewpoint, other than the plurality of viewpoints for which images are available, the trained model may use volumetric rendering techniques. For instance, for training, the processing circuitry may use rays for determining which samples to input. Then, once trained, the processing circuitry may again generate rays, but from the viewpoint for which image content needs to be generated”), comprising:
determining a target point from a target space, wherein the target space is a space in the target scene space excluding the first restriction space(See Aluru: Fig. 9, and (51), “At a block 1002, the module 182 polls current user interaction data or stored user map interaction data, such as data on past user interactions with map data displayed on the interface 134, including data such as locations highlighted by the user, map points placed on a map display by the user, and geographic regions most displayed on a map display, for example”, and “For example, the module 182 may collect data on client device locations during workweek, Monday-Friday, and use that data for pre-fetching map data develop travel patterns of the client device”); and
determining the target viewpoint based on the target point (See Aluru: Figs. 1-2, and [0027], “The processing circuitry may determine samples on a ray from a viewpoint of the plurality of viewpoints based on the point cloud. For instance, the processing circuitry may use the point cloud as a way to determine occupancy of samples on a ray. The occupancy of samples on the ray may mean information that indicates whether a sample on the ray intersects the one or more objects in the scene. In some examples, the occupancy may indicate that the sample is close to the one or more objects in the scene. The processing circuitry may use as input, for training the neural network, samples on the ray that intersect the one or more objects in the scene or are relatively close to the one or more objects in the scene. In this manner, the processing circuitry may reduce the number of samples that are input for training”; and [0076], “Sample selection unit 204 may assign the determined voxels a value indicating whether the ray intersects the determined voxels. For instance, the value indicating whether the ray intersects the determined voxels may be considered as an occupancy grid that indicates whether the voxel is on or near a point in point cloud 212 and intersects the ray of the viewpoint.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Ackerson to have determining a first restriction space in the target scene space based on the density information; and determining the first constraint condition based on the first restriction space; determining a target point from a target space, wherein the target space is a space in the target scene space excluding the first restriction space; and determining the target viewpoint based on the target point as taught by Aluru in order to reduce number of samples that are input to the neural network and shape-radiance ambiguity, which can speed up neural network training, (See Aluru: Figs. 1-2, and [0024], “As described above, the samples (e.g., pose information of samples) of a ray are input to the neural network for training. In some examples, the rate at which the processing circuitry can generate the trained model may be based on the number of samples of a ray that are input. There may be a desire to train the neural network quickly to generate the trained model so that the trained model can provide image content from different views. However, simply reducing the number of samples that are used for training the neural network to increase the speed of training (e.g., reduce training time) may result is sub-optimal trained models"). Ackerson teaches a method and system that may perform a scene reconstruction using machine learning algorithm based on various scene information, such as density information, and generate a user selected viewpoint image for the user; while Aluru teaches a system and method that may use the NeRF model for field reconstruction with the opacity/density fields used for importance sampling and filtering samples according to current geometry estimates (prefers surface-near/high-opacity points. Therefore, it is obvious to one of ordinary skill in the art to modify Ackerson by Aluru to filter out the empty points in the NeRF model training in order to reduce ambiguity and increase accuracy. The motivation to modify Ackerson by Aluru is ''Use of known technique to improve similar devices (methods, or products) in the same way''.
Regarding claim 2, Ackerson, Metzler and Aluru teach all the features with respect to claim 1 as outlined above. Further, Ackerson teaches that the method according to claim 1,
wherein a density value of a point in the first restriction space is greater than a first density threshold (See Ackerson: Figs. 3A-G, and [0070], ''With reference to FIG. 3A, some embodiments of the invention are operable to reconstruct a plenoptic field, including using incremental processes, where the plenoptic field may represent an entire scene, a portion of a scene, or a particular object or region of interest in a scene. In some embodiments, the system may first determine settings for the reconstruction of the scene 301. For example, the system may access or set a working resolution, initial size, target accuracy, relightability characteristics, or other characteristics. In some embodiments of the invention, the system may give an initial size to the scene. The size of the scene could be, for example, on the scale of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, user, or other factor that may be determined to be acceptable or advantageous. In some embodiments, including the exemplary embodiment depicted in FIG. 1B, the first camera 105 or set of image data may define the origin of the scene, and subsequent camera images, either captured by camera 105, a second camera or image sensing device 106, or otherwise, may be added to the scene and processed''; [0086], ''Such operation may optionally use an actual value and/or change in value of a radiance, other radiel characteristic(s), and/or a confidence (consistency) of radiel characteristics to decide when to terminate that sequence of ops. For example, the system may also be configured to include a specific termination criteria, computation budget, or other threshold 342, including with regard to a light transport depth reflecting an iterative and/or recursive set of calculations. In such embodiments, the system may determine if the termination criteria, computation budget, or other threshold has been exceeded as discussed elsewhere herein. If the threshold has not been exceeded, the system may be configured to repeat the process, for example beginning at step 341. If the threshold has been exceeded, the system may complete the process''; and Fig. 5, and [0066], ''As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable''. Note that the scene regions may contain high density regions, such as water, objects, glasses, etc., and the regions with objects have density higher than the empty/air regions).
Regarding claim 4, Ackerson, Metzler and Aluru teach all the features with respect to claim 2 as outlined above. Further, Ackerson teaches that the method according to claim 2, wherein the density value of the point in the first restriction space is less than a second density threshold, and the second density threshold is greater than the first density threshold (See Ackerson: Fig. 5, and [0066], ''As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable''; and [0194], ''Certain embodiments of the invention may use ML to reconstruct the light field in a scene, including, in some circumstances, constructing a physics model of the interactions of light and a matter field in the scene. In such embodiments, the system may decouple components that may contribute to the light sensed by camera pixels or another imaging device. This data may be used to determine the characteristics of the matter and objects in the scene, including non-Lambertian surfaces (e.g., human skin, cloth, mirrors, glass, and water). In some embodiments, certain surface information may be represented in a Bidirectional Light Interaction Function (BLIF) for one or more sensed locations on an object, and optionally all sensed locations on an object. The sensed locations may include locations captured by individual pixels of a camera or imaging device. The present invention may use the BLIF, and modeling based on BLIFs, to extend concepts such as a Bidirectional Reflectance Distribution Function (BDRF) and/or cosine lobe reflectance models to develop a greater level of sophistication by include light/matter interactions involving color, material, roughness, polarization, and so on''. Note that air, water, and solid objects like glass are inherently of different density: the first density threshold is used to distinguish the air (lowest density) and the water (intermediate density), and the second density threshold is used to distinguish the water and the solid objects like glass (high density), then, the claimed limitation is obvious).
Regarding claim 5, Ackerson, Metzler and Aluru teach all the features with respect to claim 2 as outlined above. Further, Ackerson teaches that the method according to claim 2, wherein determining the first constraint condition based on the first restriction space comprises:
determining structural information in the target scene space (See Ackerson: Fig. 21, and [0116], ''In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures'');
determining a second restriction space in the target scene space based on the structural information (See Ackerson: Fig. 21, and [0049, ''FIG. 21 is an illustration of a combination of a reconstruction performed with the methods described herein and a reconstruction created with another method''; and [0116], ''In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internaI structures lacks BLIF information, BLIF information couId be automatically generated using a method such as machine learning based on the BLIFs of the external structures''. Note that 2102 is mapped to the second restriction space); and
determining the first constraint condition based on the first restriction space and the second restriction space (See Ackerson: Fig. 21, and [0116], ''In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures''. Note that 2101 is mapped to the first restriction space).
Regarding claim 6, Ackerson, Metzler and Aluru teach all the features with respect to claim 5 as outlined above. Further, Ackerson teaches that the method according to claim 5, wherein the second restriction space comprises a structured information area in the target scene space and an unstructured information area within a specific distance from the structured information area (See Ackerson: Fig. 21, and [0116], ''In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures''. Note that 2102 is a bone with complex shape structure, and 2101 is almost a homogeneous region mapped to ''unstructured information area'').
Regarding claim 7, Ackerson, Metzler and Aluru teach all the features with respect to claim 1 as outlined above. Further, Ackerson teaches that the method according to claim 1, wherein the first constraint condition is that a density value of a point satisfies a density threshold limitation (See Ackerson: Figs. 14A-C, and [0103], ''Some embodiments of the invention may then use a confidence threshold or other metric to guide processing, and calculate an associated confidence or other metric associated with each mediel or other volumetric element within the scene. If a confidence threshold is used, the system may examine one or more mediels where the confidence is below the confidence threshold. In some embodiments, if the confidence is below the threshold, the system may then compare the characteristics of the mediel with various known light interaction characteristics, such as a bidirectional light interaction function (or BLIF) associated with different types of media. For example, in the example depicted in FIG. 14A, if the confidence threshold is 75, the system may be configured to perform further calculations on each of the four depicted mogels 1403 because the associated confidence is below 75. Some embodiments may use a waterfall, or sequential, order of comparison based upon what the system has calculated to be the most likely candidate characteristics for the particular mediel (e.g., most likely candidate BLIF). For example, for a particular mediel, the system may first test the mediel for containing air, then a general dielectric media, then a general metallic media, and so on''); and
determining the target viewpoint from the target scene space based on the first constraint condition (See Ackerson: Figs. 3A-G, and [0080], ''The system may calculate or refine information regarding one or more poses associated with image data 322, as described with reference to FIG. 3F and elsewhere. In some embodiments, the system may determine if one or more camera or image data viewpoints containing voxel's light field has changed 322, which may optionally be determined based on some threshold of significance which could be preset or calculated by the system. This determination may be based, in part, on the system postulating or having other information indicating a camera image or set of image data exists at a voxel 201 in the data structure, as depicted in FIG. 2. In such embodiments, for each postulated position, the system may postulate an orientation in a coarse orientation space'') comprises:
determining a preselected point from the target scene space (See Ackerson: Figs. lA-E, and [0143], ''As a third example, the embodiments described herein may be used to reconstruct a scene including water, objects submerged or partially submerged in water, one or more water drops entering a body of water, such as a swimming pool, or objects submerged in water or another liquid. In one such example, multiple water droplets and a nearby body of water may be reconstructed. In certain embodiments, the droplets may be modelled moving to and entering the water body according to the laws of physics or other characteristics that may be provided to or known by the system. In some embodiments, the droplets may be represented volumetrically, which provides a basis for the system to calculate the mass properties of each drop using known mass properties of water. The system then may, based in whole, in part, or otherwise, on the mass and/or center-of-mass of a drop, model the trajectory of each such drop to the water. In some embodiments, the system may optionally include an advanced modeling system, which may support deformations of one or more of the drops or of the swimming pool'');
determining the preselected point as a target point when a density value of the preselected point satisfies the density threshold limitation (See Ackerson: Figs. 1A-E, and [0144], ''In some embodiments, the movement of a droplet may be modeled at discrete instances in time. At a point in time where a drop may first enter the larger segment representing the water body, an operation may be performed to determine the volume of water that is common between the swimming pool and the droplet. The system may then use the results of such an operation to compensate for a volume increase in the larger segment, which may optionally be accomplished using a morphological dilation operation. Upon such an operation, one or more volume elements on the larger segment surface (the swimming pool) that interface with movable material (a drop) may be extended incrementally to compensate for the displaced water volume and may be further modified to account for the dynamic reaction of the segment surface to the interaction with the movable material. The system may use such tools and similar tools to implement a more advanced displacement model. In some embodiments, the overall process may continue for additional water displacements until the droplet has become fully incorporated into the body of water''. Note that the density of the water drop is the same as the density of the water, and this is mapped to the current cited limitation, ''the preselected point'' (the water drop) satisfies the density threshold limitation); and
determining the target viewpoint based on the target point (See Ackerson: Fig. 3A-G, and [0111], ''With reference to FIG. 3D, the system may be configured to incorporate new image data. In some embodiments, the system may initialize one or more new camera poses 331, which may be accomplished, for example, as described with reference to FIGS. 3B and 3F. Some embodiments of the invention may then place one or more new radiels into the scene at voxels containing one or more new viewpoints 332''; and [0272], ''The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint'').
Regarding claim 8, Ackerson, Metzler and Aluru teach all the features with respect to claim 7 as outlined above. Further, Ackerson teaches that the method according to claim 7, wherein determining the first constraint condition based on the density information comprises:
obtaining a density value variation range from the density information (See Ackerson: Fig. 5, and [0066], ''As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable''; and [0194], ''Certain embodiments of the invention may use ML to reconstruct the light field in a scene, including, in some circumstances, constructing a physics model of the interactions of light and a matter field in the scene. In such embodiments, the system may decouple components that may contribute to the light sensed by camera pixels or another imaging device. This data may be used to determine the characteristics of the matter and objects in the scene, including non-Lambertian surfaces (e.g., human skin, cloth, mirrors, glass, and water). In some embodiments, certain surface information may be represented in a Bidirectional Light Interaction Function (BLIF) for one or more sensed locations on an object, and optionally all sensed locations on an object. The sensed locations may include locations captured by individual pixels of a camera or imaging device. The present invention may use the BLIF, and modeling based on BLIFs, to extend concepts such as a Bidirectional Reflectance Distribution Function (BDRF) and/or cosine lobe reflectance models to develop a greater level of sophistication by include light/matter interactions involving color, material, roughness, polarization, and so on''. Note that air, water, and solid objects like glass are inherently of different density: in Physics, the density of water is defined as 1, vacuum density is zero (or almost zero, air density is less than 1 (depending on the temperature and pressure), while most solid objects have density more than 1 (except ice); to identifying air, water, and glass by their density has inherently (by Physics) a density range); and
determining the first constraint condition based on the density value variation range (See Ackerson: Fig. 16, and [0169], ''The residual layer 1604 may be applied to the predicted solution. The residual layer is optionally designed to ensure that the predicted solution satisfies the governing physics equations of the system. The residual layer may take partial derivatives of the predicted solution from the fully connected neural network with respect to the input coordinates and time, and enforce the physics equations governing the predicted solution 1603. The output of the residual layer 1605 may then be combined with a loss function that may include one or both of data constraints (such as known boundary conditions or initial conditions) and physics constraints (such as conservation laws or other governing equations). The loss function may be used to train the neural network to minimize the difference between the predicted solution and the observed data while still satisfying the underlying physics'').
Regarding claim 10, Ackerson, Metzler and Aluru teach all the features with respect to claim 1 as outlined above. Further, Ackerson, Metzler and Aluru teach that an electronic device, comprising one or more processors and a memory containing a computer program that, when being executed, causes the one or more processors (See Ackerson: Figs. 3A-G, and [0070], ''With reference to FIG. 3A, some embodiments of the invention are operable to reconstruct a plenoptic field, including using incremental processes, where the plenoptic field may represent an entire scene, a portion of a scene, or a particular object or region of interest in a scene. In some embodiments, the system may first determine settings for the reconstruction of the scene 301. For example, the system may access or set a working resolution, initial size, target accuracy, relightability characteristics, or other characteristics. In some embodiments of the invention, the system may give an initial size to the scene. The size of the scene could be, for example, on the scale of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, user, or other factor that may be determined to be acceptable or advantageous. In some embodiments, including the exemplary embodiment depicted in FIG. 1B, the first camera 105 or set of image data may define the origin of the scene, and subsequent camera images, either captured by camera 105, a second camera or image sensing device 106, or otherwise, may be added to the scene and processed'') to:
obtain density information by a target model corresponding to a target scene space (See Ackerson: Figs. lA-E, and [0138], ''In some embodiments, the higher-resolution information may be processed immediately and/or stored for later processing. The system may then construct a 3D model using all or a subset of the data available. In some embodiments, the highest resolution of the spatial model would roughly correspond to the projected sizes of the pixels''; and [0187], ''In certain embodiments, the present invention may be used in conjunction with, in parallel with, be supplemented by, or otherwise implemented using in whole or in part artificial intelligence (Al), machine learning (ML), and neural networks, including neural radiance networks, such as Neural Radiance Fields, or NeRFs, volumetric scene methods such as PlenOctrees or Plenoxels, Deep Signed Distance Functions (SDF), and Neural Volumes. or in other technology. Such methods may be used for scene reconstruction, novel view synthesis (NVS) and other uses to model radiance, density, or other information as a continuous function in a 3D space such as a volumetric space using a multilayer perceptron (MLP) or voxels such as voxel arrays. For example, given a location in space with 3D coordinates (x, y, and z) and a viewing direction, the representation will return a color (red, green, and blue) and density at that location. Deep SDF systems may be configured to learn a signed distance function in 3D space whose zero level-set represents a 2D surface. Neural Volumes systems may be configured as neural graphics primitives that may be parameterized by fully connected neural networks. NeRF systems may be configured to model the color and density of a scene. Other embodiments operate with alternative input and return information. In certain embodiments, the returned density may be a differential opacity value which includes, in part or in whole, an estimate of the radianee and other information such as color that couId be accumulated by a ray in the specified direction through the specified point''. Note that the scene reconstruction system builds a scene model representing the 3D space using the density information (volumetric) for points/voxels in the scene, the scene model is mapped to the target model, and the modeling 3D scene space is mapped to the target scene space),
wherein the target model is trained to output a viewpoint image corresponding to any viewpoint after inputting the any viewpoint in the target scene space (See Ackerson: Fig. 16, and [0169], ''The residual layer 1604 may be applied to the predicted solution. The residual layer is optionally designed to ensure that the predicted solution satisfies the governing physics equations of the system. The residual layer may take partial derivatives of the predicted solution from the fully connected neural network with respect to the input coordinates and time, and enforce the physics equations governing the predicted solution 1603. The output of the residual layer 1605 may then be combined with a loss function that may include one or both of data constraints (such as known boundary conditions or initial conditions) and physics constraints (such as conservation laws or other governing equations). The loss function may be used to train the neural network to minimize the difference between the predicted solution and the observed data while still satisfying the underlying physics''), and
the density information is a rendering parameter required for image rendering by the target model, and the density information represents transparency of a point in the target scene space (See Ackerson: Figs. lA-E, and [0187], ''In certain embodiments, the present invention may be used in conjunction with, in parallel with, be supplemented by, or otherwise implemented using in whole or in part artificial intelligence (Al), machine learning (ML), and neural networks, including neural radiance networks, such as Neural Radiance Fields, or NeRFs, volumetric scene methods such as PlenOctrees or Plenoxels, Deep Signed Distance Functions (SDF), and Neural Volumes. or in other technology. Such methods may be used for scene reconstruction, novel view synthesis (NVS) and other uses to model radiance, density, or other information as a continuous function in a 3D space such as a volumetric space using a multilayer perceptron (MLP) or voxels such as voxel arrays. For example, given a location in space with 3D coordinates (x, y, and z) and a viewing direction, the representation will return a color (red, green, and blue) and density at that location. Deep SDF systems may be configured to learn a signed distance function in 3D space whose zero level-set represents a 2D surface. Neural Volumes systems may be configured as neural graphics primitives that may be parameterized by fully connected neural networks. NeRF systems may be configured to model the color and density of a scene. Other embodiments operate with alternative input and return information. In certain embodiments, the returned density may be a differential opacity value which includes, in part or in whole, an estimate of the radiance and other information such as color that could be accumulated by a ray in the specified direction through the specified point''; and [0189], ''In some embodiments, the foregoing process may generate novel viewpoints with a high degree of realism after some level of ''learning.'' For example, this may be through Al, such as converging on estimated color values within a scene. In certain processes known in the art, the use of neural radiance networks or volumetric representations to generate novel-viewpoint images can require significant processing and/or time. Certain queries may require perhaps 500,000 to 1,000,000 multiplication and/or other operations for each point on the ray. Certain prior systems may require 30 seconds or more to generate a single 800-pixel by 800-pixel image on a powerful graphics processing unit (''GPU''), such as an Nvidia VlOO''. Note that density/opacity is a core rendering parameter in the volumetric model, it directly represents transparency/opacity of the points that is used for accurate image synthesis, and this teaching is mapped to the current cited limitation);
determine a first constraint condition based on the density information (See Ackerson: Figs. 3A-G, and [0018], ''In some embodiments, the scene reconstruction may comprise processing the image data by postulating the orientation of the digital scene data. The processing of the image data may include (i) postulating that media exists in a voxel; (ii) postulating one or more of a surface normal, a light interaction property, an exitant radiance vector, an incident light field of the media, among other properties; (iii) calculating a cost for the existence of the media in the voxel based on the postulated one or more of a surface normal, a light interaction property (e.g., a refractive index, roughness, polarized diffuse coefficient, unpolarized diffuse coefficient, or extinction coefficient), an exitant radiance vector, and an incident light field of the media; (iv) comparing the cost to a cost threshold; and (iv) accepting media as existing at a voxel when the cost is below the cost threshold''; [0086], ''Such operation may optionally use an actual value and/or change in value of a radiance, other radiel characteristic(s), and/or a confidence (consistency) of radiel characteristics to decide when to terminate that sequence of ops. For example, the system may also be configured to include a specific termination criteria, computation budget, or other threshold 342, including with regard to a light transport depth reflecting an iterative and/or recursive set of calculations. In such embodiments, the system may determine if the termination criteria, computation budget, or other threshold has been exceeded as discussed elsewhere herein. If the threshold has not been exceeded, the system may be configured to repeat the process, for example beginning at step 341. If the threshold has been exceeded, the system may complete the process''; and Figs. 14A-C, and [0108], ''In some embodiments, the invention may perform the foregoing calculations for other mediels (or all mediels) within the region or scene of interest. The system may then compare the results for one or more of the mediels to the confidence threshold or other metric, for example based on predicted radiometric characteristic minus observed characteristics associated with exitant radiels of the mediel. For mediels where the confidence threshold or other metric is not achieved, the system may be configured to perform further processing related to such mediels. For example, FIG. 14B depicts a circumstances where the system has determined the bottom right mogel 1403 of mediel 1401 did not meet the appropriate threshold. In some embodiments, the system may subdivide such mediels not meeting the threshold or other metric into two or more child mediels, such as dividing a cube-shaped mediel into eight child cube-shaped mediels. In FIG. 14B, the system has subdivided mogel 1403 into four sub-mediels 1406, each of which having an associated content postulation 1407 and confidence 1408. In the embodiment depicted in FIG. 14B, the system has now postulated sub-mediel 1409 to contain a surface, as denoted at 1413, for example, an opaque dielectric surface that may be represented by a surfel, denoted by ''S'' with a confidence of 50. The remaining sub mediels remain postulated as containing air with varying degrees of confidence''. Note that the density information (such as a refractive index, roughness, polarized diffuse coefficient, unpolarized diffuse coefficient, or extinction coefficient, opacity, transparency, etc.) are used to determine if the mediels exitant in voxel by comparing the values to a threshold, which is mapped to the density related constraints), comprising:
that determining a first restriction space in the target scene space based on the density information (See Aluru: Fig. 1, and [0025], “This disclosure describes example techniques to reduce the number of samples that need to be input for training the neural network, while ensuring that the trained model is robust and accurately generates image content. For instance, in some cases, many samples along the ray are empty (e.g., provide no color or opacity information) and do not contribute to the training, but are still sampled to ensure that samples that do contribute to the training are not skipped”; and [0057], “The following describes examples of dealing with free space sampling/computing. In volumetric rendering, to render an image, multiple points are sampled on the ray. While these samples span from the near and far point of the scene, one or more servers 102 or possibly a graphics processing unit (GPU) on personal computing device 106 (e.g., as part of volumetric rendering) may restrain the sampling to occur only within occupied voxels, i.e., voxels where the occupancy grid is one. To do so, one or more servers 102 or personal computing device 106 may scale the points to the minimum and maximum coordinates of the occupancy grid and interpolate values using nearest neighbor interpolation. In this way, in some cases, it may be possible to eliminate more than 95% of sampling that would have otherwise happened in a naive linear sampling. The elimination of 95% of sampling is provided as an example, and should not be considered as a requirement. The example techniques may be applicable to almost all inverse volumetric rendering pipelines”. Note that the free/empty regions (samples that provide no color or opacity information) is mapped to the first restriction space); and
determining the first constraint condition based on the first restriction space (See Aluru: Figs. 1-2, and [0046], “That is, as described above, for training to generate trained model 108, the amount of time needed to generate trained model 108 may be based on the number of samples on a ray that are used as input. In this disclosure, one or more servers 102 may use point clouds to filter out samples on a ray that do not contribute to the training, while selectively keeping samples on the ray that do contribute to the training for generation of trained model 108”; and [0070], “Neural network training unit 206 may be configured to train a neural network based on the determined samples on the ray to generate a trained model 216. The trained model 216 may be configured to generate image content of the scene from a viewpoint different than the plurality of viewpoints (e.g., from a viewpoint for which there is not one of images 210). For instance, processing circuitry 200 may receive a request to generate image content for the scene from a second viewpoint other than the plurality of viewpoints. For instance, personal computing device 106 may transmit the request. That is, there may be no one of images 210 that provides all of the image content from the second viewpoint of the scene.”);
determine a target viewpoint from the target scene space (See Ackerson: Fig. 3A-G, and [0111], ''With reference to FIG. 3D, the system may be configured to incorporate new image data. In some embodiments, the system may initialize one or more new camera poses 331, which may be accomplished, for example, as described with reference to FIGS. 3B and 3F. Some embodiments of the invention may then place one or more new radiels into the scene at voxels containing one or more new viewpoints 332''; and [0272], ''The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint'') based on the first constraint condition (See Metzler: Fig. 9, and [0344], ''The acquisition position and/or orientation for generation of such state verification information is optionally determined analogue to a next-best-view method (NBV) known in the art. The goal of the next best view planning according to the invention is to recognize state relevant features with high certainty (wherefore it is not necessary to recognize or survey the door 108 as a whole). In the example, a relevant feature might be the lock of door 108 and/or edges of the door panel 123 or the adjacent wall 121 or presence or form of a shadow or light cone. Preferably, the computing unit of robot 100 provides a database with event relevant features of a plurality or multitude of objects of the property to be under surveillance''. Note that NBV is a well-known algorithm used to determine the optimal views). Comprising:
determining a target point from a target space, wherein the target space is a space in the target scene space excluding the first restriction space(See Aluru: Fig. 9, and (51), “At a block 1002, the module 182 polls current user interaction data or stored user map interaction data, such as data on past user interactions with map data displayed on the interface 134, including data such as locations highlighted by the user, map points placed on a map display by the user, and geographic regions most displayed on a map display, for example”, and “For example, the module 182 may collect data on client device locations during workweek, Monday-Friday, and use that data for pre-fetching map data develop travel patterns of the client device”); and
determining the target viewpoint based on the target point (See Aluru: Figs. 1-2, and [0027], “The processing circuitry may determine samples on a ray from a viewpoint of the plurality of viewpoints based on the point cloud. For instance, the processing circuitry may use the point cloud as a way to determine occupancy of samples on a ray. The occupancy of samples on the ray may mean information that indicates whether a sample on the ray intersects the one or more objects in the scene. In some examples, the occupancy may indicate that the sample is close to the one or more objects in the scene. The processing circuitry may use as input, for training the neural network, samples on the ray that intersect the one or more objects in the scene or are relatively close to the one or more objects in the scene. In this manner, the processing circuitry may reduce the number of samples that are input for training”; and [0076], “Sample selection unit 204 may assign the determined voxels a value indicating whether the ray intersects the determined voxels. For instance, the value indicating whether the ray intersects the determined voxels may be considered as an occupancy grid that indicates whether the voxel is on or near a point in point cloud 212 and intersects the ray of the viewpoint.”); and
render a viewpoint image corresponding to the target viewpoint by the target model (See Ackerson: Fig. 21, and [0136], ''In some circumstances, there may be multiple related pixels in a region of the image. If the number of related pixels is sufficient, certain embodiments of the invention may perform a statistical analysis of the texture. Such a statistical analysis may involve the application of a set of one or more filters to the region, and preferably would include clusters of the responses to the one or more filters assembled into a texture signature. In this example, a calculated texture signature may then be added as a property to the scene model and later used to insert synthetically generated textures into renderings to provide for realistic views''; and [0241], ''In certain embodiments of this invention, the human-computer interface is capable of rendering and displaying a finished reconstruction. In some embodiments, displays may include analytic visualizations in addition to realistic views'').
Regarding claim 11, Ackerson, Metzler and Aluru teach all the features with respect to claim 10 as outlined above. Further, Ackerson teaches that the device according to claim 10,
wherein a density value of a point in the first restricted space is greater than a first density threshold (See Ackerson: Figs. 3A-G, and [0070], ''With reference to FIG. 3A, some embodiments of the invention are operable to reconstruct a plenoptic field, including using incremental processes, where the plenoptic field may represent an entire scene, a portion of a scene, or a particular object or region of interest in a scene. In some embodiments, the system may first determine settings for the reconstruction of the scene 301. For example, the system may access or set a working resolution, initial size, target accuracy, relightability characteristics, or other characteristics. In some embodiments of the invention, the system may give an initial size to the scene. The size of the scene could be, for example, on the scale of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, user, or other factor that may be determined to be acceptable or advantageous. In some embodiments, including the exemplary embodiment depicted in FIG. 1B, the first camera 105 or set of image data may define the origin of the scene, and subsequent camera images, either captured by camera 105, a second camera or image sensing device 106, or otherwise, may be added to the scene and processed''; [0086], ''Such operation may optionally use an actual value and/or change in value of a radiance, other radiel characteristic(s), and/or a confidence (consistency) of radiel characteristics to decide when to terminate that sequence of ops. For example, the system may also be configured to include a specific termination criteria, computation budget, or other threshold 342, including with regard to a light transport depth reflecting an iterative and/or recursive set of calculations. In such embodiments, the system may determine if the termination criteria, computation budget, or other threshold has been exceeded as discussed elsewhere herein. If the threshold has not been exceeded, the system may be configured to repeat the process, for example beginning at step 341. If the threshold has been exceeded, the system may complete the process''; and Fig. 5, and [0066], ''As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable''. Note that the scene regions may contain high density regions, such as water, objects, glasses, etc., and the regions with objects have density higher than the empty/air regions); and
Regarding claim 12, Ackerson, Metzler and Aluru teach all the features with respect to claim 11 as outlined above. Further, Ackerson teaches that the device according to claim 11, wherein the one or more processors are further configured to:
determine structural information in the target scene space (See Ackerson: Fig. 21, and [0116], ''In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures'');
determine a second restricted space in the target scene space based on the structural information (See Ackerson: Fig. 21, and [0049, ''FIG. 21 is an illustration of a combination of a reconstruction performed with the methods described herein and a reconstruction created with another method''; and [0116], ''In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internaI structures lacks BLIF information, BLIF information couId be automatically generated using a method such as machine learning based on the BLIFs of the external structures''. Note that 2102 is mapped to the second restriction space); and
determine the first constraint condition based on the first restricted space and the second restricted space (See Ackerson: Fig. 21, and [0116], ''In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures''. Note that 2101 is mapped to the first restriction space),
wherein the second restricted space comprises a structured information area in the target scene space and an unstructured information area within a specific distance from the structured information area (See Ackerson: Fig. 21, and [0116], ''In addition, in some embodiments of this invention, reconstructions of the opaque external structures of an object or scene could be combined with reconstructions of the internal structures of the same object or scene (including internal reconstructions created with a different method, such as X-ray imaging or MRI scanning), such as shown in FIG. 21. Internal structures could be nested within external structures to form a more complete model of the object or scene. In some embodiments, if the method used to reconstruct the internal structures lacks BLIF information, BLIF information could be automatically generated using a method such as machine learning based on the BLIFs of the external structures''. Note that 2102 is a bone with complex shape structure, and 2101 is almost a homogeneous region mapped to ''unstructured information area'').
Regarding claim 13, Ackerson, Metzler and Aluru teach all the features with respect to claim 10 as outlined above. Further, Ackerson teaches that the device according to claim 10, wherein the one or more processors are further configured to:
obtain a density value variation range from the density information (See Ackerson: Fig. 5, and [0066], ''As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable''; and [0194], ''Certain embodiments of the invention may use ML to reconstruct the light field in a scene, including, in some circumstances, constructing a physics model of the interactions of light and a matter field in the scene. In such embodiments, the system may decouple components that may contribute to the light sensed by camera pixels or another imaging device. This data may be used to determine the characteristics of the matter and objects in the scene, including non-Lambertian surfaces (e.g., human skin, cloth, mirrors, glass, and water). In some embodiments, certain surface information may be represented in a Bidirectional Light Interaction Function (BLIF) for one or more sensed locations on an object, and optionally all sensed locations on an object. The sensed locations may include locations captured by individual pixels of a camera or imaging device. The present invention may use the BLIF, and modeling based on BLIFs, to extend concepts such as a Bidirectional Reflectance Distribution Function (BDRF) and/or cosine lobe reflectance models to develop a greater level of sophistication by include light/matter interactions involving color, material, roughness, polarization, and so on''. Note that air, water, and solid objects like glass are inherently of different density: in Physics, the density of water is defined as 1, vacuum density is zero (or almost zero, air density is less than 1 (depending on the temperature and pressure), while most solid objects have density more than 1 (except ice); to identifying air, water, and glass by their density has inherently (by Physics) a density range); and
determine the first constraint condition based on the density value variation range (See Ackerson: Fig. 16, and [0169], ''The residual layer 1604 may be applied to the predicted solution. The residual layer is optionally designed to ensure that the predicted solution satisfies the governing physics equations of the system. The residual layer may take partial derivatives of the predicted solution from the fully connected neural network with respect to the input coordinates and time, and enforce the physics equations governing the predicted solution 1603. The output of the residual layer 1605 may then be combined with a loss function that may include one or both of data constraints (such as known boundary conditions or initial conditions) and physics constraints (such as conservation laws or other governing equations). The loss function may be used to train the neural network to minimize the difference between the predicted solution and the observed data while still satisfying the underlying physics'').
Regarding claim 15, Ackerson, Metzler and Aluru teach all the features with respect to claim 10 as outlined above. Further, Ackerson teaches that the device according to claim 10, wherein the one or more processors are further configured to:
determine a preselected point from the target scene space (See Ackerson: Figs. lA-E, and [0143], ''As a third example, the embodiments described herein may be used to reconstruct a scene including water, objects submerged or partially submerged in water, one or more water drops entering a body of water, such as a swimming pool, or objects submerged in water or another liquid. In one such example, multiple water droplets and a nearby body of water may be reconstructed. In certain embodiments, the droplets may be modelled moving to and entering the water body according to the laws of physics or other characteristics that may be provided to or known by the system. In some embodiments, the droplets may be represented volumetrically, which provides a basis for the system to calculate the mass properties of each drop using known mass properties of water. The system then may, based in whole, in part, or otherwise, on the mass and/or center-of-mass of a drop, model the trajectory of each such drop to the water. In some embodiments, the system may optionally include an advanced modeling system, which may support deformations of one or more of the drops or of the swimming pool'');
determine the preselected point as a target point when a density value of the preselected point satisfies a density threshold limitation (See Ackerson: Figs. lA-E, and [0059], ''A scene may contain one or more voxels, each of which may be the same size and shape, or may be selected from a range of sizes and/or shapes as determined by the user or the system. A voxel may contain a mediel, or media element, that may represent all or a portion of media sampled in the voxel. Media is a volumetric region that includes some or no matter in which light flows. Media can be homogeneous or heterogeneous. Examples of homogeneous media include: empty space, air and water. Examples of heterogeneous media include volumetric regions including the surface of a mirror (part air and part slivered glass), the surface of a pane of glass (part air and part transmissive glass) and the branch of a pine tree (part air and part organic material). Light flows in media by phenomena including absorption, reflection, transmission and scattering. Examples of media that is partially transmissive includes the branch of a pine tree and a pane of glass''. Note that when the regions of objects with high density are mapped to the ''restricted'' region, the water drops can be mapped to the ''target point'' in the scene, and is it excluded from the restricted regions); and
determine the target viewpoint based on the target point (See Ackerson: Fig. 3A-G, and [0111], ''With reference to FIG. 3D, the system may be configured to incorporate new image data. In some embodiments, the system may initialize one or more new camera poses 331, which may be accomplished, for example, as described with reference to FIGS. 3B and 3F. Some embodiments of the invention may then place one or more new radiels into the scene at voxels containing one or more new viewpoints 332''; and [0272], ''The method of embodiment 22 wherein the reconstructing the radiometric characteristics comprises using the image data by selecting a viewpoint and calculating the radiometric characteristics associated with each volumetric element along one or more corridors extending from the viewpoint'').
Regarding claim 17, Ackerson, Metzler and Aluru teach all the features with respect to claim 1 as outlined above. Further, Ackerson, Metzler and Aluru teach that a non-transitory computer-readable storage medium storing a computer program that, when being executed, causes at least one processor to perform the method according to claim 1 (See Ackerson: Figs. 3A-G, and [0070], ''With reference to FIG. 3A, some embodiments of the invention are operable to reconstruct a plenoptic field, including using incremental processes, where the plenoptic field may represent an entire scene, a portion of a scene, or a particular object or region of interest in a scene. In some embodiments, the system may first determine settings for the reconstruction of the scene 301. For example, the system may access or set a working resolution, initial size, target accuracy, relightability characteristics, or other characteristics. In some embodiments of the invention, the system may give an initial size to the scene. The size of the scene could be, for example, on the scale of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, user, or other factor that may be determined to be acceptable or advantageous. In some embodiments, including the exemplary embodiment depicted in FIG. 1B, the first camera 105 or set of image data may define the origin of the scene, and subsequent camera images, either captured by camera 105, a second camera or image sensing device 106, or otherwise, may be added to the scene and processed''):
Regarding claim 18, Ackerson, Metzler and Aluru teach all the features with respect to claim 17 as outlined above. Further, Ackerson teaches that the non-transitory computer-readable storage medium according to claim 17,
wherein a density value of a point in the first restriction space is greater than a first density threshold (See Ackerson: Figs. 3A-G, and [0070], ''With reference to FIG. 3A, some embodiments of the invention are operable to reconstruct a plenoptic field, including using incremental processes, where the plenoptic field may represent an entire scene, a portion of a scene, or a particular object or region of interest in a scene. In some embodiments, the system may first determine settings for the reconstruction of the scene 301. For example, the system may access or set a working resolution, initial size, target accuracy, relightability characteristics, or other characteristics. In some embodiments of the invention, the system may give an initial size to the scene. The size of the scene could be, for example, on the scale of a human living space for an indoor scene, a different size for an outdoor scene, or another size defined by the system, user, or other factor that may be determined to be acceptable or advantageous. In some embodiments, including the exemplary embodiment depicted in FIG. 1B, the first camera 105 or set of image data may define the origin of the scene, and subsequent camera images, either captured by camera 105, a second camera or image sensing device 106, or otherwise, may be added to the scene and processed''; [0086], ''Such operation may optionally use an actual value and/or change in value of a radiance, other radiel characteristic(s), and/or a confidence (consistency) of radiel characteristics to decide when to terminate that sequence of ops. For example, the system may also be configured to include a specific termination criteria, computation budget, or other threshold 342, including with regard to a light transport depth reflecting an iterative and/or recursive set of calculations. In such embodiments, the system may determine if the termination criteria, computation budget, or other threshold has been exceeded as discussed elsewhere herein. If the threshold has not been exceeded, the system may be configured to repeat the process, for example beginning at step 341. If the threshold has been exceeded, the system may complete the process''; and Fig. 5, and [0066], ''As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable''. Note that the scene regions may contain high density regions, such as water, objects, glasses, etc., and the regions with objects have density higher than the empty/air regions).
Regarding claim 20, Ackerson, Metzler and Aluru teach all the features with respect to claim 18 as outlined above. Further, Ackerson teaches that the non-transitory computer-readable storage medium according to claim 18, wherein the density value of the point in the first restriction space is less than a second density threshold, and the second density threshold is greater than the first density threshold (See Ackerson: Fig. 5, and [0066], ''As depicted in FIG. 5, a surfel 503 may contain more than one type of matter. In the diagram, the surfel 503 contains both glass and air with one surface separating them; a mogel 504 contains only glass; and a mixel 505 represents a corner of the pane and thus contains multiple surfaces. Mediels, in general, may contain various forms of property information. For example, surfels and mogels may contain BLIF values or other property information that can be used for relighting. In some cases, mixels may contain information to make them relightable''; and [0194], ''Certain embodiments of the invention may use ML to reconstruct the light field in a scene, including, in some circumstances, constructing a physics model of the interactions of light and a matter field in the scene. In such embodiments, the system may decouple components that may contribute to the light sensed by camera pixels or another imaging device. This data may be used to determine the characteristics of the matter and objects in the scene, including non-Lambertian surfaces (e.g., human skin, cloth, mirrors, glass, and water). In some embodiments, certain surface information may be represented in a Bidirectional Light Interaction Function (BLIF) for one or more sensed locations on an object, and optionally all sensed locations on an object. The sensed locations may include locations captured by individual pixels of a camera or imaging device. The present invention may use the BLIF, and modeling based on BLIFs, to extend concepts such as a Bidirectional Reflectance Distribution Function (BDRF) and/or cosine lobe reflectance models to develop a greater level of sophistication by include light/matter interactions involving color, material, roughness, polarization, and so on''. Note that air, water, and solid objects like glass are inherently of different density: the first density threshold is used to distinguish the air (lowest density) and the water (intermediate density), and the second density threshold is used to distinguish the water and the solid objects like glass (high density), then, the claimed limitation is obvious).
Allowable Subject Matter
Claims 9 and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The best arts searched, Ackerson, etc. (US 20230281955 A1), Metzler, etc. (US 20220005332 A1), and Aluru, etc. (US 20230388470 A1), do not teach the limitations of ''the method according to claim 1, wherein determining the target viewpoint from the target scene space based on the first constraint condition comprises: determining a target point from the target scene space based on the first constraint condition; determining a plurality of candidate viewpoints based on the target point; obtaining depth values for the plurality of candidate viewpoints, and the depth values are used to represent distances between the plurality of candidate viewpoints and an object in the target scene space; and determining the target viewpoint from the plurality of candidate viewpoints based on the depth values for the plurality of candidate viewpoints, wherein a depth value for the target viewpoint satisfies a depth value requirement."
Claim 21 is allowed. The best arts searched, Ackerson, etc. (US 20230281955 A1), Metzler, etc. (US 20220005332 A1), and Aluru, etc. (US 20230388470 A1), do not teach the limitations of ''an image generation method, comprising: obtaining density information by a target model corresponding to a target scene space, wherein the target model is trained to output a viewpoint image corresponding to any viewpoint after inputting the any viewpoint in the target scene space; and the density information is a rendering parameter required for image rendering by the target model, and the density information represents transparency of a point in the target scene space; determining a first constraint condition based on the density information; determining a target viewpoint from the target scene space based on the first constraint condition, comprising: determining a target point from the target scene space based on the first constraint condition; determining a plurality of candidate viewpoints based on the target point; obtaining depth values for the plurality of candidate viewpoints, and the depth values are used to represent distances between the plurality of candidate viewpoints and an object in the target scene space; and determining the target viewpoint from the plurality of candidate viewpoints based on the depth values for the plurality of candidate viewpoints, wherein a depth value for the target viewpoint satisfies a depth value requirement; and rendering a viewpoint image corresponding to the target viewpoint by the target model.”
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights