DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The abstract of the disclosure is objected to because the entire front page of a WIPO patent had been used as the abstract. A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
The disclosure is objected to because of the following informalities: the term “static scene (404)” should be “static model (404)” on paragraph 73. Appropriate correction is required.
The disclosure is objected to because of the following informalities: the term “distant region (e.g., sky) (402)” should be “sky model (402)” on paragraph 73. Appropriate correction is required.
The disclosure is objected to because of the following informalities: the term “dynamic actors (406)” should be “actor model (406)” on paragraph 73. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 14, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Blomqvist et al. “Baking in the Feature: Accelerating Volumetric Segmentation by Rendering Feature Maps”, 09/26/2022, arXiv:2209.12744v1 in view of Abou-Chakra et al. “Implicit Object Mapping With Noisy Data”, 04/22/2022, arXiv2204:10516v1, further in view of Rematas et al. (U.S. Pub. No. 20230281913).
Regarding claim 1, Blomqvist discloses a interpolating a plurality of hash grid features adjacent to a location in a neural hash grid (sec 3.A, “An alternative hash grid encoding was introduced in [23], which greatly speeds up the learning of NeRFs. They define a hierarchical voxel grid over the scene coordinates, with parameters at cell vertices for every level. To encode a point, parameters are looked up at each level of the grid, trilinearly interpolated and concatenated to produce an encoding. These parameters are learned jointly with those of the MLP radi ance field.”; also, sec 3.A, “To solve this problem, we propose a hybrid approach using both the hash grid encoding concatenated with low frequency positional encoding with L = 2. This enables the model to both reason about coarse spatial location, while the grid parameters allow for encoding finer details in the scene.”; neural hash grid is described as a multi-resolution feature grid, cubes containing sub-cubes which is a hierarchical voxel grid over the scene coordinates for every level. Hash grid features are that of learned features stored per cubes which parameters at cell vertices for every level are learned jointly with those of the MLP radiance field. Adjacent to a location is that of a parameter that are looked up at each level of a grid.) set of location features (sec 3.A, “To encode a point, parameters are looked up at each level of the grid, trilinearly interpolated and concatenated to produce an encoding.”; also, sec 3, “In this section, we first describe how we encode scene positions into feature vectors that are fed to the neural network.”); processing, by a multilayer perceptron (MLP) model, the set of location features to generate a set of image features for the location (sec 3, “Figure 2 shows a high-level diagram of our model, which consists of five main components: a feature encoder, a geometry MLP, a color MLP, a feature MLP, and a semantic classifier MLP. In this section, we first describe how we encode scene positions into feature vectors that are fed to the neural network”, also, Fig. 2, “Architectural diagram of our model. Scene coordinates and viewing direction are mapped to RGB, density, depth, feature maps and segmentation maps.”; also sec 3, “using the following functions: c for color, d for depth, s for the semantic vector, and f for the intermediate feature vector of our model for encoded point samples along a ray r.”); completing, using the set of image features, ray casting (sec 2.C, “To distill information in the feature maps into our 3D scene representation and to inform a downstream classifier, we propose to volumetrically render feature outputs f along a ray using (2) to produce rendered feature maps ˆ f and supervise on corresponding image features ¯ f”; also, sec 3.B, “To integrate the scalar and vector outputs of our spatial field, we define a function R to volumetrically render vector or scalar values from a function h along a given ray r:”) generate a feature image (sec 2.C, “To distill information in the feature maps into our 3D scene representation and to inform a downstream classifier, we propose to volumetrically render feature outputs f along a ray using (2) to produce rendered feature maps ˆ f and supervise on corresponding image features ¯ f”; also, Sec 3, “We use this rendering function to render pixel color values, depth, semantic outputs and image features for a given ray r:”; also, sec 3, “We then describe how use volumetric rendering to compute 2D image maps from the 3D representation and describe how we learn the parameters of the scene representation from provided data”) generating a rendered image from the feature image and processing the rendered image (sec 3.E, “The right side shows rendered color, depth and features mapped to RGB using PCA.”; also, sec 3.E, “Figure 4 shows the user interface of our system.”). Blomqvist does not disclose a computer-implemented method, defined for a target object; the target object.
However, in a similar field of endeavor, disclose defined for a target object (Sec 4, “We present a method that extends this system and enriches its representation of objects through the use of Neural Radiance Fields (NeRFs). Given a set of images Ii and their corresponding poses Xi, depth maps Di, and instance masks Si, we aim to construct a single NeRF for every object j present in the scene.”; also, sec 2, “Instant-NGP [20] is the only one that does both. This makes it the most suitable for robotic applications.”; also, sec 4, “The first challenge is bounding the NeRF to an area so that its representational power can be focused on the object of interest.”); the target object (sec 4, “In this, we rely on the object-based SLAM system to provide a bounding box Bⱼ that loosely contains the object. We use a ray-box intersection algorithm to ignore rays that do not pass through the bounding box and to limit sampling to the points therein.”; also, sec 4, “To this end, rays that are known to hit the object should encourage geometry to be formed along them.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Blomqvist's invention of a computer-implemented method in which the hash grid parameters at the cell vertices surrounding a point are looked up at each level of a hierarchical voxel grid and trilinearly interpolated to produce the encoding of that point, a feature multilayer perceptron maps the encoding to a semantic feature vector for the point, those feature outputs are volumetrically rendered along each ray to produce rendered feature maps, the rendered features are mapped to red, green, and blue values to produce an image, and that image is shown to the user in the system's user interface, with the features of Abou-Chakra's invention of a neural radiance field with a hash encoding generated for each object in the scene, together with a ray-box intersection test that ignores rays not passing through the object's bounding box and limits sampling to the points inside it, so that the hash grid is defined for a target object and the rays are cast to that object. The combination would have been obvious because Blomqvist defines one hash grid over the coordinates of a whole scene, while Abou-Chakra, using the same multi-resolution hash encoding, shows that such a field can serve as a representation of an object rather than of a scene and names the reason for bounding it, stating in section IV that "The first challenge is bounding the NeRF to an area so that its representational power can be focused on the object of interest." Both references adopt the hash encoding for speed, Blomqvist because, as it explains in section III.A, "As we are targeting an interactive system, using only frequency encoding would not be an option, as it requires many more iterations to learn high frequency details", and Abou-Chakra because, as it states in section IV, "Instant-NGP [20] with a multi-resolution hash encoding is used as the underlying NeRF model because it trains in close to real-time making it the most suitable for robotic applications." A person of ordinary skill building Blomqvist's feature rendering for a scene whose objects are of individual interest would have applied Abou-Chakra's per-object grid and bounding-box ray test to concentrate the grid's capacity on the object being rendered and to skip samples that cannot contribute to it, which serves the speed both references pursue. The predictable result is Blomqvist's interpolate, decode, and render sequence carried out on a hash grid defined for one object, with the rays cast to that object.
Rematas discloses a computer-implemented method (para 99, “FIG. 6 depicts a flow chart diagram of an example method to perform according to example embodiments of the present disclosure.”; also, para 100, “At 602, a computing system can obtain input data. The input data can be descriptive of a three-dimensional position and a two-dimensional view direction.”; also, para 101, “At 604, the computing system can process the input data with a machine-learned view synthesis model to generate a view synthesis output.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra, in which a hash grid defined for a target object is interpolated around each sampled location to obtain location features, a multilayer perceptron maps those features to image features for the location, and rays cast to the object's bounding box are volumetrically rendered into a feature image from which a displayed image is generated, with the features of Rematas's invention of a method whose steps are carried out by a computing system, the computing system obtaining the three-dimensional position and the two-dimensional view direction as input data and processing that input data with a machine-learned view synthesis model to generate a view synthesis output. The combination would have been obvious because Blomqvist describes its work as a NeRF-style algorithm and shows it running behind a graphical user interface, but it does not state that the steps of that algorithm are performed by a computing system, and Abou-Chakra supplies only the per-object field and the ray-box test. Rematas is directed to the same class of volumetric radiance field models and states in paragraphs [0100] and [0101] that a computing system obtains the position and view direction and processes them with the model to produce the rendered output, the same sequence the combination performs. A person of ordinary skill would have carried out the combination's interpolation, multilayer perceptron evaluation, ray casting, and rendering on such a computing system, as Rematas does for its own radiance field model, because a model of this kind is evaluated nowhere else. The predictable result is a computer-implemented method comprising the recited operations.
Regarding claim 14, Blomqvist discloses the system comprising (sec 3.E, “Figure 4 shows the user interface of our system; also, sec 1, “While the user is using the system, the program infers a segmentation given the current user inputs and shows it to the user.”; also, Fig. 4, “The graphical user interface of our system. The user gives sparse annotations by drawing on images in a scan with the appropriate class. The system infers a volumetric segmentation given the current sparse annotations from all views and renders a dense segmentation map which the user can correct with further annotations.”): interpolating a plurality of hash grid features adjacent to a location in a neural hash grid (sec 3.A, “An alternative hash grid encoding was introduced in [23], which greatly speeds up the learning of NeRFs. They define a hierarchical voxel grid over the scene coordinates, with parameters at cell vertices for every level. To encode a point, parameters are looked up at each level of the grid, trilinearly interpolated and concatenated to produce an encoding. These parameters are learned jointly with those of the MLP radi ance field.”; also, sec 3.A, “To solve this problem, we propose a hybrid approach using both the hash grid encoding concatenated with low frequency positional encoding with L = 2. This enables the model to both reason about coarse spatial location, while the grid parameters allow for encoding finer details in the scene.”; neural hash grid is described as a multi-resolution feature grid, cubes containing sub-cubes which is a hierarchical voxel grid over the scene coordinates for every level. Hash grid features are that of learned features stored per cubes which parameters at cell vertices for every level are learned jointly with those of the MLP radiance field. Adjacent to a location is that of a parameter that are looked up at each level of a grid.) set of location features (sec 3.A, “To encode a point, parameters are looked up at each level of the grid, trilinearly interpolated and concatenated to produce an encoding.”; also, sec 3, “In this section, we first describe how we encode scene positions into feature vectors that are fed to the neural network.”); processing, by a multilayer perceptron (MLP) model, the set of location features to generate a set of image features for the location (sec 3, “Figure 2 shows a high-level diagram of our model, which consists of five main components: a feature encoder, a geometry MLP, a color MLP, a feature MLP, and a semantic classifier MLP. In this section, we first describe how we encode scene positions into feature vectors that are fed to the neural network”, also, Fig. 2, “Architectural diagram of our model. Scene coordinates and viewing direction are mapped to RGB, density, depth, feature maps and segmentation maps.”; also sec 3, “using the following functions: c for color, d for depth, s for the semantic vector, and f for the intermediate feature vector of our model for encoded point samples along a ray r.”); completing, using the set of image features, ray casting (sec 3.C, “To distill information in the feature maps into our 3D scene representation and to inform a downstream classifier, we propose to volumetrically render feature outputs f along a ray using (2) to produce rendered feature maps ˆ f and supervise on corresponding image features ¯ f”; also, sec 3.B, “integrate the scalar and vector outputs of our spatial field, we define a function R to volumetrically render vector or scalar values from a function h along a given ray r:”) generate a feature image (sec 2.C, “To distill information in the feature maps into our 3D scene representation and to inform a downstream classifier, we propose to volumetrically render feature outputs f along a ray using (2) to produce rendered feature maps ˆ f and supervise on corresponding image features ¯ f”; also, Sec 3, “We use this rendering function to render pixel color values, depth, semantic outputs and image features for a given ray r:”; also, sec 3, “We then describe how use volumetric rendering to compute 2D image maps from the 3D representation and describe how we learn the parameters of the scene representation from provided data”); generating a rendered image from the feature image; and processing the rendered image (sec 3.E, “The right side shows rendered color, depth and features mapped to RGB using PCA.”). Blomqvist does not disclose memory; and a computer processor comprising computer readable program code for performing operations, defined for a target object, and the target object.
However, in a similar field of endeavor, Abou-Chakra discloses defined for a target object (Sec 4, “We present a method that extends this system and enriches its representation of objects through the use of Neural Radiance Fields (NeRFs). Given a set of images Ii and their corresponding poses Xi, depth maps Di, and instance masks Si, we aim to construct a single NeRF for every object j present in the scene.”; also, sec 2, “Instant-NGP [20] is the only one that does both. This makes it the most suitable for robotic applications.”; also, sec 4, “The first challenge is bounding the NeRF to an area so that its representational power can be focused on the object of interest.”); the target object (sec 4, “In this, we rely on the object-based SLAM system to provide a bounding box Bⱼ that loosely contains the object. We use a ray-box intersection algorithm to ignore rays that do not pass through the bounding box and to limit sampling to the points therein.”; also, sec 4, “To this end, rays that are known to hit the object should encourage geometry to be formed along them.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Blomqvist's invention of a system in which the hash grid parameters at the cell vertices surrounding a point are looked up at each level of a hierarchical voxel grid and trilinearly interpolated to produce the encoding of that point, a feature multilayer perceptron maps the encoding to a semantic feature vector for the point, those feature outputs are volumetrically rendered along each ray to produce rendered feature maps, the rendered features are mapped to red, green, and blue values to produce an image, and that image is shown to the user in the system's user interface, with the features of Abou-Chakra's invention of a neural radiance field with a hash encoding generated for each object in the scene, together with a ray-box intersection test that ignores rays not passing through the object's bounding box and limits sampling to the points inside it, so that the hash grid is defined for a target object and the rays are cast to that object. The combination would have been obvious because Blomqvist defines one hash grid over the coordinates of a whole scene, while Abou-Chakra, using the same multi-resolution hash encoding, shows that such a field can serve as a representation of an object rather than of a scene and names the reason for bounding it, stating in section IV that "The first challenge is bounding the NeRF to an area so that its representational power can be focused on the object of interest." Both references adopt the hash encoding for speed, Blomqvist because, as it explains in section III.A, "As we are targeting an interactive system, using only frequency encoding would not be an option, as it requires many more iterations to learn high frequency details", and Abou-Chakra because, as it states in section IV, "Instant-NGP [20] with a multi-resolution hash encoding is used as the underlying NeRF model because it trains in close to real-time making it the most suitable for robotic applications." A person of ordinary skill building Blomqvist's feature rendering for a scene whose objects are of individual interest would have applied Abou-Chakra's per-object grid and bounding-box ray test to concentrate the grid's capacity on the object being rendered and to skip samples that cannot contribute to it, which serves the speed both references pursue. The predictable result is Blomqvist's interpolate, decode, and render sequence carried out on a hash grid defined for one object, with the rays cast to that object.
Rematas discloses memory (para 53, “The user computing device 102 includes one or more processors 112 and a memory 114.”; also, para 55, “In some implementations, the one or more view synthesis models 120 can be received from the server computing system 130 over network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112.”); and a computer processor comprising computer readable program code for performing operations (para 53, “The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected.”; also, para 53, “The memory 114 can store data 116 and instructions 118 which are executed by the processor 112 to cause the user computing device 102 to perform operations.”; also, para 55, “In some implementations, the one or more view synthesis models 120 can be received from the server computing system 130 over network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra, in which a system interpolates a hash grid defined for a target object around each sampled location, maps the interpolated features to image features with a multilayer perceptron, and volumetrically renders rays cast to the object's bounding box into a feature image from which a displayed image is generated, with the features of Rematas's invention of a user computing device that includes one or more processors and a memory, the memory storing data and instructions that the processors execute to cause the device to perform operations, and view synthesis models that are stored in that memory and then used or otherwise implemented by the processors. The combination would have been obvious because Blomqvist describes its segmentation as a system and as a program that runs while the user works, but it does not describe the computing hardware on which that program is stored and executed, and a neural rendering method of this kind can be practiced only on such hardware. Rematas is directed to the same class of volumetric radiance field models and, in paragraphs [0053] and [0055], describes the conventional arrangement in which such models are stored in memory and implemented by processors, adding in paragraph [0055] that the device "can implement multiple parallel instances of a single view synthesis model 120" for parallel rendering. A person of ordinary skill would have stored the combination's operations as instructions in such a memory and executed them on such a processor, as Rematas does for its own radiance field models, with the predictable result of a system of a memory and a computer processor carrying the program code that performs the operations.
Regarding claim 19, Blomqvist discloses interpolating a plurality of hash grid features adjacent to a location in a neural hash grid (sec 3, “An alternative hash grid encoding was introduced in [23], which greatly speeds up the learning of NeRFs. They define a hierarchical voxel grid over the scene coordinates, with parameters at cell vertices for every level. To encode a point, parameters are looked up at each level of the grid, trilinearly interpolated and concatenated to produce an encoding. These parameters are learned jointly with those of the MLP radiance field.”; also, sec 3.A, “To solve this problem, we propose a hybrid approach using both the hash grid encoding concatenated with low frequency positional encoding with L = 2. This enables the model to both reason about coarse spatial location, while the grid parameters allow for encoding finer details in the scene.”) obtain a set of location features (sec 3.A, “To encode a point, parameters are looked up at each level of the grid, trilinearly interpolated and concatenated to produce an encoding.”; also, sec 3, “In this section, we first describe how we encode scene positions into feature vectors that are fed to the neural network.”); processing, by a multilayer perceptron (MLP) model, the set of location features to generate a set of image features for the location (sec 3, “Figure 2 shows a high-level diagram of our model, which consists of five main components: a feature encoder, a geometry MLP, a color MLP, a feature MLP, and a semantic classifier MLP. In this section, we first describe how we encode scene positions into feature vectors that are fed to the neural network”, also, Fig. 2, “Architectural diagram of our model. Scene coordinates and viewing direction are mapped to RGB, density, depth, feature maps and segmentation maps.”; also sec 3, “using the following functions: c for color, d for depth, s for the semantic vector, and f for the intermediate feature vector of our model for encoded point samples along a ray r.”); completing, using the set of image features, ray casting (sec 3.C, “To distill information in the feature maps into our 3D scene representation and to inform a downstream classifier, we propose to volumetrically render feature outputs f along a ray using (2) to produce rendered feature maps ˆ f and supervise on corresponding image features ¯”; also, sec 3.B, “To integrate the scalar and vector outputs of our spatial field, we define a function R to volumetrically render vector or scalar values from a function h along a given ray r: “) generate a feature image (sec 2.C, “To distill information in the feature maps into our 3D scene representation and to inform a downstream classifier, we propose to volumetrically render feature outputs f along a ray using (2) to produce rendered feature maps ˆ f and supervise on corresponding image features ¯ f”; also, Sec 3, “We use this rendering function to render pixel color values, depth, semantic outputs and image features for a given ray r:”; also, sec 3, “We then describe how use volumetric rendering to compute 2D image maps from the 3D representation and describe how we learn the parameters of the scene representation from provided data”); generating a rendered image from the feature image; and processing the rendered image (sec 3.E, “The right side shows rendered color, depth and features mapped to RGB using PCA.”; also, sec 3.E, “Figure 4 shows the user interface of our system.”). Blomqvist does not disclose the non-transitory computer readable medium comprising computer readable program code for performing operations, defined for a target object, and to the target object.
However, in a similar field of endeavor, Abou-Chakra discloses defined for a target object (Sec 4, “We present a method that extends this system and enriches its representation of objects through the use of Neural Radiance Fields (NeRFs). Given a set of images Ii and their corresponding poses Xi, depth maps Di, and instance masks Si, we aim to construct a single NeRF for every object j present in the scene.”; also, sec 2, “Instant-NGP [20] is the only one that does both. This makes it the most suitable for robotic applications.”; also, sec 4, “The first challenge is bounding the NeRF to an area so that its representational power can be focused on the object of interest.”); the target object (sec 4, “In this, we rely on the object-based SLAM system to provide a bounding box Bⱼ that loosely contains the object. We use a ray-box intersection algorithm to ignore rays that do not pass through the bounding box and to limit sampling to the points therein.”; also, sec 4, “To this end, rays that are known to hit the object should encourage geometry to be formed along them.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Blomqvist's invention of a program in which the hash grid parameters at the cell vertices surrounding a point are looked up at each level of a hierarchical voxel grid and trilinearly interpolated to produce the encoding of that point, a feature multilayer perceptron maps the encoding to a semantic feature vector for the point, those feature outputs are volumetrically rendered along each ray to produce rendered feature maps, the rendered features are mapped to red, green, and blue values to produce an image, and that image is shown to the user in the system's user interface, with the features of Abou-Chakra's invention of a neural radiance field with a hash encoding generated for each object in the scene, together with a ray-box intersection test that ignores rays not passing through the object's bounding box and limits sampling to the points inside it, so that the hash grid is defined for a target object and the rays are cast to that object. The combination would have been obvious because Blomqvist defines one hash grid over the coordinates of a whole scene, while Abou-Chakra, using the same multi-resolution hash encoding, shows that such a field can serve as a representation of an object rather than of a scene and names the reason for bounding it, stating in section IV that "The first challenge is bounding the NeRF to an area so that its representational power can be focused on the object of interest." Both references adopt the hash encoding for speed, Blomqvist because, as it explains in section III.A, "As we are targeting an interactive system, using only frequency encoding would not be an option, as it requires many more iterations to learn high frequency details", and Abou-Chakra because, as it states in section IV, "Instant-NGP [20] with a multi-resolution hash encoding is used as the underlying NeRF model because it trains in close to real-time making it the most suitable for robotic applications." A person of ordinary skill building Blomqvist's feature rendering for a scene whose objects are of individual interest would have applied Abou-Chakra's per-object grid and bounding-box ray test to concentrate the grid's capacity on the object being rendered and to skip samples that cannot contribute to it, which serves the speed both references pursue. The predictable result is Blomqvist's interpolate, decode, and render sequence carried out on a hash grid defined for one object, with the rays cast to that object.
Ramatas discloses a non-transitory computer readable medium comprising computer readable program code for performing operations (para 53, “The memory 114 can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof.”; also, para 53, “The memory 114 can store data 116 and instructions 118 which are executed by the processor 112 to cause the user computing device 102 to perform operations.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra, in which program code interpolates a hash grid defined for a target object around each sampled location, maps the interpolated features to image features with a multilayer perceptron, and volumetrically renders rays cast to the object's bounding box into a feature image from which a displayed image is generated, with the features of Rematas's invention of a memory that includes one or more non-transitory computer-readable storage mediums and that stores data and instructions which the device's processors execute to cause the device to perform operations. The combination would have been obvious because Blomqvist describes its segmentation as a system and as a program that runs while the user works, but it does not describe the computing hardware on which that program is stored and executed, and a neural rendering method of this kind can be practiced only on such hardware. Rematas is directed to the same class of volumetric radiance field models and, in paragraphs [0053] and [0055], describes the conventional arrangement in which such models are stored in memory and implemented by processors, adding in paragraph [0055] that the device "can implement multiple parallel instances of a single view synthesis model 120" for parallel rendering. A person of ordinary skill would have stored the combination's operations as instructions in such a memory and executed them on such a processor, as Rematas does for its own radiance field models, with the predictable result of a non-transitory computer readable medium comprising the program code that performs the operations.
Claim(s) 2-3, 15-16 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Blomqvist et al. “Baking in the Feature: Accelerating Volumetric Segmentation by Rendering Feature Maps”, 09/26/2022, arXiv:2209.12744v1 as modified by Abou-Chakra et al. “Implicit Object Mapping With Noisy Data”, 04/22/2022, arXiv2204:10516v1 and Rematas et al. (U.S. Pub. No. 20230281913), further in view of Niemeyer et al. “GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields”, 11/24/2020, arXive:2011.12100v1.
Regarding claim 2, Blomqvist as modified by Abou-Chakra and Rematas discloses the computer-implemented method of claim 1, further comprising: executing a convolutional neural network (CNN) decoder on the feature image to generate the rendered image.
However, in a similar field of endeavor, Niemeyer discloses executing a convolutional neural network (CNN) decoder on the feature image to generate the rendered image (sec 1, ” A neural renderer processes these feature images and out puts the final renderings.”; also, sec 3.3, “We then upsample the low resolution feature maps to higher-resolution RGB images using 2D neural rendering.”; also, sec 3.3, “We parameterize πneural θ as a 2D convolutional neural network (CNN) with leaky ReLU [55,88] activation (Fig. 4) and combine nearest neighbor upsampling with 3 × 3 convolutions to increase the spatial resolution.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra and Rematas, in which a hash grid defined for a target object is interpolated around each sampled location to obtain location features, a multilayer perceptron maps those features to image features for the location, and rays cast to the object's bounding box are volumetrically rendered into a feature image from which a displayed image is generated, with the features of Niemeyer's invention of a neural rendering operator, parameterized as a two-dimensional convolutional neural network, that maps the volume-rendered feature image to the final synthesized image by processing it through blocks of nearest neighbor upsampling and 3 × 3 convolutions. The combination would have been obvious because Blomqvist turns its rendered feature maps into a viewable image only by mapping the features to color with principal component analysis, a visualization that is not trained to reproduce the scene's appearance, while Niemeyer is directed to the same kind of volume-rendered feature image and supplies a learned convolutional decoder for exactly the step of converting it into a final image. Niemeyer states in section 3.3 that "We then upsample the low-resolution feature maps to higher-resolution RGB images using 2D neural rendering", and reports of that arrangement that "As evidenced by our experiments, this has two advantages: increased rendering speed and improved image quality." A person of ordinary skill seeking a realistic image from the combination's feature image, rather than a false-color view of it, would have adopted Niemeyer's decoder for those two stated advantages, and would have expected it to work because the combination's feature image, like Niemeyer's, is an accumulation along each ray of per-location features output by a multilayer perceptron. The predictable result is the combination's feature image passed through a convolutional neural network decoder that outputs the rendered image.
Regarding claim 3, Blomqvist as modified by Abou-Chakra, Rematas, and Niemeyer discloses the computer-implemented method of claim 2, wherein Niemeyer discloses the completing the ray casting to generate the feature image generates the feature image in a first resolution (sec 3.3, “For efficiency, we render feature images at resolution 162 which is lower than the output resolution of 642 or 2562 pixels.”; also, sec 3.3, “The entire feature image is obtained by evaluating πvol at every pixel.”), wherein the CNN decoder upscales the feature image in the first resolution to a second resolution when generating the rendered image (sec 3.3, “We then upsample the low-resolution feature maps to higher-resolution RGB images using 2D neural rendering”; also, sec 3.3, “We param eterize πneural θ as a 2D convolutional neural network (CNN) with leaky ReLU [55,88] activation (Fig. 4) and combine nearest neighbor upsampling with 3 × 3 convolutions to in crease the spatial resolution.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra, Rematas, and Niemeyer, in which a hash grid defined for a target object is interpolated around each sampled location to obtain location features, a multilayer perceptron maps those features to image features for the location, and rays cast to the object's bounding box are volumetrically rendered into a feature image from which a displayed image is generated, and a convolutional neural network decoder turns the feature image into the rendered image, with the features of Niemeyer's invention of rendering the feature images at a resolution of 16², lower than the output resolution of 64² or 256² pixels, and upsampling those low-resolution feature maps to higher-resolution images in the convolutional neural network by combining nearest neighbor upsampling with 3 × 3 convolutions. The combination would have been obvious because, once a decoder sits between the feature image and the output, the resolution at which the feature image is volume rendered becomes a free choice, and every feature-image pixel costs a full ray of volumetric sampling. Niemeyer makes that choice expressly and for that reason, stating in section 3.3 that "For efficiency, we render feature images at resolution 16² which is lower than the output resolution of 64² or 256² pixels", and reporting in section 3.3 that the arrangement gives "increased rendering speed and improved image quality." A person of ordinary skill who had adopted Niemeyer's decoder would also have adopted the low-resolution rendering and in-decoder upsampling it was designed around, with the predictable result that the ray casting generates the feature image at a first resolution and the decoder upscales it to a second resolution in generating the rendered image.
Regarding claim 15, Blomqvist as modified by Abou-Chakra and Rematas discloses the system of claim 14, wherein the operations further comprise: executing a convolutional neural network (CNN) decoder on the feature image to generate the rendered image.
However, in a similar field of endeavor, Niemeyer discloses executing a convolutional neural network (CNN) decoder on the feature image to generate the rendered image (sec 3.3, “The feature image IV is processed by n blocks of nearest neighbor upsampling and 3×3 convolutions with leaky ReLU activations. At every resolution, we map the feature image to an RGB image with a 3×3 convolution and add it to the previous output via bilinear upsampling. We apply a sigmoid activation to obtain the final image ˆ I.”; also, sec 3.3, “We then upsample the low resolution feature maps to higher-resolution RGB images using 2D neural rendering.”; also, sec 3.3, “We param eterize πneural θ as a 2D convolutional neural network (CNN) with leaky ReLU [55,88] activation (Fig. 4) and combine nearest neighbor upsampling with 3 × 3 convolutions to in crease the spatial resolution.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra and Rematas, in which a system of a memory and a processor executing stored program code interpolates a hash grid defined for a target object around each sampled location, maps the interpolated features to image features with a multilayer perceptron, and volumetrically renders rays cast to the object's bounding box into a feature image from which a displayed image is generated, with the features of Niemeyer's invention of a neural rendering operator, parameterized as a two-dimensional convolutional neural network, that maps the volume-rendered feature image to the final synthesized image by processing it through blocks of nearest neighbor upsampling and 3 × 3 convolutions. The combination would have been obvious because Blomqvist turns its rendered feature maps into a viewable image only by mapping the features to color with principal component analysis, a visualization that is not trained to reproduce the scene's appearance, while Niemeyer is directed to the same kind of volume-rendered feature image and supplies a learned convolutional decoder for exactly the step of converting it into a final image. Niemeyer states in section 3.3 that "We then upsample the low-resolution feature maps to higher-resolution RGB images using 2D neural rendering", and reports of that arrangement that "As evidenced by our experiments, this has two advantages: increased rendering speed and improved image quality." A person of ordinary skill seeking a realistic image from the combination's feature image, rather than a false-color view of it, would have adopted Niemeyer's decoder for those two stated advantages, and would have expected it to work because the combination's feature image, like Niemeyer's, is an accumulation along each ray of per-location features output by a multilayer perceptron. The predictable result is the combination's feature image passed through a convolutional neural network decoder that outputs the rendered image.
Regarding claim 16, Blomqvist as modified by Abou-Chakra, Rematas, and Niemeyer discloses the system of claim 15, wherein Niemeyer further discloses the completing the ray casting to generate the feature image generates the feature image in a first resolution (sec 3.3, “For efficiency, we render feature images at resolution 162 which is lower than the output resolution of 642 or 2562 pixels.”; also, sec 3.3, “The entire feature image is obtained by evaluating πvol at every pixel.”), wherein the CNN decoder upscales the feature image in the first resolution to a second resolution when generating the rendered image (sec 3.3, “We then upsample the low resolution feature maps to higher-resolution RGB images using 2D neural rendering.”; also, sec 3.3, “We parameterize πneural θ as a 2D convolutional neural network (CNN) with leaky ReLU [55,88] activation (Fig. 4) and combine nearest neighbor upsampling with 3 × 3 convolutions to increase the spatial resolution.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra, Rematas, and Niemeyer, in which a system of a memory and a processor executing stored program code interpolates a hash grid defined for a target object around each sampled location, maps the interpolated features to image features with a multilayer perceptron, and volumetrically renders rays cast to the object's bounding box into a feature image from which a displayed image is generated, and a convolutional neural network decoder turns the feature image into the rendered image, with the features of Niemeyer's invention of rendering the feature images at a resolution of 16², lower than the output resolution of 64² or 256² pixels, and upsampling those low-resolution feature maps to higher-resolution images in the convolutional neural network by combining nearest neighbor upsampling with 3 × 3 convolutions. The combination would have been obvious because, once a decoder sits between the feature image and the output, the resolution at which the feature image is volume rendered becomes a free choice, and every feature-image pixel costs a full ray of volumetric sampling. Niemeyer makes that choice expressly and for that reason, stating in section 3.3 that "For efficiency, we render feature images at resolution 16² which is lower than the output resolution of 64² or 256² pixels", and reporting in section 3.3 that the arrangement gives "increased rendering speed and improved image quality." A person of ordinary skill who had adopted Niemeyer's decoder would also have adopted the low-resolution rendering and in-decoder upsampling it was designed around, with the predictable result that the ray casting generates the feature image at a first resolution and the decoder upscales it to a second resolution in generating the rendered image.
Regarding claim 20, Blomqvist as modified by Abou-Chakra and Rematas discloses the non-transitory computer readable medium of claim 19, wherein the operations further comprise:executing a convolutional neural network (CNN) decoder on the feature image to generate the rendered image.
However, in a similar field of endeavor, Niemeyer discloses wherein the operations further comprise: executing a convolutional neural network (CNN) decoder on the feature image to generate the rendered image (sec 3.3, “The feature image IV is processed by n blocks of nearest neighbor upsampling and 3×3 convolutions with leaky ReLU activations. At every resolution, we map the feature image to an RGB image with a 3×3 convolution and add it to the previous output via bilinear upsampling. We apply a sigmoid activation to obtain the final image ˆ I.”; also, sec 3.3, “We then upsample the low resolution feature maps to higher-resolution RGB images using 2D neural rendering.”; also, sec 3.3, “We param eterize πneural θ as a 2D convolutional neural network (CNN) with leaky ReLU [55,88] activation (Fig. 4) and combine nearest neighbor upsampling with 3 × 3 convolutions to in crease the spatial resolution.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra and Rematas, in which program code stored on a non-transitory computer-readable storage medium causes a processor to interpolate a hash grid defined for a target object around each sampled location, map the interpolated features to image features with a multilayer perceptron, and volumetrically render rays cast to the object's bounding box into a feature image from which a displayed image is generated, with the features of Niemeyer's invention of a neural rendering operator, parameterized as a two-dimensional convolutional neural network, that maps the volume-rendered feature image to the final synthesized image by processing it through blocks of nearest neighbor upsampling and 3 × 3 convolutions. The combination would have been obvious because Blomqvist turns its rendered feature maps into a viewable image only by mapping the features to color with principal component analysis, a visualization that is not trained to reproduce the scene's appearance, while Niemeyer is directed to the same kind of volume-rendered feature image and supplies a learned convolutional decoder for exactly the step of converting it into a final image. Niemeyer states in section 3.3 that "We then upsample the low-resolution feature maps to higher-resolution RGB images using 2D neural rendering", and reports of that arrangement that "As evidenced by our experiments, this has two advantages: increased rendering speed and improved image quality." A person of ordinary skill seeking a realistic image from the combination's feature image, rather than a false-color view of it, would have adopted Niemeyer's decoder for those two stated advantages, and would have expected it to work because the combination's feature image, like Niemeyer's, is an accumulation along each ray of per-location features output by a multilayer perceptron. The predictable result is the combination's feature image passed through a convolutional neural network decoder that outputs the rendered image.
Claim(s) 4 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Blomqvist et al. “Baking in the Feature: Accelerating Volumetric Segmentation by Rendering Feature Maps”, 09/26/2022, arXiv:2209.12744v1 as modified by Abou-Chakra et al. “Implicit Object Mapping With Noisy Data”, 04/22/2022, arXiv2204:10516v1, Rematas et al. (U.S. Pub. No. 20230281913), and Niemeyer et al. “GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields”, 11/24/2020, arXive:2011.12100v1, further in view of Barron et al. “Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields”, 08/14/2021, arXiv:2103.13415v3.
Regarding claim 4, Blomqvist as modified by Abou-Chakra, Rematas, and Niemeyer discloses the computer-implemented method of claim 2, completing the ray casting to generate the feature image corrects a rendering artifact in the feature image.
However, in a similar field of endeavor, Barron disclose wherein the completing the ray casting to generate the feature image corrects a rendering artifact in the feature image (sec 3, “As discussed, NeRF’s point-sampling makes it vulnerable to issues related to sampling and aliasing: Though a pixel’s color is the integration of all incoming radiance within the pixel’s frustum, NeRF casts a single infinitesimally narrow ray per pixel, resulting in aliasing. Mip-NeRF ameliorates this issue by casting a cone from each pixel. Instead of performing point-sampling along each ray, we divide the cone being cast into a series of conical frustums (cones cut perpendicular to their axis).”); also, sec 3.1, “Here we describe mip-NeRF’s rendering and featuriza tion procedure, in which we cast a cone and featurize conical frustums along that cone”; also, Fig. 3, “NeRF works by extracting point-sampled positional encoding features (shown here as dots) along each pixel’s ray. Those point-sampled features ignore the shape and size of the volume viewed by each ray, so two different cameras imaging the same position at different scales may produce the same ambiguous point-sampled feature, there by significantly degrading NeRF’s performance. In contrast, Mip-NeRF casts cones instead of rays and explicitly models the volume of each sampled conical frustum (shown here as trapezoids), thus resolving this ambiguity.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra, Rematas, and Niemeyer, in which a hash grid defined for a target object is interpolated around each sampled location to obtain location features, a multilayer perceptron maps those features to image features for the location, and rays cast to the object's bounding box are volumetrically rendered into a feature image from which a displayed image is generated, and a convolutional neural network decoder turns the feature image into the rendered image, with the features of Barron's invention of casting a cone from each pixel instead of a single narrow ray, dividing that cone into conical frustums along its axis, and featurizing the volume of each frustum, which removes the aliasing that casting one infinitesimally narrow ray per pixel produces in the rendered output. The combination would have been obvious because the combination forms its feature image by casting rays through a volumetric field in the same way as the neural radiance fields Barron addresses, and it therefore carries the same defect, which Barron identifies in section 3, stating that "Though a pixel’s color is the integration of all incoming radiance within the pixel’s frustum, NeRF casts a single infinitesimally narrow ray per pixel, resulting in aliasing." Barron supplies the remedy in the same section, stating that "Mip-NeRF ameliorates this issue by casting a cone from each pixel", and describes in section 3.1 the rendering procedure "in which we cast a cone and featurize conical frustums along that cone." A person of ordinary skill would have applied that remedy at the ray casting stage rather than after it, because aliasing present in the feature image is carried by the decoder into the rendered image and cannot be undone there, and because the cone replaces the single ray cast through each pixel rather than adding further rays to it, so the correction does not multiply the number of rays the combination must render. The predictable result is a ray casting step that, by casting a cone through each pixel and featurizing the volume it covers, corrects aliasing artifacts in the feature image it generates.
Regarding claim 17, Blomqvist as modified by Abou-Chakra, Rematas, Niemeyer discloses the system of claim 15, ray casting to generate the feature image corrects a rendering artifact in the feature image.
However, in a similar field of endeavor, Barron discloses wherein the completing the ray casting to generate the feature image corrects a rendering artifact in the feature image (sec 3, “As discussed, NeRF’s point-sampling makes it vulnerable to issues related to sampling and aliasing: Though a pixel’s color is the integration of all incoming radiance within the pixel’s frustum, NeRF casts a single infinitesimally narrow ray per pixel, resulting in aliasing. Mip-NeRF ameliorates this issue by casting a cone from each pixel. Instead of performing point-sampling along each ray, we divide the cone being cast into a series of conical frustums (cones cut perpendicular to their axis).”); also, sec 3.1, “Here we describe mip-NeRF’s rendering and featuriza tion procedure, in which we cast a cone and featurize conical frustums along that cone”; also, Fig. 3, “NeRF works by extracting point-sampled positional encoding features (shown here as dots) along each pixel’s ray. Those point-sampled features ignore the shape and size of the volume viewed by each ray, so two different cameras imaging the same position at different scales may produce the same ambiguous point-sampled feature, there by significantly degrading NeRF’s performance. In contrast, Mip-NeRF casts cones instead of rays and explicitly models the volume of each sampled conical frustum (shown here as trapezoids), thus resolving this ambiguity.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra, Rematas, and Niemeyer, in which a system of a memory and a processor executing stored program code interpolates a hash grid defined for a target object around each sampled location, maps the interpolated features to image features with a multilayer perceptron, and volumetrically renders rays cast to the object's bounding box into a feature image from which a displayed image is generated, and a convolutional neural network decoder turns the feature image into the rendered image, with the features of Barron's invention of casting a cone from each pixel instead of a single narrow ray, dividing that cone into conical frustums along its axis, and featurizing the volume of each frustum, which removes the aliasing that casting one infinitesimally narrow ray per pixel produces in the rendered output. The combination would have been obvious because the combination forms its feature image by casting rays through a volumetric field in the same way as the neural radiance fields Barron addresses, and it therefore carries the same defect, which Barron identifies in section 3, stating that "Though a pixel’s color is the integration of all incoming radiance within the pixel’s frustum, NeRF casts a single infinitesimally narrow ray per pixel, resulting in aliasing." Barron supplies the remedy in the same section, stating that "Mip-NeRF ameliorates this issue by casting a cone from each pixel", and describes in section 3.1 the rendering procedure "in which we cast a cone and featurize conical frustums along that cone." A person of ordinary skill would have applied that remedy at the ray casting stage rather than after it, because aliasing present in the feature image is carried by the decoder into the rendered image and cannot be undone there, and because the cone replaces the single ray cast through each pixel rather than adding further rays to it, so the correction does not multiply the number of rays the combination must render. The predictable result is a ray casting step that, by casting a cone through each pixel and featurizing the volume it covers, corrects aliasing artifacts in the feature image it generates.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Blomqvist et al. “Baking in the Feature: Accelerating Volumetric Segmentation by Rendering Feature Maps”, 09/26/2022, arXiv:2209.12744v1 as modified by Abou-Chakra et al. “Implicit Object Mapping With Noisy Data”, 04/22/2022, arXiv2204:10516v1 and Rematas et al. (U.S. Pub. No. 20230281913), further in view Ost et al. “Neural Scene Graphs for Dynamic Scenes”, 03/05/2021, arXiv:2011.10379v3.
Regarding claim 6, Blomqvist as modified by Abou-Chakra and Rematas discloses the computer-implemented method of claim 1, wherein Blomqvist further comprising: target object is intercepted by a ray during the ray casting; and selecting, from a plurality of neural hash grids for a plurality of objects, the neural hash grid matching the target object based on the target object being intercepted by the ray.
However, in a similar field of endeavor, Abou-Chakra discloses determining that the target object is intercepted by a ray during the ray casting (sec 4, “In this, we rely on the object-based SLAM system to provide a bounding box Bⱼ that loosely contains the object. We use a ray-box intersection algorithm to ignore rays that do not pass through the bounding box and to limit sampling to the points therein.”; also, sec 4, “To this end, rays that are known to hit the object should encourage geometry to be formed along them.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra and Rematas, in which a hash grid defined for a target object is interpolated around each sampled location to obtain location features, a multilayer perceptron maps those features to image features for the location, and rays cast to the object's bounding box are volumetrically rendered into a feature image from which a displayed image is generated, with the features of Abou-Chakra's invention of a ray-box intersection algorithm that tests each cast ray against the bounding box supplied for the object, ignores the rays that do not pass through that box, and limits the sampling to the points inside it. The combination would have been obvious because the combination already casts its rays to the bounding box of the object whose grid it evaluates, so the renderer must decide, ray by ray, whether a given ray reaches that object at all, and Abou-Chakra supplies the test that makes that decision. Abou-Chakra states the test in section IV, explaining that it relies on the mapping system "to provide a bounding box Bⱼ that loosely contains the object" and that "We use a ray-box intersection algorithm to ignore rays that do not pass through the bounding box and to limit sampling to the points therein", and describes the rays the test admits as those "that are known to hit the object." A person of ordinary skill would have applied that test for the reason Abou-Chakra gives, namely to confine the sampling to the object the grid represents, which is also the economy both references pursue in adopting the hash encoding. The predictable result is that the renderer determines, from the ray-box test, that the target object is intercepted by a ray during the ray casting.
Ost discloses selecting, from a plurality of neural hash grids for a plurality of objects, the neural hash grid matching the target object based on the target object being intercepted by the ray (sec 3.2, “Each object is represented by a neural radiance field in the local space of its node and position po"; also, sec 4.1, “We check each ray from the camera C for inter sections with all dynamic nodes Fθo , by translating the ray to an object local frame and applying an efficient AABB ray intersection test as proposed by Majercik et al. [18].”; also, sec 4.1, “Along this ray, we sample points at all graph nodes intersected.”; also, sec 4.1, “At each point where a representation model is hit, a color and volumetric density are computed, and we calculate the pixel color by applying a differentiable integration along the ray.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra and Rematas, in which a hash grid defined for a target object is interpolated around each sampled location to obtain location features, a multilayer perceptron maps those features to image features for the location, and rays cast to the object's bounding box are volumetrically rendered into a feature image from which a displayed image is generated and the renderer determines from a ray-box test whether the target object is intercepted by a ray, with the features of Ost's invention of a scene in which each object carries its own radiance field in the local space of its node, every camera ray is tested for intersection against all of the object nodes, and the points sampled along that ray are taken at the nodes the ray was found to intersect, the representation model of each such node being evaluated at those points. The combination would have been obvious because Abou-Chakra builds one hash-encoded field for every object in a scene but describes the ray-box test only as a filter applied while a single object's field is being fit, and it never says which field a renderer should evaluate when a ray crosses a scene holding several objects. Ost answers that question directly for a scene of per-object fields, stating in section 4.1 that "We check each ray from the camera C for intersections with all dynamic nodes F_θₒ, by translating the ray to an object local frame and applying an efficient AABB-ray intersection test as proposed by Majercik et al. [18].", that "Along this ray, we sample points at all graph nodes intersected", and that "At each point where a representation model is hit, a color and volumetric density are computed." A person of ordinary skill rendering a scene in which each object carries its own hash grid would have adopted Ost's per-ray test against all of the objects, because a collection of per-object fields cannot be rendered at all unless the renderer determines, for each ray, which object's field to evaluate, and Ost supplies exactly that determination for exactly that arrangement. The predictable result is that, of the plurality of neural hash grids kept for the plurality of objects, the grid selected and evaluated for a given ray is the grid of the object that the ray was found to intercept.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Blomqvist et al. “Baking in the Feature: Accelerating Volumetric Segmentation by Rendering Feature Maps”, 09/26/2022, arXiv:2209.12744v1 as modified by Abou-Chakra et al. “Implicit Object Mapping With Noisy Data”, 04/22/2022, arXiv2204:10516v1 and Rematas et al. (U.S. Pub. No. 20230281913), further in view Muller et al. “Instant Neural Graphics Primitives with a Multiresolution Hash Encoding”, 05/04/2022, arXiv:2201.05989v2.
Regarding claim 7, Blomqvist as modified by Abou-Chakra and Rematas discloses the computer-implemented method of claim 1, wherein Blomqvist further comprising: loss based on a difference between the neural hash grid and a sensor image (sec 3.B, “We define the same photometric loss Lrgb as in NeRF [1] and a depth loss Ld similar to the one used by [15]:”; also, sec 3.B, “where ¯ c(r) is the ground truth and ˆ c(r) the predicted color for ray r, ¯d(r) is the ground truth depth (if available) and ˆ d(r) the integrated depth predictions along ray r.”; also, sec 3, “We optimize the combined loss using stochastic gradient descent using Adam over rays r sampled from the images:”; also, sec 4, “The user can move through frames in a captured RGB-D video sequence and draw annotations,”); initializing the neural hash grid with the plurality of hash grid features set to a plurality of random values; and backpropagating the loss through the MLP model and the neural hash grid to update the plurality of hash grid features.
However, in a similar field of endeavor, Muller disclose initializing the neural hash grid with the plurality of hash grid features set to a plurality of random values (sec 3, “We map each corner to an entry in the level’s respective feature vector array, which has fixed size of at most 𝑇.”; also, sec 4, “We initialize the hash table entries using the uniform distribution U(−10−4,10−4) to provide a small amount of randomness while encouraging initial predictions close to zero.”); and backpropagating the loss through the MLP model and the neural hash grid to update the plurality of hash grid features (Fig. 3, “To train the encoding, loss gradients are backpropagated through the MLP (5), the concatenation (4), the linear interpolation (3), and then accumulated in the looked-up feature vectors”; also, sec 4, “We jointly train the neural network weights and the hash table entries by applying Adam [Kingma and Ba 2014], where we set 𝛽1 = 0.9,𝛽2 = 0.99,𝜖 = 10−15, The choice of 𝛽1 and 𝛽2 makes only a small difference, but the small value of 𝜖 = 10−15 can significantly accelerate the convergence of the hash table entries when their gradients are sparse and weak.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra and Rematas, in which a hash grid defined for a target object is interpolated around each sampled location to obtain location features, a multilayer perceptron maps those features to image features for the location, and rays cast to the object's bounding box are volumetrically rendered into a feature image from which a displayed image is generated, and the hash grid parameters are learned jointly with the multilayer perceptron from a photometric loss between the color predicted along each ray and the ground truth color of images captured with a red, green, blue and depth camera, with the features of Müller's invention of initializing the hash table entries from a uniform distribution over a small interval about zero, and of training the encoding by backpropagating loss gradients through the multilayer perceptron, the concatenation, and the linear interpolation, and accumulating them in the looked-up feature vectors. The combination would have been obvious because Blomqvist attributes its hash grid encoding to Müller and states in section III.A that the grid parameters "are learned jointly with those of the MLP radiance field", but does not itself say what those parameters hold before training or by what path the training gradient reaches them, and Müller, the encoding Blomqvist adopts, supplies both. Müller explains in section 4 that the uniform initialization is chosen "to provide a small amount of randomness while encouraging initial predictions close to zero", and although Müller reports that "We also tried a variety of different distributions, including zero-initialization, which all resulted in a very slightly worse initial convergence speed.", the random initialization is the scheme Müller adopts and the one it reports as converging fastest. A person of ordinary skill implementing Blomqvist's jointly learned grid would have followed the initialization and the backpropagation path of the very encoding Blomqvist builds on, with the predictable result that the hash grid features start as random values and are updated by the loss backpropagated through the multilayer perceptron and the hash grid.
Claim(s) 9 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Blomqvist et al. “Baking in the Feature: Accelerating Volumetric Segmentation by Rendering Feature Maps”, 09/26/2022, arXiv:2209.12744v1 as modified by Abou-Chakra et al. “Implicit Object Mapping With Noisy Data”, 04/22/2022, arXiv2204:10516v1, Rematas et al. (U.S. Pub. No. 20230281913), and Muller et al. “Instant Neural Graphics Primitives with a Multiresolution Hash Encoding”, 05/04/2022, arXiv:2201.05989v2, further in view of Niemeyer et al. “GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields”, 11/24/2020, arXive:2011.12100v1.
Regarding claim 9, Blomqvist as modified by Abou-Chakra and Muller discloses the computer-implemented method of claim 7, wherein Blomqvist further comprising:generating the rendered image, the rendered image to a real image to generate an image loss (sec 3.B, “We define the same photometric loss Lrgb as in NeRF [1] and a depth loss Ld similar to the one used by [15]:”; also, sec 3.B, “where ¯ c(r) is the ground truth and ˆ c(r) the predicted color for ray r, ¯d(r) is the ground truth depth (if available) and ˆ d(r) the integrated depth predictions along ray r.”; also, sec 3, “We optimize the combined loss using stochastic gradient descent using Adam over rays r sampled from the images:”), wherein calculate the loss uses the image loss (sec 3.D, “We optimize the combined loss using stochastic gradient descent using Adam over rays r sampled from the images:”; also, sec 3.A, “We define the same photometric loss Lrgb as in NeRF [1] and a depth loss Ld similar to the one used by [15]:”). Blomqvist does not disclose executing a convolutional neural network (CNN) decoder on the feature image to generate the rendered image.
However, in a similar field of endeavor, Niemeyer discloses executing a convolutional neural network (CNN) decoder on the feature image to generate the rendered image (Fig. 4, “The feature image IV is processed by n blocks of nearest neighbor upsampling and 3×3 convolutions with leaky ReLU activations. At every resolution, we map the feature image to an RGB image with a 3×3 convolution and add it to the previous output via bilinear upsampling. We apply a sigmoid activation to obtain the final image ˆ I”; also, sec 3.3, “We then upsample the low resolution feature maps to higher-resolution RGB images using 2D neural rendering.”; also, sec 3.3, “We parameterize πneural θ as a 2D convolutional neural network (CNN) with leaky ReLU [55,88] activation (Fig. 4) and combine nearest neighbor upsampling with 3 × 3 convolutions to increase the spatial resolution.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra, Rematas, and Müller, in which a hash grid defined for a target object is interpolated around each sampled location to obtain location features, a multilayer perceptron maps those features to image features for the location, and rays cast to the object's bounding box are volumetrically rendered into a feature image from which a displayed image is generated, the hash grid features are initialized from a small uniform distribution, and the photometric loss between the color predicted along each ray and the ground truth color of the captured images is backpropagated through the multilayer perceptron and into the grid, with the features of Niemeyer's invention of a neural rendering operator, parameterized as a two-dimensional convolutional neural network, that maps the volume-rendered feature image to the final synthesized image by processing it through blocks of nearest neighbor upsampling and 3 × 3 convolutions. The combination would have been obvious because Blomqvist turns its rendered feature maps into a viewable image only by mapping the features to color with principal component analysis, a visualization that is not trained to reproduce the scene's appearance, while Niemeyer is directed to the same kind of volume-rendered feature image and supplies a learned convolutional decoder for exactly the step of converting it into a final image. Niemeyer states in section 3.3 that "We then upsample the low-resolution feature maps to higher-resolution RGB images using 2D neural rendering", and reports of that arrangement that "As evidenced by our experiments, this has two advantages: increased rendering speed and improved image quality." A person of ordinary skill seeking a realistic image from the combination's feature image, rather than a false-color view of it, would have adopted Niemeyer's decoder for those two stated advantages, and would have expected it to work because the combination's feature image, like Niemeyer's, is an accumulation along each ray of per-location features output by a multilayer perceptron. Because the decoder is the last stage before the image the combination displays, a person of ordinary skill would have taken the photometric comparison against the captured frame at the decoder's output rather than at the accumulated feature values, so that the same loss trains the decoder along with the multilayer perceptron and the grid. The predictable result is the combination's feature image passed through a convolutional neural network decoder that outputs the rendered image, which is then compared with the captured image to produce the image loss that the loss calculation uses.
Regarding claim 10, Blomqvist as modified by Abou-Chakra, Rematas, Müller, and Niemeyer discloses the computer-implemented method of claim 9, wherein Blomqvist further discloses comparing the rendered image to the real image comprises: detecting a plurality of differences between pixels of the rendered image with corresponding pixels of the real image (sec 3.B, “We define the same photometric loss Lrgb as in NeRF [1] and a depth loss Ld similar to the one used by [15]:”; also, sec 3, “where ¯ c(r) is the ground truth and ˆ c(r) the predicted color for ray r, ¯d(r) is the ground truth depth (if available) and ˆ d(r) the integrated depth predictions along ray r.”; also, sec 3.D, “When sampling examples for optimization, we select half the samples uniformly from all pixels.”); and calculating the image loss based on the plurality of differences (sec 3.B, “We define the same photometric loss Lrgb as in NeRF [1] and a depth loss Ld similar to the one used by [15]:”; also, sec 3.D, “We optimize the combined loss using stochastic gradient descent using Adam over rays r sampled from the images”; also, sec 3.D, “When sampling examples for optimization, we select half the samples uniformly from all pixels.”).
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Blomqvist et al. “Baking in the Feature: Accelerating Volumetric Segmentation by Rendering Feature Maps”, 09/26/2022, arXiv:2209.12744v1 as modified by Abou-Chakra et al. “Implicit Object Mapping With Noisy Data”, 04/22/2022, arXiv2204:10516v1 and Rematas et al. (U.S. Pub. No. 20230281913), further in view of Sitzmann et al. “Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations”, 01/28/2020, arXiv:1906.01618v2.
Regarding claim 12, Blomqvist as modified by Abou-Chakra and Rematas discloses the computer-implemented method of claim 1, further comprising: modeling each of a plurality of objects with corresponding latent code to generate a plurality of constraint vectors for the plurality of objects; training a hypernetwork to use the corresponding latent code in a constraint vector of the plurality of constraint vectors corresponding to the object; and jointly optimizing the plurality of constraint vectors across the plurality of objects to constrain a representation of the object.
However, in a similar field of endeavor, Sitzmann discloses modeling each of a plurality of objects with corresponding latent code to generate a plurality of constraint vectors for the plurality of objects (sec 3.3, “For this purpose, each object instance Cj is represented by its own latent code zj.”; also, sec 3.3, “To find the latent code vectors zj, we follow an auto-decoder frame work [33]”; also, sec 3.3, “We assume scenes of the same class have common shape and appearance properties that can be fully characterized by a set of latent variables z ∈ Rk, k < l.”); training a hypernetwork to use the corresponding latent code in a constraint vector of the plurality of constraint vectors corresponding to the object (sec 3.3, “This architecture was previously introduced as a Hypernetwork [70], a neural network that regresses the parameters of another neural network.”; also, sec 3.4, “To summarize, given a dataset D = {Cj}M j=1 of instance datasets C = {(Ii,Ei,Ki)}N i=1, we aim to find the parameters ψ of Ψ that maps latent vectors zj to the parameters of the respective scene representation φj, the parameters θ of the neural rendering function Θ, as well as the latent codes zj themselves.”; also, sec 3.3, “The zj are free variables and are optimized jointly with the parameters of the hypernetwork Ψ and the neural renderer Θ.”); and jointly optimizing the plurality of constraint vectors across the plurality of objects to constrain a representation of the object (sec 3.3, “The zj are free variables and are optimized jointly with the parameters of the hypernetwork Ψ and the neural renderer Θ.”; also, sec 3.4, “To summarize, given a dataset D = {Cj}M j=1 of instance datasets C = {(Ii,Ei,Ki)}N i=1, we aim to find the parameters ψ of Ψ that maps latent vectors zj to the parameters of the respective scene representation φj, the parameters θ of the neural rendering function Θ, as well as the latent codes zj themselves.”; also, sec 3.3, “Equivalently, this assumes that all parameters φj live in a k-dimensional subspace of Rl.”; also, sec 3.3, “We assume that the prior distribution over the zj is a zero-mean multivariate Gaussian with a diagonal covariance matrix.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra and Rematas, in which a hash grid defined for a target object is interpolated around each sampled location to obtain location features, a multilayer perceptron maps those features to image features for the location, and rays cast to the object's bounding box are volumetrically rendered into a feature image from which a displayed image is generated, with the features of Sitzmann's invention of representing each object instance by its own latent code found through an auto-decoder framework, a hypernetwork that maps each latent code to the parameters of that instance's scene representation, and latent codes that are optimized jointly with the parameters of the hypernetwork and the neural renderer across all instances of a class, so that every instance's parameters live in a shared low-dimensional subspace. The combination would have been obvious because the combination fits a separate field to each object from whatever views of that object happen to be captured, and Abou-Chakra, which supplies the per-object fields, states in section IV that it constructs "a single NeRF for every object j present in the scene" from whatever views of that object the captured sequence contains, while Sitzmann addresses the same limited-observation problem by generalizing, in its words in section 3.3, "from learning to represent a single scene to learning shape and appearance priors over several instances of a single class." Sitzmann states in section 3.4 that, once those parameters are found, "we may use the trained model for few-shot reconstruction of a new object instance" by fixing the hypernetwork and renderer and estimating only a new latent code for that instance. A person of ordinary skill would have applied Sitzmann's latent codes and hypernetwork to the combination's per-object representations so that each object's representation is constrained by what was learned from the others, with the predictable result of a plurality of per-object latent vectors, a hypernetwork trained to use each object's vector, and joint optimization of those vectors across the objects to constrain each object's representation.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Blomqvist et al. “Baking in the Feature: Accelerating Volumetric Segmentation by Rendering Feature Maps”, 09/26/2022, arXiv:2209.12744v1 as modified by Abou-Chakra et al. “Implicit Object Mapping With Noisy Data”, 04/22/2022, arXiv2204:10516v1 and Rematas et al. (U.S. Pub. No. 20230281913), further in view of Kundu et al. “Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation”, 05/09/2022, arXiv:2205.04334v1.
Regarding claim 13, Blomqvist as modified by Abou-Chakra and Rematas discloses the computer-implemented method of claim 1, further comprising: jointly optimizing a rotation and a translation component of a pose of the target object at each of a plurality of timesteps.
However, in a similar field of endeavor, Kundu discloses jointly optimizing a rotation and a translation component of a pose of the target object at each of a plurality of timesteps (sec 3.1, “The track is parameterized by a sequence of transformation matrices, one at each of at a set of discrete timestamps.”; also, sec 3.1, “For each timestamp, we create a rotation matrix R ∈ R3×3 and a translation vector t ∈ R3”; also, sec 3.1, “We optimize all parameters of the MLP and object track Tk jointly”; also, sec 3.1, “In order to optimize rotation, we orthogonalize R after each gradient descent step using SVD, which projects it back onto SO(3).”; also, sec 1, “We jointly optimize all (stuff and things) neural fields and object poses, allowing our method to cope with noisy object poses and image segmentations”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Blomqvist in view of Abou-Chakra and Rematas, in which a hash grid defined for a target object is interpolated around each sampled location to obtain location features, a multilayer perceptron maps those features to image features for the location, and rays cast to the object's bounding box are volumetrically rendered into a feature image from which a displayed image is generated, with the features of Kundu's invention of an object track parameterized by a rotation matrix and a translation vector at each of a set of discrete timestamps and optimized jointly with the parameters of the object's neural field. The combination would have been obvious because the combination places each object's grid according to a bounding box that Abou-Chakra, in section IV, describes as supplied by its mapping system and as one "that loosely contains the object", and Kundu identifies the consequence of relying on such boxes, stating in section 3.1 that "Optimizing object track parameters is important as initial boxes (even GT boxes) may be noisy." Kundu states in section 3.3 that "We jointly optimize all network parameters θ and object tracks �� to reproduce the observed RGB images and predicted 2D semantic images". A person of ordinary skill rendering a moving object from a per-object field positioned by imperfect boxes would have refined the object's pose as Kundu does, with the predictable result that the rotation and translation components of the target object's pose at each of a plurality of timesteps are jointly optimized.
Allowable Subject Matter
Claims 5, 8, 11, and 18 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Claim 8 recites that "the plurality of random values is used to initialize nonempty portions of the neural hash grid, and wherein empty portions of the neural hash grid are initialized with a predefined value." The claim therefore requires one grid whose portions are initialized in two different ways according to whether they are empty, with random values where the grid is nonempty and a single predefined value where it is empty. In the located art, the multiresolution hash encoding initializes every hash table entry by drawing it from the same small uniform distribution, so that empty and occupied space begin alike; a sparse hierarchical mapping approach that stores its features in hash tables randomly initializes the features at the corners of the octree cells it creates around observed points, but allocates no features at all in empty space, so that nothing there is initialized with any value; a hierarchical feature-grid mapping approach initializes its coarse and mid-level feature grids with random values, reports that random initialization gives slightly better convergence than initializing a grid from a fixed feature vector corresponding to free space, and zero-initializes its fine-level grid, so that it applies one treatment to an entire grid level and never selects between treatments according to whether a portion of the grid is empty; and an approach that represents the background and each object with separate multilayer perceptrons treats empty and non-empty regions differently at initialization, but does so by setting the bias of each network's density prediction layer to one constant for the mostly empty background volume and to another constant for the mostly non-empty object volumes, which acts on network outputs rather than on stored grid features. The difference matters because the claimed grid is a stored set of features that covers empty space as well as occupied space, and the claim gives the empty portions a known starting value while leaving the portions that must come to represent the object's geometry and appearance free to start from random values. The located art gives one of ordinary skill no reason to combine the two treatments within a single grid: the random-initialization art applies one distribution to every entry and reports that the other distributions it tried, zero-initialization among them, gave a very slightly worse initial convergence speed; the sparse-mapping art avoids the question by never storing empty space; the hierarchical-grid art applies a fixed free-space value, if at all, to a whole grid level and reports that random initialization of that level converged slightly better; and the bias-initialization art sets a fixed value for the non-empty object volumes as well as for the empty background and reports that its fixed-value initialization is effective and robust compared to random initialization, which points toward fixed values on both sides rather than random values on the non-empty side. Reaching the claim would require using the empty-versus-non-empty distinction of the output-bias art to choose, portion by portion within one stored grid, between the random values of the hash-encoding art and the free-space value that the hierarchical-grid art applies only to a whole grid level, a sequence that none of the located art suggests
Claim 11 recites, after the rendered image is generated, "partitioning, after generating the rendered image, the rendered image into a plurality of image patches" and "partitioning a real image into a plurality of real image patches," then "executing a discriminator model on the plurality of image patches and the plurality of real image patches to predict a plurality of classifications," and finally two losses divided by whether those classifications were right: "generating a discriminator loss based on the plurality of classifications that are inaccurate" and "generating a CNN loss based on the plurality of classifications that are accurate." The claim therefore requires that each of the two images be divided into a set of patches, that the discriminator return a classification per patch, and that the correctness of each classification decide which of the two losses it feeds.
The located art does not reach that arrangement. The patch-based image translation art runs a single discriminator convolutionally across an image and averages all of its responses into one output, which slides one receptive field over the image rather than dividing the image into a set of patches, and which performs no separate partitioning act on a real image. The generative radiance field art that trains on patches draws one randomly placed and randomly scaled patch per training step from the generator and a matching patch from a real image, which samples the image domain rather than partitioning either image. The vision transformer generative art does partition an image into non-overlapping patches on a predefined grid, but it does so to form the token sequence its networks consume, not by dividing an already generated rendered image after that image is generated, and its discriminator derives a single score from a classification embedding rather than a classification for each patch. The convolutional volume rendering art divides its input images into patches and produces its own output patch by patch, which is the reverse of partitioning an output image once the image exists. None of this art supplies a discriminator that returns a plurality of per-patch classifications for a partitioned rendered image and a partitioned real image alike.
The losses are the further difference. In every located reference the generator and the discriminator descend a single undifferentiated objective, whether the non-saturating minimax objective with gradient penalty used by the generative radiance field art or the least-squares variant used by the convolutional volume rendering art. No located reference conditions one loss on the classifications the discriminator got right and the other on the classifications it got wrong, and none divides its training signal by classification correctness at all. Reaching the claim would require taking the patch tokenization of the transformer art, applying it to the output of the decoder after that output is rendered rather than to the network input, obtaining a per-patch classification that none of the located discriminators emits, and then splitting a single joint objective into two losses according to which classifications were correct, a sequence that none of the located art suggests.
Claims 5 and 18 recite "executing a LiDAR decoder using an object appearance generated from the neural hash grid to generate a LiDAR point cloud." The specification describes the LiDAR decoder at paragraph [0067] as "a neural network that is configured to generate a LiDAR point based on the output of the shared MLP model (342)," whose points each carry "a pair of depth and intensity that is accumulated along a LiDAR ray," and states that "A decoder MLP model of the LiDAR decoder (350) takes, as input, the accumulated volume rendered features and outputs the intensity." The claims therefore require three things together: a decoder that is itself a trained network, an input to that decoder that is the appearance the model produces from the grid rather than geometry alone, and a LiDAR point cloud as the output.
The located art supplies the point cloud but not the other two. The radiance field art that incorporates lidar casts rays from the lidar origins and directions through the trained field, estimates each ray's termination distance by sampling the field's volumetric density function and accumulating its transmittance, and then obtains the point arithmetically from the ray origin, the ray direction and that distance. That is a closed-form integration of geometry, performed without any decoder, and it consumes the density the field predicts rather than the appearance the model generates; the same art computes no intensity at all. The lidar simulation art does apply a machine-learned model in the course of producing a lidar point cloud, but that model predicts a dropout probability for points in an already completed ray-cast cloud and so removes points rather than decoding them into existence, and the channels it consumes are recorded sensor measurements such as range, recorded intensity and incidence angle rather than anything a learned representation generated; that art obtains intensity by taking the average of the recorded intensities of the nearest neighbors, which is a lookup into captured data rather than a decoding of a model output. The implicit mapping art that operates on lidar decodes its features to a signed distance or an occupancy value and emits a surface mesh, and treats lidar point clouds as the input it is built from rather than as something it generates.
The difference matters because the claims place the decoder downstream of the very quantity the rest of the claim builds, namely the per-location output that the multilayer perceptron generates from the interpolated grid features, and require that output to determine the lidar point rather than merely the geometry along the ray. Reaching the claims would require taking the lidar ray casting of the radiance field art, replacing its closed-form integration with a trained decoder that the art does not use there, and feeding that decoder the appearance output of the model rather than the recorded sensor channels that the lidar simulation art feeds its network, a sequence that none of the located art suggests.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jai Li whose telephone number is (571)272-1170. The examiner can normally be reached Mon-Thu between 06:00-16:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAI W LI/Junior Patent Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
