DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s submission filed 11/25/2025 has been entered. The claims 1-3, 8-11 and 14-16 have been amended. The claims 1-20 are pending in the current application.
Response to Arguments
Applicant's arguments filed 11/21/2025 have been fully considered but they are not persuasive.
In other words, Xu teaches the new claim limitation of instructions to apply a fine model to extract fine features at a fine granularity and a coarse model to extract coarse features at a coarse granularity that are refined into refined features, and combine the refined features with the fine features to form the appearance features (
Xu teaches at Paragraph 0033 that the UV map-aligned pose-image features 130 are extracted from the driving views 110.
Xu teaches at Paragraph 0026 that the concatenator 128, which may be configured as a neural texture model, receives as input the UV map-aligned image features 118 (fine features) and the UV map-aligned pose features 126 (coarse features or rough features), concatenates the UV map-aligned image features 118 and the UV map-aligned pose features 126, and generates and outputs a plurality of UV map-aligned pose-image features 130, or concatenated pose-image feature vectors, based on the UV map-aligned image features 118 and the UV map-aligned pose features 126. Xu teaches at Paragraph 0026 that the UV map-aligned image features 118 (fine features) may capture texture information where details such as wrinkles in the clothing can be realistically rendered and the UV map-aligned pose features (coarse features) capture information sufficient to render the general shape of clothing worn by the user (coarse features).
Xu teaches at Paragraph 0031 that these three-dimensional volumetric features encoded as the volumetric features 144 of PID-NeRF in 3D space are rasterized (refined) into rasterized multi-channel volumetric features 148 in image space, and each pixel is predicted by accumulating features of consecutive samples along a corresponding ray by a volumetric renderer 146 implementing a neural network. The first three channels of the rasterized multi-channel volumetric features 148 may be RGB, which may be supervised by downsampled ground truth images 162, using volume rendering loss 164. Alternatively, other color encoding may be used, such as CMYK, etc.
Xu teaches at Paragraph 0033 that the 2D textural features 138 and the rasterized multi-channel volumetric features 148 are fused together by an attentional volumetric textural feature fuser 140, which may include a convolutional neural network configured to downsample the textural features 138 (fine features) to the same size as the rasterized multi-channel volumetric features 148 (rough features or coarse features).
It is understood that the fine features are downsampled to be the same size as the coarse features.
Xu teaches at Paragraph 0045 The above-described system and methods use an effective scheme to encode UV map-aligned pose-image features, and leverages these to learn a pose- and image-conditioned downsampled NeRF (PID-NeRF) from low resolution images….extracting 2D textural features to achieve efficient, high quality, geometry-aware neural rendering of human avatars. The UV map-aligned encoder may accept arbitrary driving views as input, which may be leveraged for faithful rendering, and a texture loss may enforce full texture completion. Such faithful rendering may be used for improving rendering quality in telepresence applications.
Xu teaches at Paragraph 0060 generating a coarse human mesh representing the user based on a template mesh and the skeletal pose of the user; constructing a UV positional map based on the coarse human mesh; constructing a texture map based on the one or more driving views and the coarse human mesh; extracting a plurality of image features from the texture map, the image features being aligned to the UV positional map; extracting a plurality of pose features from the UV positional map, the pose features being aligned to the UV positional map; generating a plurality of pose-image features based on the UV map-aligned image features and the UV map-aligned pose features; and rendering an avatar based on the plurality of pose-image features).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. US-PGPUB No. 2024/0096041 (hereinafter Xu) in view of
Kumar et al. US-PGPUB No. 2025/0157179 (hereinafter Kumar);
Kuo et al. US-PGPUB No. 2025/0157148 (hereinafter Kuo);
Deng et al. US-PGPUB No. 2025/0285356 (hereinafter Deng);
Kum et al. US-PGPUB No. 2025/0037444 (hereinafter Kum);
Kheradmand et al. US-Patent No. 12,354,337 (hereinafter Kheradmand).
Re Claim 1:
Xu teaches a pose system, comprising:
one or more processors (Xu teaches at Paragraph 0016 the processor 28);
a memory communicably coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to (Xu teaches at Paragraph 0015 that the processor is configured to execute instructions stored in the memory 30):
acquire target information and sensor data of a surrounding environment that includes a person, the target information defining a target space that includes a target pose and a target camera view (Xu teaches at Paragraph 0014 that an avatar generation program 16 receives the skeletal pose 102 and the one or more driving views 110 and at FIG. 1 acquiring the camera images of real person 5 by the cameras 24A-24B);
extract appearance features of the person from the sensor data; including instructions to apply a fine model to extract fine features at a fine granularity and a coarse model to extract coarse features at a coarse granularity that are refined into refined features, and combine the refined features with the fine features to form the appearance features (Xu teaches at Paragraph 0033 that the UV map-aligned pose-image features 130 are extracted from the driving views 110.
Xu teaches at Paragraph 0026 that the concatenator 128, which may be configured as a neural texture model, receives as input the UV map-aligned image features 118 (fine features) and the UV map-aligned pose features 126 (coarse features or rough features), concatenates the UV map-aligned image features 118 and the UV map-aligned pose features 126, and generates and outputs a plurality of UV map-aligned pose-image features 130, or concatenated pose-image feature vectors, based on the UV map-aligned image features 118 and the UV map-aligned pose features 126. Xu teaches at Paragraph 0026 that the UV map-aligned image features 118 may capture texture information where details such as wrinkles in the clothing can be realistically rendered and the UV map-aligned pose features capture information sufficient to render the general shape of clothing worn by the user (coarse features).
Xu teaches at Paragraph 0031 that these three-dimensional volumetric features encoded as the volumetric features 144 of PID-NeRF in 3D space are rasterized (refined) into rasterized multi-channel volumetric features 148 (Ψ.sub.vol.sup.im) in image space, and each pixel is predicted by accumulating features of consecutive samples along a corresponding ray by a volumetric renderer 146 implementing a neural network. The first three channels of the rasterized multi-channel volumetric features 148 may be RGB, which may be supervised by downsampled ground truth images 162, using volume rendering loss 164. Alternatively, other color encoding may be used, such as CMYK, etc.
Xu teaches at Paragraph 0033 that the 2D textural features 138 and the rasterized multi-channel volumetric features 148 are fused together by an attentional volumetric textural feature fuser 140, which may include a convolutional neural network configured to downsample the textural features 138 (fine features) to the same size as the rasterized multi-channel volumetric features 148 (rough features or coarse features).
Xu teaches at Paragraph 0045 The above-described system and methods use an effective scheme to encode UV map-aligned pose-image features, and leverages these to learn a pose- and image-conditioned downsampled NeRF (PID-NeRF) from low resolution images….extracting 2D textural features to achieve efficient, high quality, geometry-aware neural rendering of human avatars. The UV map-aligned encoder may accept arbitrary driving views as input, which may be leveraged for faithful rendering, and a texture loss may enforce full texture completion. Such faithful rendering may be used for improving rendering quality in telepresence applications.
Xu teaches at Paragraph 0060 generating a coarse human mesh representing the user based on a template mesh and the skeletal pose of the user; constructing a UV positional map based on the coarse human mesh; constructing a texture map based on the one or more driving views and the coarse human mesh; extracting a plurality of image features from the texture map, the image features being aligned to the UV positional map; extracting a plurality of pose features from the UV positional map, the pose features being aligned to the UV positional map; generating a plurality of pose-image features based on the UV map-aligned image features and the UV map-aligned pose features; and rendering an avatar based on the plurality of pose-image features).
Xu at least suggests the claim limitation:
map the appearance features into the target space, including aggregating the appearance features into an aggregated feature map (
Xu teaches at Paragraph 0026 that the concatenator 128, which may be configured as a neural texture model, receives as input the UV map-aligned image features 118 and the UV map-aligned pose features 126, concatenates the UV map-aligned image features 118 and the UV map-aligned pose features 126, and generates and outputs a plurality of UV map-aligned pose-image features 130, or concatenated pose-image feature vectors, based on the UV map-aligned image features 118 and the UV map-aligned pose features 126.
Xu teaches at Paragraph 0039 that the UV map-aligned pose features and the UV map-aligned image features are concatenated into UV map-aligned pose-image features and the UV map-aligned pose-image features are transformed from UV space to image space to extract or generate and output image space-aligned pose-image features and at Paragraph 0040 that a 2D feature map is generated based on the UV map-aligned pose-image features. At step 225, the 2D feature map is regressed using a neural radiance field to generate features in 3D space. At step 226, the features in 3D space are rasterized into multi-channel volumetric features. Xu teaches at Paragraph 0041 that the high-dimensional textural features and the multi-channel volumetric features are fused together via attentional volumetric textural feature fusion.
Xu teaches at Paragraph 0059 that the processor is further configured to generate textural features and multi-channel volumetric features from the pose-image features, channels of the multi-channel volumetric features corresponding to color channels of one or a plurality of pixel images of the one or more driving views; and fuse the textural features and the multi-channel volumetric features to render the avatar);
render the target camera view of the person in the target pose according to the aggregated feature map; and provide the target camera view (
Xu teaches at Paragraph 0026 that the concatenator 128, which may be configured as a neural texture model, receives as input the UV map-aligned image features 118 and the UV map-aligned pose features 126, concatenates the UV map-aligned image features 118 and the UV map-aligned pose features 126, and generates and outputs a plurality of UV map-aligned pose-image features 130, or concatenated pose-image feature vectors, based on the UV map-aligned image features 118 and the UV map-aligned pose features 126.
Xu teaches at Paragraph [0029] The UV map-aligned pose-image features 130 are then transformed from UV space to image space by a feature renderer 132, which outputs image space-aligned features 134. The image space-aligned features 134 are further regressed by a texture encoder 136 implementing a two-dimensional convolutional neural network to output high-dimensional textural features 138 in image space.
Xu teaches at Paragraph 0060 generating a coarse human mesh representing the user based on a template mesh and the skeletal pose of the user; constructing a UV positional map based on the coarse human mesh; constructing a texture map based on the one or more driving views and the coarse human mesh; extracting a plurality of image features from the texture map, the image features being aligned to the UV positional map; extracting a plurality of pose features from the UV positional map, the pose features being aligned to the UV positional map; generating a plurality of pose-image features based on the UV map-aligned image features and the UV map-aligned pose features; and rendering an avatar based on the plurality of pose-image features.
Xu teaches at Paragraph 0041 that the high-dimensional textural features and the multi-channel volumetric features are fused together via attentional volumetric textural feature fusion. At step 230, the fused features are converted into a target avatar via a textural renderer 230. At step 232, the target avatar is rendered and displayed on the display).
Kuo teaches a pose system, comprising:
one or more processors;
a memory communicably coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to (Kuo teaches at FIG. 4 and Paragraph 0041 that computer instructions stored on one or more computer readable storage medium that, when executed by one or more processors, perform the recited operations):
acquire target information and sensor data of a surrounding environment that includes a person, the target information defining a target space that includes a target pose and a target camera view (Kuo teaches at Paragraph 0023 that a highly detailed 3D facial model (not shown) can be generated using expensive camera equipment that captures an individual's face from multiple angles and at Paragraph 0028 that the images may provide views from different angles around the object and at Paragraph 0030 the mean face 204 (target pose) may be a 3D morphable model (3DMM) that represents the geometry of a default or generic head/face and at Paragraph 0032 that 0032] The predictor 212 may use the UV aligned feature map 202 and label map 210 to align vertices of the mean face position map 208 to corresponding pixels of the UV aligned feature map 202. Kuo teaches at Paragraph 0026 that correspondences between features of the UV aligned feature map and the mean face position map may be determined to generate displacement values which may indicate how far and in what direction a particular vertex of the mean face position map should be moved to align the vertex with a pixel of the corresponding feature…to generate a fine position map---a target pose.
Kuo teaches at Paragraph 0035 that the fine position map 314 (pose map) may be iteratively refined and may describe vertices of a fine face mesh projected into the 2D UV space and may be reprojected into 3D space to obtain a topological fine face mesh of the face captured by the images);
extract appearance features of the person from the sensor data, including instructions to apply a fine model to extract fine features at a fine granularity and a coarse model to extract coarse features at a coarse granularity that are refined into refined features, and combine the refined features with the fine features to form the appearance features (Kuo teaches at Paragraph [0026] Features of a set of images of an object, such as a user's head, may be detected from a set of images taken of the object. Feature maps for the detected features may be aligned on a common axis, such as the UV axis. The feature maps for the set of images may be merged (e.g., fused) into a single UV aligned feature map and at Paragraph 0028 that images of the set of images 102 may provide multiple views of the object from different angles and may be passed into a set of features extractors 104A-104N for obtaining features from the images of the set of images 102.
Kuo teaches at Paragraph 0029 that the multi-view feature fusion engine 108 may use features of the feature maps to align the multiple feature maps on common axes, such as a UV axis, to generate a highly detailed, UV aligned feature map 110 (fine features). The multi-view feature fusion engine 108 may also align textures of the images of the set of images 102 based on the alignment of the multiple feature maps to generate a texture map for the face.
Kuo teaches at Paragraph 0030 that UV aligned feature map 110 of FIG. 1, may be combined with a mean face 204(coarse features) to generate an intermediate residual position map 206.
Kuo teaches at Paragraph 0030 that the mean face 204 may be a topological mesh and the mean face 204 may be sampled into the UV coordinate system such that each 3D vertex of the mean face 204 (coarse features) is flattened into a 2D UV space to generate a mean face position map 208.
Kuo teaches at Paragraph 0032 that the predictor 212 may determine correspondences between certain detected features in the UV aligned feature map 202 with certain vertices of the mean face position map 208.
Kuo teaches at Paragraph [0035] In some cases, the fine residual position map 306 may be applied to the intermediate position map 308 to obtain a fine position map 314. In some cases, the fine position map 314 may be further iteratively refined in a manner similar to the intermediate position map 308. In other cases, the fine position map 314 (coarse features) may be output along with a texture map 316 (fine feature). In some cases, the texture map 316 may be the texture map discussed above with respect to FIG. 1. The fine position map 314 may describe vertices of a fine face mesh projected into the 2D UV space. The fine position map 314 may be reprojected into 3D space to obtain a topological fine face mesh of the face captured by the images (e.g., of the set of images 102 of FIG. 1). The texture map 316 may then be applied to the fine face mesh to generate a representation of the face captured by the images);
map the appearance features into the target space, including aggregating the appearance features into an aggregated feature map (
Kuo teaches at Paragraph 0035 that the fine position map 314 (pose map) may be iteratively refined and may describe vertices of a fine face mesh projected into the 2D UV space and may be reprojected into 3D space to obtain a topological fine face mesh of the face captured by the images.
Kuo teaches at Paragraph [0029] The multi-view feature fusion engine 108 may align and fuse the multi-view feature maps 106 to generate a UV aligned feature map 110. For example, the multi-view feature fusion engine 108 may use features of the feature maps to align the multiple feature maps on common axes, such as a UV axis, to generate a highly detailed, UV aligned feature map 110. The UV aligned feature map 110 may represent features from the images of the set of images 102, allowing points (e.g., pixels) of the UV aligned feature map 110 to be super sampled. In some examples, multiple features for a particular pixel of the UV aligned feature map 110 may be fused into a single feature. In some cases, the UV aligned feature map 110 may include features from all of the images of the set of images 102. In some cases, the multi-view feature fusion engine 108 may also align textures of the images of the set of images 102 based on the alignment of the multiple feature maps to generate a texture map for the face.
Kuo teaches at Paragraph [0039] At block 406, the computing device (or component thereof) may fuse the plurality of feature maps (e.g., by a multi-view feature fusion engine 108 of FIG. 1) based on features of the plurality of feature maps along a common axis to generate an aligned feature map. The computing device (or component thereof) may align textures of the plurality of images based on the fusing of the plurality of feature maps to generate a texture map; and apply the texture map to the fine face mesh to obtain a representation of the face);
render the target camera view of the person in the target pose according to the aggregated feature map; and provide the target camera view (
(Kuo teaches at Paragraph 0035 that the fine position map 314 (pose map) may be iteratively refined and may describe vertices of a fine face mesh projected into the 2D UV space and may be reprojected into 3D space to obtain a topological fine face mesh of the face captured by the images.
Kuo teaches at Paragraph [0039] At block 406, the computing device (or component thereof) may fuse the plurality of feature maps (e.g., by a multi-view feature fusion engine 108 of FIG. 1) based on features of the plurality of feature maps along a common axis to generate an aligned feature map. The computing device (or component thereof) may align textures of the plurality of images based on the fusing of the plurality of feature maps to generate a texture map; and apply the texture map to the fine face mesh to obtain a representation of the face.
Kuo teaches at Paragraph [0029] The multi-view feature fusion engine 108 may align and fuse the multi-view feature maps 106 to generate a UV aligned feature map 110. For example, the multi-view feature fusion engine 108 may use features of the feature maps to align the multiple feature maps on common axes, such as a UV axis, to generate a highly detailed, UV aligned feature map 110. The UV aligned feature map 110 may represent features from the images of the set of images 102, allowing points (e.g., pixels) of the UV aligned feature map 110 to be super sampled. In some examples, multiple features for a particular pixel of the UV aligned feature map 110 may be fused into a single feature. In some cases, the UV aligned feature map 110 may include features from all of the images of the set of images 102. In some cases, the multi-view feature fusion engine 108 may also align textures of the images of the set of images 102 based on the alignment of the multiple feature maps to generate a texture map for the face).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Kuo’s teaching of combining/fusing the appearance features from the various enrollment images of the captured person into Xu’s system of fusing the appearance features based on the captured images of a person to have synthesized an person image based on the target pose. One of the ordinary skill in the art would have been motivated to have provided a target camera view of the person in the target viewing angle (the target pose) based on the captured images of the person from various viewing angles.
Deng teaches a pose system, comprising:
one or more processors;
a memory communicably coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to:
acquire target information and sensor data of a surrounding environment that includes a person, the target information defining a target space that includes a target pose and a target camera view (Deng teaches at FIG. 3 and Paragraph 0043 acquiring a target viewing angle 308 and at Paragraph 0044 capturing the enrollment images 102 of a person for different viewing angles);
extract appearance features of the person from the sensor data (
Deng teaches at Paragraph [0054] that Appearance-code extractor 408 may generate image features 432 based on each of enrollment images 404 and a corresponding one of image masks 406. For example, appearance-code extractor 408 may include an encoder 410 to encode an enrollment image 404 (fine features) along with a corresponding image mask 406 (coarse features) to generate image features 432.
Deng teaches at Paragraph [0058] Combiner 416 may combine image features 432 from various enrollment images 404 based on selected mask 414 to generate combined appearance features 430. Combiner 416 may be, or may include, a machine-learning model trained to generate combined appearance features based on image features and masks);
map the appearance features into the target space, including aggregating the appearance features into an aggregated feature map (
Deng teaches at Paragraph 0044 that system 300 may select enrollment images 102 from among the larger set of images based on the viewing angle from which enrollment images 102 were captured being similar to target viewing angle 308.
Deng teaches at Paragraph [0047] Target viewing angle 308 may be a viewing angle from which synthesized image 108 is to be rendered. For example, target viewing angle 308 may include coordinates (e.g., relative coordinate) describing a location relative to the head of the person of enrollment images 102 from which synthesized image 108 should appear to be captured..
Deng teaches at Paragraph 0063 that transformer 506 may transform appearance code 504 determined based on enrollment data 402 into appearance texture 508 that may be relevant to target viewing angle 308.
Deng teaches at Paragraph [0079] At block 1004, the computing device (or one or more components thereof) may determine, from the selected enrollment images, appearance features representing at least one characteristic of the subject in the selected enrollment images. For example, appearance-code extractor 408 of FIG. 4 may generate image features 432 representing characteristics of the subject.
Deng teaches at Paragraph [0083] At block 1006, the computing device (or one or more components thereof) may combine the appearance features based on a three-dimensional geometry of the subject to generate combined appearance features. For example, combiner 416 of combine image features 432 to generate combined appearance features 430 based on a three-dimensional geometry of the subject.
Deng teaches at Paragraph [0084] In some aspects, to combine the appearance features, the computing device (or one or more components thereof) may process the appearance features using a machine-learning model to generate the combined appearance features. For example, system 400 may use combiner 416, which may be, or may include, a machine-learning model, to generate combined appearance features 430.
Deng teaches at Paragraph [0085] At block 1008, the computing device (or one or more components thereof) may generate an image of the subject from the target viewing angle based on a texture image of the subject (fine features) and the combined appearance features (coarse features). For example, image synthesizer 418 of FIG. 4 may generate synthesized image 108 based on combined appearance features 430 and texture image 422.
Deng teaches at Paragraph [0057] Additionally or alternatively, selective mask 412 may generate selected masks 414 to indicate which portions of enrollment images 404 are relevant to synthesized image 108. For example, selected masks 414 may indicate which pixels of enrollment image 404 to use to generate features based on hair as the hair is viewed from target viewing angle 308. Examples of selective masks are illustrated with regard to FIG. 6 and FIG. 7.
Deng teaches at Paragraph [0058] Combiner 416 may combine image features 432 from various enrollment images 404 based on selected mask 414 to generate combined appearance features 430. Combiner 416 may be, or may include, a machine-learning model trained to generate combined appearance features based on image features and masks.
Deng teaches at Paragraph [0059] Image synthesizer 418 may generate synthesized image 108 based on hair mask 420, texture image 422, hair normal 424, and combined appearance features 430. For example, image synthesizer 418 may synthesize texture image 422 and combined appearance features 430 to generate synthesized image 108);
render the target camera view of the person in the target pose according to the aggregated feature map; and provide the target camera view (
Deng teaches at Paragraph 0044 that system 300 may select enrollment images 102 from among the larger set of images based on the viewing angle from which enrollment images 102 were captured being similar to target viewing angle 308.
Deng teaches at Paragraph [0047] Target viewing angle 308 may be a viewing angle from which synthesized image 108 is to be rendered. For example, target viewing angle 308 may include coordinates (e.g., relative coordinate) describing a location relative to the head of the person of enrollment images 102 from which synthesized image 108 should appear to be captured..
Deng teaches at Paragraph 0063 that transformer 506 may transform appearance code 504 determined based on enrollment data 402 into appearance texture 508 that may be relevant to target viewing angle 308.
Deng teaches at Paragraph [0079] At block 1004, the computing device (or one or more components thereof) may determine, from the selected enrollment images, appearance features representing at least one characteristic of the subject in the selected enrollment images. For example, appearance-code extractor 408 of FIG. 4 may generate image features 432 representing characteristics of the subject.
Deng teaches at Paragraph [0083] At block 1006, the computing device (or one or more components thereof) may combine the appearance features based on a three-dimensional geometry of the subject to generate combined appearance features. For example, combiner 416 of combine image features 432 to generate combined appearance features 430 based on a three-dimensional geometry of the subject.
Deng teaches at Paragraph [0084] In some aspects, to combine the appearance features, the computing device (or one or more components thereof) may process the appearance features using a machine-learning model to generate the combined appearance features. For example, system 400 may use combiner 416, which may be, or may include, a machine-learning model, to generate combined appearance features 430.
Deng teaches at Paragraph [0085] At block 1008, the computing device (or one or more components thereof) may generate an image of the subject from the target viewing angle based on a texture image of the subject and the combined appearance features. For example, image synthesizer 418 of FIG. 4 may generate synthesized image 108 based on combined appearance features 430 and texture image 422.
Deng teaches at Paragraph [0059] Image synthesizer 418 may generate synthesized image 108 based on hair mask 420, texture image 422, hair normal 424, and combined appearance features 430. For example, image synthesizer 418 may synthesize texture image 422 and combined appearance features 430 to generate synthesized image 108).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Deng’s teaching of combining/fusing the appearance features from the various enrollment images of the captured person into Xu’s system of fusing the appearance features based on the captured images of a person to have synthesized an person image based on the target pose. One of the ordinary skill in the art would have been motivated to have provided a target camera view of the person in the target viewing angle (the target pose) based on the captured images of the person from various viewing angles.
Kumar/Kum teaches a pose system, comprising:
one or more processors (Kum teaches at Paragraph 0048-0049 that the processor 120 may be configured to process instructions stored in the memory 110);
a memory communicably coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to (Kum teaches at Paragraph 0048-0049 that the processor 120 may be configured to process instructions stored in the memory 110):
acquire target information and sensor data of a surrounding environment that includes a person, the target information defining a target space that includes a target pose and a target camera view (
Kumar teaches at Paragraph 0063 that encoder-decoder architecture 200 to transform perspective view features 206 into BEV features that represent the one or more objects within the 3D environment on a grid structure from a perspective looking down at the one or more objects from a position above the one or more objects. Since encoder-decoder architecture 200 may be part of an ADAS for controlling a vehicle, and since vehicles move generally across the ground in a way that is observable from a bird's eye perspective, generating BEV features (e.g., fused features from multiple cameras) may allow a control unit (e.g., control unit 142 and/or control unit 196) of FIG. 1 to control the vehicle based on the representation of the one or more objects from a bird's eye perspective.
Kumar teaches at Paragraph 0028 that cameras 104 may be any type of camera configured to capture video or image data in the environment and at Paragraph 0043 that view synthesis unit 140 may be configured to extract features from a respective image from each camera of a plurality of cameras. Kumar teaches at Paragraph [0082] Decoder 242 may be configured to generate a first output 246 based on the fused set of BEV features in fused image 172. The first output 246 may comprise a 2D BEV representation of the 3D environment corresponding to processing system 100. For example, when processing system 100 is part of an ADAS for controlling a vehicle, the first output 246 may indicate a BEV view of one or more roads, road signs, road markers, traffic lights, vehicles, pedestrians, and other objects within the 3D environment corresponding to processing system 100. This may allow processing system 100 to use the first output 246 to control the vehicle within the 3D environment.
Kum teaches at Paragraph 0065-0066 that the processor 120 may extract a camera feature map from data of a camera and may extract a radar feature map from data of a radar sensor);
extract appearance features of the person from the sensor data (Kum teaches at Paragraph 0065-0066 that the processor 120 may extract a camera feature map from data of a camera and may extract a radar feature map from data of a radar sensor. Kumar teaches at Paragraph 0086 that [0086] Processing system 100 may be configured to extract features from a respective image from each camera of a plurality of cameras.
Kumar teaches at Paragraph 0068 that projection unit 208 may be further configured to aggregate the features from each of the respective images to each respective cell of the fused image to generate aggregated features and at Paragraph 0062 that perspective view features 206 may include key points that are matched across a group of two or more camera images of camera images 202. Key points may allow encoder-decoder architecture 200 to determine one or more characteristics of motion and pose of objects. Perspective view features 206 may, in some examples, include depth-based features that indicate a distance of one or more objects from the camera, but this is not required. Perspective view features 206 may include any one or combination of image features that indicate characteristics of camera images 202.);
map the appearance features into the target space, including aggregating the appearance features into an aggregated feature map (
Kumar teaches at Paragraph 0054 that camera images 202 may represent one or more perspective views of one or more objects within a 3D space where processing system 100 is located. That is, the one or more perspective views may represent views from the perspective of processing system 100.
Kumar teaches at Paragraph [0089] To fuse the features (502), processing system 100 may be further configured to aggregate, based on the contribution to the respective cell and a respective set of learnable parameters for each cell, the features from each of the respective images to each respective cell of the fused image to generate aggregated features (506).
Kumar teaches at Paragraph 0063 that encoder-decoder architecture 200 to transform perspective view features 206 into BEV features that represent the one or more objects within the 3D environment on a grid structure from a perspective looking down at the one or more objects from a position above the one or more objects. Since encoder-decoder architecture 200 may be part of an ADAS for controlling a vehicle, and since vehicles move generally across the ground in a way that is observable from a bird's eye perspective, generating BEV features (e.g., fused features from multiple cameras) may allow a control unit (e.g., control unit 142 and/or control unit 196) of FIG. 1 to control the vehicle based on the representation of the one or more objects from a bird's eye perspective.
Kum teaches at Paragraph 0068 that the processor 120 may extract a feature map in a 3D space by fusing spatial and informative characteristics of data that are provided by the sensors and at Paragraph 0070 that an image feature map and a radar feature map may be fused and at Paragraph 0092 that the processor 120 may transform the image feature map into a BEV representation by concatenating the first frustum view feature map and the second frustum view feature map and at Paragraph 0100 that the camera feature map and the radar feature map of a BEV representation that are obtained each time may be fused and the final BEV feature map can be improved by fusing the BEV feature maps);
render the target camera view of the person in the target pose according to the aggregated feature map; and provide the target camera view (
Kumar teaches at Paragraph [0064] Projection unit 208 may transform perspective view features 206 into fused features in fused image 172. Such a transformation may be referred to as a PV-to-BEV projection and Paragraph [0087] Processing system 100 may fuse the features into a fused image having a grid structure (502). The fused image may be any type of image having a mesh or grid structure that may be reconstructed or synthesized from a plurality of different cameras.
Kum teaches at Paragraph [0101] FIG. 6 illustrates an example of a camera-radar fusion feature map, a camera feature map, and a radar feature map in an embodiment of the present disclosure. A 3-D feature map (i.e., the feature map in FIG. 6(A)) obtained through the camera-radar fusion shows that an environment in which diffused reflection that is difficult to detect when radar is solely used occurs or an object having low reflexibility like a person can be well detected, and a vehicle out of a front vehicle thereof, which is rarely seen in a camera image, or an object at a very long distance can also be well detected.
Kum teaches at Paragraph [0069] In step S240, the processor 120 may perform an element technology necessary to perceive an autonomous driving environment by using the 3-D feature map that has been obtained through the fusion of the data of the camera and the radar sensor. The element technology necessary to perceive the autonomous driving environment may include 3-D object detection, 3-D object tracking, drivable area segmentation, and lane segmentation. The entire system can efficiently operate because various element technologies can operate by attaching a detector specified for each element technology to the 3-D feature map including information of a vehicle driving environment.).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Kumar/Kum’s teaching of fusing the camera image feature map and the radar feature map into Xu’s system of fusing the appearance features based on the captured images of a person to have synthesized an person image based on the target pose based on the feature maps collected from different sensors. One of the ordinary skill in the art would have been motivated to have provided a person image in the target pose based on the captured images from different sensors.
Kheradmand teaches a pose system, comprising:
one or more processors;
a memory communicably coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to:
acquire target information and sensor data of a surrounding environment that includes a person, the target information defining a target space that includes a target pose and a target camera view (Kheradmand teaches at FIGS. 4-5 receiving a plurality of input images 540A-C of the surrounding environment of the person and receiving a target pose 544);
extract appearance features of the person from the sensor data (Kheradmand teaches at column 5, lines 30-55 that the machine learning models may include a second machine learning model that generates a first appearance feature based on the first image and the pose data comprising the target pose and a second appearance feature based on the second image and the pose data comprising the target pose);
map the appearance features into the target space, including aggregating the appearance features into an aggregated feature map (Kheradmand teaches at column 5, lines 35-66 that the first appearance feature and the second appearance feature can be combined into a combined appearance feature and a third machine learning model generates the output image based on the pose feature and a combined appearance feature);
render the target camera view of the person in the target pose according to the aggregated feature map; and provide the target camera view (Kheradmand teaches at column 5, lines 35-66 that the first appearance feature and the second appearance feature can be combined into a combined appearance feature and a third machine learning model generates the output image based on the pose feature and a combined appearance feature).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Kheradmand’s teaching of combining/fusing the appearance features from the various images of the captured person into Xu’s system of fusing the appearance features based on the captured images of a person to have synthesized an person image based on the target pose. One of the ordinary skill in the art would have been motivated to have provided a target camera view of the person in the target pose based on the captured images of the person from various viewing angles.
Re Claim 2:
The claim 2 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the instructions to map the appearance features into the target space, include instructions to project a ray in three dimensions using a projection function of the target camera view, including transforming source mesh vertices to the target space using the target pose.
Xu further teaches the claim limitation that the instructions to map the appearance features into the target space, include instructions to project a ray in three dimensions using a projection function of the target camera view, including transforming source mesh vertices to the target space using the target pose (
Xu teaches mapping the appearance features (e.g., the UV map-aligned pose-image features 130) from UV space into the image space. Xu teaches projecting a ray in 3D by projecting the coarse human mesh into UV space of the driving view using the projection function of the driving view and then converting the UV map-aligned pose features from UV space of the driving view to image space using the skeleton pose of a user.
Xu teaches at Paragraph [0029] The UV map-aligned pose-image features 130 are then transformed from UV space to image space by a feature renderer 132, which outputs image space-aligned features 134. The image space-aligned features 134 are further regressed by a texture encoder 136 implementing a two-dimensional convolutional neural network to output high-dimensional textural features 138 in image space.
Xu teaches at Paragraph 0038 that a partial texture map is constructed and outputted comprising unwrapped views of the driving views in UV space, based on the one or more driving views and the coarse human mesh outputted at step 206. At step 216, UV map-aligned image features are extracted or generated from the partial texture map and outputted via a neural network.
Xu teaches at Paragraph 0027 that the target surface normal of the coarse human mesh 108 is obtained and projected into UV space. Xu teaches at Paragraph [0060] that obtaining one or more driving views; calculating a skeletal pose of a user; generating a coarse human mesh (coarse features) representing the user based on a template mesh and the skeletal pose of the user.
Xu teaches at Paragraph 0031 that these three-dimensional volumetric features encoded as the volumetric features 144 of PID-NeRF in 3D space are rasterized (refined/projected) into rasterized multi-channel volumetric features 148 in image space, and each pixel is predicted by accumulating features of consecutive samples along a corresponding ray by a volumetric renderer 146 implementing a neural network. The first three channels of the rasterized multi-channel volumetric features 148 may be RGB, which may be supervised by downsampled ground truth images 162, using volume rendering loss 164. Alternatively, other color encoding may be used, such as CMYK, etc.
Xu teaches at Paragraph 0033 that the 2D textural features 138 and the rasterized multi-channel volumetric features 148 are fused together by an attentional volumetric textural feature fuser 140, which may include a convolutional neural network configured to downsample the textural features 138 (fine features) to the same size as the rasterized multi-channel volumetric features 148 (rough features or coarse features).
Xu teaches at Paragraph 0045 that the above-described system and methods use an effective scheme to encode UV map-aligned pose-image features, and leverages these to learn a pose- and image-conditioned downsampled NeRF (PID-NeRF) from low resolution images….extracting 2D textural features to achieve efficient, high quality, geometry-aware neural rendering of human avatars. The UV map-aligned encoder may accept arbitrary driving views as input, which may be leveraged for faithful rendering, and a texture loss may enforce full texture completion. Such faithful rendering may be used for improving rendering quality in telepresence applications.
Xu teaches at FIG. 3B and Paragraph 0021 that the coarse human mesh 108 is converted to the rasterized UV coordinate map of posed mesh and at Paragraph 0022 that the UV correspondences are leveraged between image pixels of a plurality of driving views 110 and the fitted coarse human mesh 108 and FIG. 3B illustrates an example image of the partial texture map 114 containing the unwrapped views, with pixel values from the inputted driving views painted at appropriate locations in the UV-aligned partial texture map 114 containing the unwrapped views.
Xu teaches at Paragraph [0060] that obtaining one or more driving views; calculating a skeletal pose of a user; generating a coarse human mesh (coarse features) representing the user based on a template mesh and the skeletal pose of the user; constructing a UV positional map based on the coarse human mesh; constructing a texture map based on the one or more driving views and the coarse human mesh; extracting a plurality of image features (fine features) from the texture map, the image features being aligned to the UV positional map; extracting a plurality of pose features from the UV positional map, the pose features being aligned to the UV positional map; generating a plurality of pose-image features based on the UV map-aligned image features and the UV map-aligned pose features; and rendering an avatar based on the plurality of pose-image features (corresponding to the claimed fine appearance features).
Xu teaches at Paragraph 0061 that the pose features being aligned to the UV positional map; generate a plurality of pose-image features based on the UV map-aligned image features and UV map-aligned pose features; generate textural features and multi-channel volumetric features from the pose-image features, channels of the multi-channel volumetric features corresponding to color channels of the one or the plurality of pixel images of the one or more driving views; fuse the textural features and the multi-channel volumetric features together; and render an avatar based on the fused textural features and the multi-channel volumetric features, the avatar including the wrinkles in the clothing of the user).
Re Claim 3:
The claim 3 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that a fine model that extracts the fine features and a coarse model that extracts the coarse features are encoders, and wherein the instructions to refine the coarse features include instructions to apply a transformer model to generate the refined features.
Xu, Kuo and Deng further teach the claim limitation that a fine model that extracts the fine features and a coarse model that extracts the coarse features are encoders, and wherein the instructions to refine the coarse features include instructions to apply a transformer model to generate the refined features (
Xu teaches at Paragraph [0029] The UV map-aligned pose-image features 130 are then transformed from UV space to image space by a feature renderer 132, which outputs image space-aligned features 134. The image space-aligned features 134 are further regressed by a texture encoder 136 implementing a two-dimensional convolutional neural network to output high-dimensional textural features 138 in image space.
Xu teaches at Paragraph 0026 that the concatenator 128, which may be configured as a neural texture model, receives as input the UV map-aligned image features 118 (fine features) and the UV map-aligned pose features 126 (coarse features or rough features), concatenates the UV map-aligned image features 118 and the UV map-aligned pose features 126, and generates and outputs a plurality of UV map-aligned pose-image features 130, or concatenated pose-image feature vectors, based on the UV map-aligned image features 118 and the UV map-aligned pose features 126. Xu teaches at Paragraph 0026 that the UV map-aligned image features 118 may capture texture information where details such as wrinkles in the clothing can be realistically rendered and the UV map-aligned pose features capture information sufficient to render the general shape of clothing worn by the user (coarse features).
Xu teaches at Paragraph 0031 that these three-dimensional volumetric features encoded as the volumetric features 144 of PID-NeRF in 3D space are rasterized (refined) into rasterized multi-channel volumetric features 148 (Ψ.sub.vol.sup.im) in image space, and each pixel is predicted by accumulating features of consecutive samples along a corresponding ray by a volumetric renderer 146 implementing a neural network. The first three channels of the rasterized multi-channel volumetric features 148 may be RGB, which may be supervised by downsampled ground truth images 162, using volume rendering loss 164. Alternatively, other color encoding may be used, such as CMYK, etc.
Xu teaches at Paragraph 0033 that the 2D textural features 138 and the rasterized multi-channel volumetric features 148 are fused together by an attentional volumetric textural feature fuser 140, which may include a convolutional neural network configured to downsample the textural features 138 (fine features) to the same size as the rasterized multi-channel volumetric features 148 (rough features or coarse features).
Xu teaches at Paragraph 0045 The above-described system and methods use an effective scheme to encode UV map-aligned pose-image features, and leverages these to learn a pose- and image-conditioned downsampled NeRF (PID-NeRF) from low resolution images….extracting 2D textural features to achieve efficient, high quality, geometry-aware neural rendering of human avatars. The UV map-aligned encoder may accept arbitrary driving views as input, which may be leveraged for faithful rendering, and a texture loss may enforce full texture completion. Such faithful rendering may be used for improving rendering quality in telepresence applications.
Kuo teaches at Paragraph 0029 that the multi-view feature fusion engine 108 may use features of the feature maps to align the multiple feature maps on common axes, such as a UV axis, to generate a highly detailed, UV aligned feature map 110 (fine features). The multi-view feature fusion engine 108 may also align textures of the images of the set of images 102 based on the alignment of the multiple feature maps to generate a texture map for the face.
Kuo teaches at Paragraph 0030 that UV aligned feature map 110 of FIG. 1, may be combined with a mean face 204(coarse features) to generate an intermediate residual position map 206.
Kuo teaches at Paragraph 0030 that the mean face 204 may be a topological mesh and the mean face 204 may be sampled into the UV coordinate system such that each 3D vertex of the mean face 204 (coarse features) is flattened (transformed) into a 2D UV space to generate a mean face position map 208.
Deng teaches at FIG. 4 and Paragraph 0054 that the appearance-code extractor 408 including encoder 410 generates the image features 432 (coarse features) as the coarse model. Deng teaches at Paragraph 0082 that the appearance-code-extractor 408 (encoder) at transformer 5406 may transform characteristics features to generate image features 432 (fine features).
Deng teaches at Paragraph 0044 that system 300 may select enrollment images 102 from among the larger set of images based on the viewing angle from which enrollment images 102 were captured being similar to target viewing angle 308.
Deng teaches at Paragraph [0047] Target viewing angle 308 may be a viewing angle from which synthesized image 108 is to be rendered. For example, target viewing angle 308 may include coordinates (e.g., relative coordinate) describing a location relative to the head of the person of enrollment images 102 from which synthesized image 108 should appear to be captured..
Deng teaches at Paragraph 0063 that transformer 506 may transform appearance code 504 determined based on enrollment data 402 into appearance texture 508 that may be relevant to target viewing angle 308.
Deng teaches at Paragraph [0079] At block 1004, the computing device (or one or more components thereof) may determine, from the selected enrollment images, appearance features representing at least one characteristic of the subject in the selected enrollment images. For example, appearance-code extractor 408 of FIG. 4 may generate image features 432 representing characteristics of the subject.
Deng teaches at Paragraph [0082] In some aspects, to generate the appearance features, the computing device (or one or more components thereof) may process the plurality of enrollment images using machine-learning model to generate image features (coarse features); pool the image features based on a mask indicative of the at least one characteristic to generate characteristic features; and transform the characteristic features based on the selected enrollment images to generate the appearance features (fine features). For example, appearance-code extractor 408 of FIG. 4 and/or encoders 410 of FIG. 4 and FIG. 5 may process enrollment images 404 to generate image features. Further appearance-code extractor 408, for example, at pooling 502 may pool the image features (e.g., image features 432) based on a mask indicative of a characteristic. Further, appearance-code extractor 408, for example, at transformer 506, may transform characteristic features based on the selected enrollment images to generate image features 432.
Deng teaches at Paragraph [0083] At block 1006, the computing device (or one or more components thereof) may combine the appearance features based on a three-dimensional geometry of the subject to generate combined appearance features. For example, combiner 416 of combine image features 432 to generate combined appearance features 430 based on a three-dimensional geometry of the subject.
Deng teaches at Paragraph [0084] In some aspects, to combine the appearance features, the computing device (or one or more components thereof) may process the appearance features using a machine-learning model to generate the combined appearance features. For example, system 400 may use combiner 416, which may be, or may include, a machine-learning model, to generate combined appearance features 430.
Deng teaches at Paragraph [0085] At block 1008, the computing device (or one or more components thereof) may generate an image of the subject from the target viewing angle based on a texture image of the subject and the combined appearance features. For example, image synthesizer 418 of FIG. 4 may generate synthesized image 108 based on combined appearance features 430 and texture image 422.
Deng teaches at Paragraph [0054] that appearance-code extractor 408 may generate image features 432 based on each of enrollment images 404 and a corresponding one of image masks 406. For example, appearance-code extractor 408 may include an encoder 410 to encode an enrollment image 404 along with a corresponding image mask 406 to generate image features 432.
Xu teaches at Paragraph 0045 that the UV map-aligned encoder may accept arbitrary driving views as input, which may be leveraged for faithful rendering.
Accordingly, the UV-aligned feature extraction module 105 includes an encoder to generate the UV-aligned pose-image features 130.
Xu teaches at FIG. 3B and Paragraph 0029 that the image space-aligned features 134 are further regressed by a texture encoder 136 implementing a two-dimensional convolutional neural network to output high-dimensional textural features 138 in image space. Xu teaches at Paragraph 0031 that these three-dimensional volumetric features encoded as the volumetric features 144 of PID-NeRF in 3D space are rasterized into rasterized multi-channel volumetric features 148 in image space.
Xu teaches at Paragraph 0026 that the concatenator 128, which may be configured as a neural texture model, receives as input the UV map-aligned image features 118 and the UV map-aligned pose features 126, concatenates the UV map-aligned image features 118 and the UV map-aligned pose features 126, and generates and outputs a plurality of UV map-aligned pose-image features 130, or concatenated pose-image feature vectors, based on the UV map-aligned image features 118 and the UV map-aligned pose features 126.
Xu teaches at Paragraph [0029] The UV map-aligned pose-image features 130 are then transformed from UV space to image space by a feature renderer 132, which outputs image space-aligned features 134. The image space-aligned features 134 are further regressed by a texture encoder 136 implementing a two-dimensional convolutional neural network to output high-dimensional textural features 138 in image space.
Xu teaches at Paragraph [0023] Image convolutional neural network 116 has been trained to recognize image features (or, more specifically, texture features) that are contained in the UV-aligned partial texture map 114. For example, Gray-Level Co-occurrence Matrix (GLCM) and Local Binary Pattern (LBP) are two texture features that may be extracted by the image convolutional neural network 116. Responsive to receiving input of the partial texture map 114, the image convolutional neural network 116 extracts and outputs a plurality of UV map-aligned image features 118 from the partial texture map 114.
Xu teaches at FIG. 3B and Paragraph 0021 that the coarse human mesh 108 is converted to the rasterized UV coordinate map of posed mesh and at Paragraph 0022 that the UV correspondences are leveraged between image pixels of a plurality of driving views 110 and the fitted coarse human mesh 108 and FIG. 3B illustrates an example image of the partial texture map 114 containing the unwrapped views, with pixel values from the inputted driving views painted at appropriate locations in the UV-aligned partial texture map 114 containing the unwrapped views.
Xu teaches at Paragraph [0060] that obtaining one or more driving views; calculating a skeletal pose of a user; generating a coarse human mesh (coarse features) representing the user based on a template mesh and the skeletal pose of the user; constructing a UV positional map based on the coarse human mesh; constructing a texture map based on the one or more driving views and the coarse human mesh; extracting a plurality of image features (fine features) from the texture map, the image features being aligned to the UV positional map; extracting a plurality of pose features from the UV positional map, the pose features being aligned to the UV positional map; generating a plurality of pose-image features based on the UV map-aligned image features and the UV map-aligned pose features; and rendering an avatar based on the plurality of pose-image features (corresponding to the claimed fine appearance features).
Xu teaches at Paragraph 0061 that the pose features being aligned to the UV positional map; generate a plurality of pose-image features based on the UV map-aligned image features and UV map-aligned pose features; generate textural features and multi-channel volumetric features from the pose-image features, channels of the multi-channel volumetric features corresponding to color channels of the one or the plurality of pixel images of the one or more driving views; fuse the textural features and the multi-channel volumetric features together; and render an avatar based on the fused textural features and the multi-channel volumetric features, the avatar including the wrinkles in the clothing of the user).
Re Claim 4:
The claim 4 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the instructions to extract the appearance features include instructions to lift the appearance features from a two-dimensional representation to a three-dimensional representation.
Xu further teaches the claim limitation that the instructions to extract the appearance features include instructions to lift the appearance features from a two-dimensional representation to a three-dimensional representation (
Xu teaches at Paragraph [0031] Turning to the 3D volumetric representation module 139, to render the observation space for arbitrary position o and view direction d as shown in FIG. 3B, the UV map-aligned pose-image features 130 are queried, using a barycentric or bilinear interpolation function 141, for all sampled points along camera rays and integrated into a 2D feature map 142. Continuing with reference to both FIGS. 3A and 3B, a pose- and image-conditioned neural radiance field (PID-NeRF) neural network (F.sub.θ) 143 regresses the 2D feature map 142 to generate three-dimensional volumetric features 144 of PID-NeRF in 3D space. These three-dimensional volumetric features encoded as the volumetric features 144 of PID-NeRF in 3D space are rasterized into rasterized multi-channel volumetric features 148 (Ψ.sub.vol.sup.im) in image space, and each pixel is predicted by accumulating features of consecutive samples along a corresponding ray by a volumetric renderer 146 implementing a neural network. The first three channels of the rasterized multi-channel volumetric features 148 may be RGB, which may be supervised by down-sampled ground truth images 162, using volume rendering loss 164. Alternatively, other color encoding may be used, such as CMYK, etc.
Xu teaches at Paragraph 0026 that the concatenator 128, which may be configured as a neural texture model, receives as input the UV map-aligned image features 118 and the UV map-aligned pose features 126, concatenates the UV map-aligned image features 118 and the UV map-aligned pose features 126, and generates and outputs a plurality of UV map-aligned pose-image features 130, or concatenated pose-image feature vectors, based on the UV map-aligned image features 118 and the UV map-aligned pose features 126.
Xu teaches at Paragraph [0029] The UV map-aligned pose-image features 130 are then transformed from UV space to image space by a feature renderer 132, which outputs image space-aligned features 134. The image space-aligned features 134 are further regressed by a texture encoder 136 implementing a two-dimensional convolutional neural network to output high-dimensional textural features 138 in image space.
Xu teaches at Paragraph [0023] Image convolutional neural network 116 has been trained to recognize image features (or, more specifically, texture features) that are contained in the UV-aligned partial texture map 114. For example, Gray-Level Co-occurrence Matrix (GLCM) and Local Binary Pattern (LBP) are two texture features that may be extracted by the image convolutional neural network 116. Responsive to receiving input of the partial texture map 114, the image convolutional neural network 116 extracts and outputs a plurality of UV map-aligned image features 118 from the partial texture map 114.
Xu teaches at FIG. 3B and Paragraph 0021 that the coarse human mesh 108 is converted to the rasterized UV coordinate map of posed mesh and at Paragraph 0022 that the UV correspondences are leveraged between image pixels of a plurality of driving views 110 and the fitted coarse human mesh 108 and FIG. 3B illustrates an example image of the partial texture map 114 containing the unwrapped views, with pixel values from the inputted driving views painted at appropriate locations in the UV-aligned partial texture map 114 containing the unwrapped views.
Xu teaches at Paragraph [0060] that obtaining one or more driving views; calculating a skeletal pose of a user; generating a coarse human mesh (coarse features) representing the user based on a template mesh and the skeletal pose of the user; constructing a UV positional map based on the coarse human mesh; constructing a texture map based on the one or more driving views and the coarse human mesh; extracting a plurality of image features (fine features) from the texture map, the image features being aligned to the UV positional map; extracting a plurality of pose features from the UV positional map, the pose features being aligned to the UV positional map; generating a plurality of pose-image features based on the UV map-aligned image features and the UV map-aligned pose features; and rendering an avatar based on the plurality of pose-image features (corresponding to the claimed fine appearance features).
Xu teaches at Paragraph 0061 that the pose features being aligned to the UV positional map; generate a plurality of pose-image features based on the UV map-aligned image features and UV map-aligned pose features; generate textural features and multi-channel volumetric features from the pose-image features, channels of the multi-channel volumetric features corresponding to color channels of the one or the plurality of pixel images of the one or more driving views; fuse the textural features and the multi-channel volumetric features together; and render an avatar based on the fused textural features and the multi-channel volumetric features, the avatar including the wrinkles in the clothing of the user).
Re Claim 5:
The claim 5 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the instructions to map the appearance features include instructions to transform source mesh vertices of the appearance features to the target space using the target pose and to populate a two-dimensional target feature map for pixels of the target camera view, and wherein the instructions to aggregate the appearance features into the aggregated feature map include instructions to apply a multi-view transform to the two-dimensional target feature map that includes multiple ones of the appearance features per pixel of the target camera view to generate the aggregated feature map.
Xu and Kuo further teach the claim limitation that the instructions to map the appearance features include instructions to transform source mesh vertices of the appearance features to the target space using the target pose and to populate a two-dimensional target feature map for pixels of the target camera view, and wherein the instructions to aggregate the appearance features into the aggregated feature map include instructions to apply a multi-view transform to the two-dimensional target feature map that includes multiple ones of the appearance features per pixel of the target camera view to generate the aggregated feature map (
Xu teaches at Paragraph 0059 constructing a texture map representing the user based on a template mesh and the skeletal pose of the user, constructing a UV positional map based on the coarse human mesh and constructing a texture map based on the one or more driving views and the coarse human mesh and extracting a plurality of image features from the texture map to be aligned to the UV positional map. The processor is further configured to generate textural features and multi-channel volumetric features from the pose-image features, channels of the multi-channel volumetric features corresponding to color channels of one or a plurality of pixel images of the one or more driving views; and fuse the textural features and the multi-channel volumetric features to render the avatar. Xu teaches at Paragraph 0060 the one or more driving views are transformed into a partial texture map comprising a plurality of unwrapped texture maps in UV space; and the plurality of unwrapped texture maps are unioned and averaged to construct the texture map based on the one or more driving views and the coarse human mesh.
Xu teaches at Paragraph 0060 that generating a coarse human mesh representing the user based on a template mesh and the skeletal pose of the user; constructing a UV positional map based on the coarse human mesh; constructing a texture map based on the one or more driving views and the coarse human mesh; extracting a plurality of image features from the texture map, the image features being aligned to the UV positional map; extracting a plurality of pose features from the UV positional map, the pose features being aligned to the UV positional map; generating a plurality of pose-image features based on the UV map-aligned image features and the UV map-aligned pose features; and rendering an avatar based on the plurality of pose-image features.
Xu teaches at Paragraph 0041 that the high-dimensional textural features and the multi-channel volumetric features are fused together via attentional volumetric textural feature fusion. At step 230, the fused features are converted into a target avatar via a textural renderer 230. At step 232, the target avatar is rendered and displayed on the display.
Xu teaches at Paragraph 0026 that the concatenator 128, which may be configured as a neural texture model, receives as input the UV map-aligned image features 118 and the UV map-aligned pose features 126, concatenates the UV map-aligned image features 118 and the UV map-aligned pose features 126, and generates and outputs a plurality of UV map-aligned pose-image features 130, or concatenated pose-image feature vectors, based on the UV map-aligned image features 118 and the UV map-aligned pose features 126.
Xu teaches at Paragraph 0039 that the UV map-aligned pose features and the UV map-aligned image features are concatenated into UV map-aligned pose-image features and the UV map-aligned pose-image features are transformed from UV space to image space to extract or generate and output image space-aligned pose-image features and at Paragraph 0040 that a 2D feature map is generated based on the UV map-aligned pose-image features. At step 225, the 2D feature map is regressed using a neural radiance field to generate features in 3D space. At step 226, the features in 3D space are rasterized into multi-channel volumetric features. Xu teaches at Paragraph 0041 that the high-dimensional textural features and the multi-channel volumetric features are fused together via attentional volumetric textural feature fusion.
Xu teaches at Paragraph 0059 that the processor is further configured to generate textural features and multi-channel volumetric features from the pose-image features, channels of the multi-channel volumetric features corresponding to color channels of one or a plurality of pixel images of the one or more driving views; and fuse the textural features and the multi-channel volumetric features to render the avatar.
Kuo teaches at Paragraph 0035 that the fine position map 314 (pose map) may be iteratively refined and may describe vertices of a fine face mesh projected into the 2D UV space and may be reprojected into 3D space to obtain a topological fine face mesh of the face captured by the images.
Kuo teaches at Paragraph [0039] At block 406, the computing device (or component thereof) may fuse the plurality of feature maps (e.g., by a multi-view feature fusion engine 108 of FIG. 1) based on features of the plurality of feature maps along a common axis to generate an aligned feature map. The computing device (or component thereof) may align textures of the plurality of images based on the fusing of the plurality of feature maps to generate a texture map; and apply the texture map to the fine face mesh to obtain a representation of the face.
Kuo teaches at Paragraph [0029] The multi-view feature fusion engine 108 may align and fuse the multi-view feature maps 106 to generate a UV aligned feature map 110. For example, the multi-view feature fusion engine 108 may use features of the feature maps to align the multiple feature maps on common axes, such as a UV axis, to generate a highly detailed, UV aligned feature map 110. The UV aligned feature map 110 may represent features from the images of the set of images 102, allowing points (e.g., pixels) of the UV aligned feature map 110 to be super sampled. In some examples, multiple features for a particular pixel of the UV aligned feature map 110 may be fused into a single feature. In some cases, the UV aligned feature map 110 may include features from all of the images of the set of images 102. In some cases, the multi-view feature fusion engine 108 may also align textures of the images of the set of images 102 based on the alignment of the multiple feature maps to generate a texture map for the face).
Re Claim 6:
The claim 6 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the instructions to render the target camera view include instructions to apply an image rendering network that is conditioned on the target pose to the aggregated feature map.
Deng, Kheradmand and Kuo further teach the claim limitation that the instructions to render the target camera view include instructions to apply an image rendering network that is conditioned on the target pose to the aggregated feature map (
Kheradmand teaches at column 5, lines 35-66 that the first appearance feature and the second appearance feature can be combined into a combined appearance feature and a third machine learning model generates the output image based on the pose feature and a combined appearance feature.
Deng teaches at Paragraph 0044 that system 300 may select enrollment images 102 from among the larger set of images based on the viewing angle from which enrollment images 102 were captured being similar to target viewing angle 308.
Deng teaches at Paragraph [0047] Target viewing angle 308 may be a viewing angle from which synthesized image 108 is to be rendered. For example, target viewing angle 308 may include coordinates (e.g., relative coordinate) describing a location relative to the head of the person of enrollment images 102 from which synthesized image 108 should appear to be captured..
Deng teaches at Paragraph 0063 that transformer 506 may transform appearance code 504 determined based on enrollment data 402 into appearance texture 508 that may be relevant to target viewing angle 308.
Deng teaches at Paragraph [0079] At block 1004, the computing device (or one or more components thereof) may determine, from the selected enrollment images, appearance features representing at least one characteristic of the subject in the selected enrollment images. For example, appearance-code extractor 408 of FIG. 4 may generate image features 432 representing characteristics of the subject.
Deng teaches at Paragraph [0082] In some aspects, to generate the appearance features, the computing device (or one or more components thereof) may process the plurality of enrollment images using machine-learning model to generate image features (coarse features); pool the image features based on a mask indicative of the at least one characteristic to generate characteristic features; and transform the characteristic features based on the selected enrollment images to generate the appearance features (fine features). For example, appearance-code extractor 408 of FIG. 4 and/or encoders 410 of FIG. 4 and FIG. 5 may process enrollment images 404 to generate image features. Further appearance-code extractor 408, for example, at pooling 502 may pool the image features (e.g., image features 432) based on a mask indicative of a characteristic. Further, appearance-code extractor 408, for example, at transformer 506, may transform characteristic features based on the selected enrollment images to generate image features 432.
Deng teaches at Paragraph [0083] At block 1006, the computing device (or one or more components thereof) may combine the appearance features based on a three-dimensional geometry of the subject to generate combined appearance features. For example, combiner 416 of combine image features 432 to generate combined appearance features 430 based on a three-dimensional geometry of the subject.
Deng teaches at Paragraph [0084] In some aspects, to combine the appearance features, the computing device (or one or more components thereof) may process the appearance features using a machine-learning model to generate the combined appearance features. For example, system 400 may use combiner 416, which may be, or may include, a machine-learning model, to generate combined appearance features 430.
Deng teaches at Paragraph [0085] At block 1008, the computing device (or one or more components thereof) may generate an image of the subject from the target viewing angle based on a texture image of the subject and the combined appearance features. For example, image synthesizer 418 of FIG. 4 may generate synthesized image 108 based on combined appearance features 430 and texture image 422.
Kuo teaches at Paragraph 0035 that the fine position map 314 (pose map) may be iteratively refined and may describe vertices of a fine face mesh projected into the 2D UV space and may be reprojected into 3D space to obtain a topological fine face mesh of the face captured by the images.
Kuo teaches at Paragraph [0039] At block 406, the computing device (or component thereof) may fuse the plurality of feature maps (e.g., by a multi-view feature fusion engine 108 of FIG. 1) based on features of the plurality of feature maps along a common axis to generate an aligned feature map. The computing device (or component thereof) may align textures of the plurality of images based on the fusing of the plurality of feature maps to generate a texture map; and apply the texture map to the fine face mesh to obtain a representation of the face.
Kuo teaches at Paragraph [0029] The multi-view feature fusion engine 108 may align and fuse the multi-view feature maps 106 to generate a UV aligned feature map 110. For example, the multi-view feature fusion engine 108 may use features of the feature maps to align the multiple feature maps on common axes, such as a UV axis, to generate a highly detailed, UV aligned feature map 110. The UV aligned feature map 110 may represent features from the images of the set of images 102, allowing points (e.g., pixels) of the UV aligned feature map 110 to be super sampled. In some examples, multiple features for a particular pixel of the UV aligned feature map 110 may be fused into a single feature. In some cases, the UV aligned feature map 110 may include features from all of the images of the set of images 102. In some cases, the multi-view feature fusion engine 108 may also align textures of the images of the set of images 102 based on the alignment of the multiple feature maps to generate a texture map for the face).
Re Claim 7:
The claim 7 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the instructions to provide the target camera view include instructions to perform one or more of communicate the target camera view to a path planner of a vehicle and simulate the target camera view according to a request to monitor the person, and wherein the sensor data includes one or more of RGB monocular images and LiDAR data.
Kumar, Kum and Kuo further teach the claim limitation that the instructions to provide the target camera view include instructions to perform one or more of communicate the target camera view to a path planner of a vehicle and simulate the target camera view according to a request to monitor the person, and wherein the sensor data includes one or more of RGB monocular images and LiDAR data (
Kumar teaches at FIG. 1 and Paragraph 0031-0037 that the controller 106 may be an autonomous or assisted driving controller and may control acceleration, braking and/or navigation of vehicle through the environment surrounding vehicle and the controller 106 includes processing circuitry 110 configured for generating a fused image having fused features from a plurality of different cameras or other sensors and processing circuitry may include view synthesis unit 140 and at Paragraph 0082 that the first output 246 may indicate a BEV view of one or more roads, road signs, road markers, traffic lights, vehicles, pedestrians, and other objects within the 3D environment corresponding to processing system 100. This may allow processing system 100 to use the first output 246 to control the vehicle within the 3D environment.
Kumar teaches at Paragraph 0025 that processing system 100 may be applicable for use with any multi-camera and/or multi-sensor system where the output the cameras/sensors are used to create a fused, synthesized, and/or reconstructed output. That is, processing system 100 may be used for any view synthesis or view construction use case where a single output (e.g., fused image) with a mesh or grid structure is created from multiple sources. Examples may include extended reality (XR) systems, virtual reality (VR) systems, spherical or 3-D video, and others.
Kumar teaches at Paragraph [0026] Processing system 100 may include LiDAR system 102 (optional), camera(s) 104, controller 106, one or more sensor(s) 108, input/output device(s) 120, wireless connectivity component 130, and memory 160.
Kumar teaches at Paragraph [0064] Projection unit 208 may transform perspective view features 206 into fused features in fused image 172. Such a transformation may be referred to as a PV-to-BEV projection and at Paragraph 0042 that subsequent processing of the fused image, such as object detection, depth detection, and/or image segmentation may be improved and autonomous driving decisions may be more accurate and at Paragraph 0090 that control unit 142 may then make one or more autonomous driving decisions based on the output of the object detection decoder and/or the segmentation decoder.
Kumar teaches at Paragraph [0087] Processing system 100 may fuse the features into a fused image having a grid structure (502). The fused image may be any type of image having a mesh or grid structure that may be reconstructed or synthesized from a plurality of different cameras.
Kumar teaches at Paragraph 0025 that Processing system 100 may be applicable for use with any multi-camera and/or multi-sensor system where the output the cameras/sensors are used to create a fused, synthesized, and/or reconstructed output. That is, processing system 100 may be used for any view synthesis or view construction use case where a single output (e.g., fused image) with a mesh or grid structure is created from multiple sources.
Kum teaches at Paragraph [0004] The autonomous driving technology may basically include a step (perception area) of perceiving a surrounding environment, a step (determination area) of planning a driving path from a perceived environment, and a step (control area) of a vehicle traveling along the planed driving path.
Kum teaches at Paragraph 0069 that he processor 120 may perform an element technology necessary to perceive an autonomous driving environment by using the 3-D feature map that has been obtained through the fusion of the data of the camera and the radar sensor.
Kum teaches at Paragraph [0101] FIG. 6 illustrates an example of a camera-radar fusion feature map, a camera feature map, and a radar feature map in an embodiment of the present disclosure. A 3-D feature map (i.e., the feature map in FIG. 6(A)) obtained through the camera-radar fusion shows that an environment in which diffused reflection that is difficult to detect when radar is solely used occurs or an object having low reflexibility like a person can be well detected, and a vehicle out of a front vehicle thereof, which is rarely seen in a camera image, or an object at a very long distance can also be well detected.
Kuo teaches at Paragraph [0029] The multi-view feature fusion engine 108 may align and fuse the multi-view feature maps 106 to generate a UV aligned feature map 110. For example, the multi-view feature fusion engine 108 may use features of the feature maps to align the multiple feature maps on common axes, such as a UV axis, to generate a highly detailed, UV aligned feature map 110. The UV aligned feature map 110 may represent features from the images of the set of images 102, allowing points (e.g., pixels) of the UV aligned feature map 110 to be super sampled. In some examples, multiple features for a particular pixel of the UV aligned feature map 110 may be fused into a single feature. In some cases, the UV aligned feature map 110 may include features from all of the images of the set of images 102. In some cases, the multi-view feature fusion engine 108 may also align textures of the images of the set of images 102 based on the alignment of the multiple feature maps to generate a texture map for the face).
Re Claim 8:
The claim 8 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the pose system is integrated with one of: a vehicle and a roadside unit (RSU), wherein the target information defines characteristics of the target camera view and a pose of the person in the target camera view.
Kumar, Kum and Kuo further teach the claim limitation that the pose system is integrated with one of: a vehicle and a roadside unit (RSU), wherein the target information defines characteristics of the target camera view and a pose of the person in the target camera view (
Kumar teaches that the target information defines the perspective views (poses) of one or more objects (e.g., pedestrians) within the BEV view.
Kumar teaches at Paragraph [0064] Projection unit 208 may transform perspective view features 206 into fused features in fused image 172. Such a transformation may be referred to as a PV-to-BEV projection and Paragraph [0087] Processing system 100 may fuse the features into a fused image having a grid structure (502). The fused image may be any type of image having a mesh or grid structure that may be reconstructed or synthesized from a plurality of different cameras.
Kumar teaches at FIG. 1 and Paragraph 0031-0037 that the controller 106 may be an autonomous or assisted driving controller and may control acceleration, braking and/or navigation of vehicle through the environment surrounding vehicle and the controller 106 includes processing circuitry 110 configured for generating a fused image having fused features from a plurality of different cameras or other sensors and processing circuitry may include view synthesis unit 140 and at Paragraph 0082 that the first output 246 may indicate a BEV view of one or more roads, road signs, road markers, traffic lights, vehicles, pedestrians, and other objects within the 3D environment corresponding to processing system 100. This may allow processing system 100 to use the first output 246 to control the vehicle within the 3D environment.
Kumar teaches at Paragraph 0054 that camera images 202 may represent one or more perspective views of one or more objects within a 3D space where processing system 100 is located. That is, the one or more perspective views may represent views from the perspective of processing system 100.
Kumar teaches at Paragraph [0089] To fuse the features (502), processing system 100 may be further configured to aggregate, based on the contribution to the respective cell and a respective set of learnable parameters for each cell, the features from each of the respective images to each respective cell of the fused image to generate aggregated features (506).
Kumar teaches at Paragraph 0063 that encoder-decoder architecture 200 to transform perspective view features 206 into BEV features that represent the one or more objects within the 3D environment on a grid structure from a perspective looking down at the one or more objects from a position above the one or more objects. Since encoder-decoder architecture 200 may be part of an ADAS for controlling a vehicle, and since vehicles move generally across the ground in a way that is observable from a bird's eye perspective, generating BEV features (e.g., fused features from multiple cameras) may allow a control unit (e.g., control unit 142 and/or control unit 196) of FIG. 1 to control the vehicle based on the representation of the one or more objects from a bird's eye perspective.
Kum teaches at Paragraph 0068 that the processor 120 may extract a feature map in a 3D space by fusing spatial and informative characteristics of data that are provided by the sensors and at Paragraph 0070 that an image feature map and a radar feature map may be fused and at Paragraph 0092 that the processor 120 may transform the image feature map into a BEV representation by concatenating the first frustum view feature map and the second frustum view feature map and at Paragraph 0100 that the camera feature map and the radar feature map of a BEV representation that are obtained each time may be fused and the final BEV feature map can be improved by fusing the BEV feature maps.
Kum teaches at Paragraph [0101] FIG. 6 illustrates an example of a camera-radar fusion feature map, a camera feature map, and a radar feature map in an embodiment of the present disclosure. A 3-D feature map (i.e., the feature map in FIG. 6(A)) obtained through the camera-radar fusion shows that an environment in which diffused reflection that is difficult to detect when radar is solely used occurs or an object having low reflexibility like a person can be well detected, and a vehicle out of a front vehicle thereof, which is rarely seen in a camera image, or an object at a very long distance can also be well detected.
Kum teaches at Paragraph [0069] In step S240, the processor 120 may perform an element technology necessary to perceive an autonomous driving environment by using the 3-D feature map that has been obtained through the fusion of the data of the camera and the radar sensor. The element technology necessary to perceive the autonomous driving environment may include 3-D object detection, 3-D object tracking, drivable area segmentation, and lane segmentation. The entire system can efficiently operate because various element technologies can operate by attaching a detector specified for each element technology to the 3-D feature map including information of a vehicle driving environment.
Kum teaches at Paragraph [0004] The autonomous driving technology may basically include a step (perception area) of perceiving a surrounding environment, a step (determination area) of planning a driving path from a perceived environment, and a step (control area) of a vehicle traveling along the planed driving path.
Kum teaches at Paragraph 0069 that he processor 120 may perform an element technology necessary to perceive an autonomous driving environment by using the 3-D feature map that has been obtained through the fusion of the data of the camera and the radar sensor.
Kum teaches at Paragraph [0101] FIG. 6 illustrates an example of a camera-radar fusion feature map, a camera feature map, and a radar feature map in an embodiment of the present disclosure. A 3-D feature map (i.e., the feature map in FIG. 6(A)) obtained through the camera-radar fusion shows that an environment in which diffused reflection that is difficult to detect when radar is solely used occurs or an object having low reflexibility like a person can be well detected, and a vehicle out of a front vehicle thereof, which is rarely seen in a camera image, or an object at a very long distance can also be well detected.
Kuo teaches at Paragraph [0029] The multi-view feature fusion engine 108 may align and fuse the multi-view feature maps 106 to generate a UV aligned feature map 110. For example, the multi-view feature fusion engine 108 may use features of the feature maps to align the multiple feature maps on common axes, such as a UV axis, to generate a highly detailed, UV aligned feature map 110. The UV aligned feature map 110 may represent features from the images of the set of images 102, allowing points (e.g., pixels) of the UV aligned feature map 110 to be super sampled. In some examples, multiple features for a particular pixel of the UV aligned feature map 110 may be fused into a single feature. In some cases, the UV aligned feature map 110 may include features from all of the images of the set of images 102. In some cases, the multi-view feature fusion engine 108 may also align textures of the images of the set of images 102 based on the alignment of the multiple feature maps to generate a texture map for the face).
Re Claim 9:
The claim 9 is in parallel with the claim 1 in the form of computer program product. The claim 9 is subject to the same rationale of rejection as the claim 1.
Re Claim 10:
The claim 10 is in parallel with the claim 2 in the form of computer program product. The claim 10 is subject to the same rationale of rejection as the claim 2.
Re Claim 11:
The claim 11 is in parallel with the claim 3 in the form of computer program product. The claim 11 is subject to the same rationale of rejection as the claim 3.
Re Claim 12:
The claim 12 encompasses the same scope of invention as that of the claim 9 except additional claim limitation that the instructions to extract the appearance features include instructions to lift the appearance features from a two-dimensional representation to a three-dimensional representation.
The claim 12 is in parallel with the claim 4 in the form of computer program product. The claim 12 is subject to the same rationale of rejection as the claim 4.
Re Claim 13:
The claim 13 encompasses the same scope of invention as that of the claim 9 except additional claim limitation that the instructions to map the appearance features include instructions to transform source mesh vertices of the appearance features to the target space using the target pose and to populate a two-dimensional target feature map for pixels of the target camera view, and wherein the instructions to aggregate the appearance features into the aggregated feature map include instructions to apply a multi-view transform to the two-dimensional target feature map that includes multiple ones of the appearance features per pixel of the target camera view to generate the aggregated feature map.
The claim 13 is in parallel with the claim 5 in the form of computer program product. The claim 13 is subject to the same rationale of rejection as the claim 5.
Re Claim 14:
The claim 14 is in parallel with the claim 1 in a method form. The claim 14 is subject to the same rationale of rejection as the claim 1.
Re Claim 15:
The claim 15 is in parallel with the claim 2 in a method form. The claim 15 is subject to the same rationale of rejection as the claim 2.
Re Claim 16:
The claim 16 is in parallel with the claim 3 in a method form. The claim 16 is subject to the same rationale of rejection as the claim 3.
Re Claim 17:
The claim 17 encompasses the same scope of invention as that of the claim 14 except additional claim limitation that extracting the appearance features includes lifting the appearance features from a two-dimensional representation to a three-dimensional representation.
The claim 17 is in parallel with the claim 4 in a method form. The claim 17 is subject to the same rationale of rejection as the claim 4.
Re Claim 18:
The claim 18 encompasses the same scope of invention as that of the claim 14 except additional claim limitation that mapping the appearance features includes transforming source mesh vertices of the appearance features to the target space using the target pose and populating a two-dimensional target feature map for pixels of the target camera view, and
wherein aggregating the appearance features into the aggregated feature map includes applying a multi-view transform to the two-dimensional target feature map that includes multiple ones of the appearance features per pixel of the target camera view to generate the aggregated feature map.
The claim 18 is in parallel with the claim 5 in a method form. The claim 18 is subject to the same rationale of rejection as the claim 5.
Re Claim 19:
The claim 19 encompasses the same scope of invention as that of the claim 14 except additional claim limitation that rendering the target camera view includes applying an image rendering network that is conditioned on the target pose to the aggregated feature map.
The claim 19 is in parallel with the claim 6 in a method form. The claim 19 is subject to the same rationale of rejection as the claim 6.
Re Claim 20:
The claim 20 encompasses the same scope of invention as that of the claim 14 except additional claim limitation that providing the target camera view includes one or more of communicating the target camera view to a path planner of a vehicle and simulating the target camera view according to a request to monitor the person, and wherein the sensor data includes one or more of RGB monocular images and LiDAR data.
The claim 20 is in parallel with the claim 7 in a method form. The claim 20 is subject to the same rationale of rejection as the claim 7.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIN CHENG WANG whose telephone number is (571)272-7665. The examiner can normally be reached Mon-Fri 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JIN CHENG WANG/Primary Examiner, Art Unit 2617