DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The abstract of the disclosure is objected to because the abstract is similar to Claim 1 and it should not be written as a claim. A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 5-8, 12-15, and 19-20 are rejected under 35 U.S.C. 102(a)(1)/(a)(2)as being anticipated by Wu et al. (US 20240212252 A1) (hereinafter Wu).
Regarding claim 1, Wu discloses An image synthesis method, the method comprising: ([0005] “An embodiment of this application provides a method and an apparatus for training a video generation model, a storage medium, and a computer device. The purpose is to improve movement coordination of a talking portrait video during generation.”; [0006] “In one aspect, an embodiment of this application provides a method for training a video generation model. The method is performed by a computer device, and the method includes: obtaining a training video of a target user; extracting a phonetic feature of the target user, an expression parameter of the target user from the training video, and a head parameter of the target user; synthesizing the phonetic feature of the target user, the expression parameter of the target user, and the head parameter of the target user to obtain a condition input of the training video; and performing network training on a neural radiance field based on the condition input, three-dimensional coordinates, and a viewing direction to obtain a video generation model, and the video generation model being configured to perform object reconstruction on a target video of the target user to obtain a corresponding reconstructed video of the target user.”)
determining one or more landmarks of a target image; ([0045] “In some embodiments, the step of extracting a phonetic feature of the target user, an expression parameter of the target user, and a head parameter of the target user from the training video includes the following steps:”; [0051] “The three-dimensional face reconstruction may refer to reconstructing a three-dimensional model of a face from one or more two-dimensional images. In an embodiment of this application, the two-dimensional image is a video frame of the training video.”)
processing the one or more landmarks to obtain a spatial structure feature of each of the one or more landmarks; ([0051] “The three-dimensional face reconstruction may refer to reconstructing a three-dimensional model of a face from one or more two-dimensional images. In an embodiment of this application, the two-dimensional image is a video frame of the training video. Therefore, the three-dimensional face reconstruction in this embodiment of this application refers to reconstructing the target user in the training video to obtain a three-dimensional face. The face shape representation includes a face shape and expression changes learned by the model from the three-dimensional face, and the expression parameter is determined based on the expression changes in the face shape representation.”; [0053] “Further, the expression coefficient e may be used as the expression parameter of a reconstructed three-dimensional face shape.”)
PNG
media_image1.png
666
516
media_image1.png
Greyscale
)
determining a sampling point based on a position of an image capture device and a pixel in a preview of the target image provided by the image capture device; ([0068] “In this application, the neural radiance field is used to synthesize information of space sampling points to obtain a two-dimensional view. The camera ray is a ray emitted from a camera in capturing an image of a scene, and the camera ray corresponds to a pixel in a video frames. When the camera captures an image of a three-dimensional scene, a pixel on the obtained two-dimensional image actually corresponds to a projection set of all continuous space sampling points on a camera ray emitted from the camera.”)
determining a position feature of the sampling point; ([0069] “The neural radiance field predicts an RGB color value (a color value) and density information (volume density) of the space sampling point based on the input three-dimensional coordinates and viewing direction of the space sampling point. Therefore, the three-dimensional coordinates and the viewing direction of the sampling point on the camera ray needs to be known.”; [0070] “In an embodiment, the three-dimensional coordinates x=(x, y, z) and the viewing direction d=(θ, ϕ) of the space sampling point may be preset. Specifically, because the position of the space sampling point determines the position of the pixel in the final two-dimensional plane image, the three-dimensional coordinates of the space sampling point can be set based on the position information of the pixel on the two-dimensional plane image. For example, pixel coordinates can be converted into the three-dimensional coordinates of the space sampling point on the camera ray in unified world coordinates based on internal and external parameters of the camera. Further, a viewing direction may be determined based on a preset shooting angle of the camera when shooting a scene. Alternatively, the viewing direction may be set in advance based on an observation angle of a character in an obtained reference video.”)
mapping an audio signal to the one or more landmarks of the target image; and (FIG. 3; FIG.8; [0118] “Because the color value is related to the position of the space sampling point and the viewing direction, the image reconstruction loss can guide the single neural radiance field to predict different lighting effects at the space sampling point from different perspectives. Finally, the color integration can be used to make the pixel corresponding to the camera ray more colorful, enhancing a display effect of the reconstructed video. When the object reconstruction is performed on the target video of the target user based on the video generation model, the obtained reconstructed video can be synchronized with the mouth motion of the target video, the changes in the mouth shape and audio can be accurately matched, and the coordination between the head motion and the shoulder motion of the reconstructed portrait can be maintained, greatly improving display realism of the reconstructed video.”; [0180] “FIG. 8 shows an implementation effect diagram of a method for training a video generation model. In this application, the implicit representation ability based on a single neural radiance field greatly improves the realism of the talking portrait video. The method for training a video generation model can be applied to application scenarios such as video conference, video chatting, livestreaming, and digital human that require reconstruction of talking portrait videos. The expression parameter and the phonetic feature are trained as the driving source of the single neural radiance field, a head pose and a facial expression in FIG. 8 (1) that accurately match the target video, and a mouth shape in FIG. 8 (2) that synchronizes with the audio of the target video can be obtained, all having different good appearances. In this application, the head pose information and the head position information are added in each video frame to the condition input of the single neural radiance field, guiding generation of the shoulder area, adapting to the head position, and finally generating natural, stable and coordinated shoulders in FIG. 8 (3). This avoids the problem of head-shoulder incoordination problem caused by rigid modeling of the head and shoulders.”
PNG
media_image2.png
349
728
media_image2.png
Greyscale
PNG
media_image3.png
423
590
media_image3.png
Greyscale
)
generating a synthetic image of the target image according to the spatial structure feature of (i) each of the one or more landmarks, (ii) the audio signal, and (iii) the position feature of the sampling point. (FIG. 3; FIG. 5; [0061] “S140: Perform network training on a preset single neural radiance field based on the condition input, three-dimensional coordinates, and a viewing direction, to obtain a video generation model.”; [0070] “In an embodiment, the three-dimensional coordinates x=(x, y, z) and the viewing direction d=(θ, ϕ) of the space sampling point may be preset. Specifically, because the position of the space sampling point determines the position of the pixel in the final two-dimensional plane image, the three-dimensional coordinates of the space sampling point can be set based on the position information of the pixel on the two-dimensional plane image. For example, pixel coordinates can be converted into the three-dimensional coordinates of the space sampling point on the camera ray in unified world coordinates based on internal and external parameters of the camera. Further, a viewing direction may be determined based on a preset shooting angle of the camera when shooting a scene. Alternatively, the viewing direction may be set in advance based on an observation angle of a character in an obtained reference video.”; [0130] “S230: The computer device extracts the condition input corresponding to the training video of the target user.”; [0131] “In this application, the head pose information and the head position information of the user are introduced into the condition input, so that the neural radiance field can implicitly estimate the motion state of the shoulder based on the head pose information and the head position information, enabling the generated reconstructed portrait to maintain coordination between the head motion and the shoulder motion.”; [0132] “In an embodiment of this application, a manner of obtaining the condition input corresponding to the training video of the target user includes: obtaining the training video; extracting a phonetic feature of the target user, an expression parameter of the target user, and a head parameter of the target user from the training video, the head parameter being used for representing the head pose information and the head position information of the target user; and synthesizing the phonetic feature of the target user, the expression parameter of the target user, and the head parameter of the target user to obtain the condition input of the training video.”; [0133] “In some embodiments, the step of extracting, by the computer device, a phonetic feature of the target user, an expression parameter of the target user, and a head parameter of the target user from the training video includes the following steps:”; [0144] S240: “The computer device performs network training on a preset single neural radiance field based on the condition input, three-dimensional coordinates, and a viewing direction, to obtain a video generation model.”; [0176] “S250: The computer device performs object reconstruction on the target video of the target user based on the video generation model to obtain the reconstructed video corresponding to the target user.”)
Regarding claim 5, Wu discloses The image synthesis method according to claim1, wherein the determining the one or more landmarks further comprises:
performing landmark extraction on the target image through a landmark extraction model to obtain the one or more landmarks of the target image. ([0052] “In an embodiment, a corresponding expression parameter may be obtained from each video frame of the training video. In some embodiments, a 3D Morphable Model (3D Morphable Model, 3DMM) can be used to obtain the expression parameter from each video frame. The 3D morphable model can perform three-dimensional reconstruction on a two-dimensional face in a single video frame to obtain a corresponding three-dimensional face, that is, a three-dimensional face shape, and the face shape representation v of the three-dimensional face shape is:”)
Regarding claim 6, Wu discloses The image synthesis method according to claim 1, wherein the determining the position feature further comprises:
obtaining a camera position of the target image in space based on a target tracking algorithm; ([0070] “In an embodiment, the three-dimensional coordinates x=(x, y, z) and the viewing direction d=(θ, ϕ) of the space sampling point may be preset. Specifically, because the position of the space sampling point determines the position of the pixel in the final two-dimensional plane image, the three-dimensional coordinates of the space sampling point can be set based on the position information of the pixel on the two-dimensional plane image. For example, pixel coordinates can be converted into the three-dimensional coordinates of the space sampling point on the camera ray in unified world coordinates based on internal and external parameters of the camera. Further, a viewing direction may be determined based on a preset shooting angle of the camera when shooting a scene. Alternatively, the viewing direction may be set in advance based on an observation angle of a character in an obtained reference video.”; [0089] “In addition, volume density (Volume Density) can be understood as a probability that a camera ray r is terminated when passing an infinitesimal particle at the location x of a space sampling point. This probability is differentiable, which means the opacity of this space sampling point. Because the space sampling points on a camera ray are continuous, the color value of the pixel corresponding to this camera ray on the two-dimensional image can be obtained by integration. FIG. 4 shows a schematic diagram of a camera ray. The camera ray (Ray) can be marked as r(t)=o+td. o represents the origin of the camera ray, d represents the angle of the camera ray, and the near boundary and the far boundary on the camera ray t are separately represented as t.sub.n and t.sub.f.”)
determining, based on the camera position of the target image in the space as a start point in an observation direction of the camera position for a pixel, a ray corresponding to a set of points from the start point to the pixel in the preview of the target image; ([0068] “In this application, the neural radiance field is used to synthesize information of space sampling points to obtain a two-dimensional view. The camera ray is a ray emitted from a camera in capturing an image of a scene, and the camera ray corresponds to a pixel in a video frames. When the camera captures an image of a three-dimensional scene, a pixel on the obtained two-dimensional image actually corresponds to a projection set of all continuous space sampling points on a camera ray emitted from the camera.”; [0088] “In an embodiment of this application, the neural radiance field obtains the color information and the density information of a three-dimensional space sampling point. When the camera captures an image of the scene, a pixel on the obtained two-dimensional image actually corresponds to all continuous space sampling points on a camera ray emitted from the camera. Therefore, it is necessary to obtain a color value of this camera ray finally rendered on the two-dimensional image based on all space sampling points on this camera ray.”; [0089] “In addition, volume density (Volume Density) can be understood as a probability that a camera ray r is terminated when passing an infinitesimal particle at the location x of a space sampling point. This probability is differentiable, which means the opacity of this space sampling point. Because the space sampling points on a camera ray are continuous, the color value of the pixel corresponding to this camera ray on the two-dimensional image can be obtained by integration. FIG. 4 shows a schematic diagram of a camera ray. The camera ray (Ray) can be marked as r(t)=o+td. o represents the origin of the camera ray, d represents the angle of the camera ray, and the near boundary and the far boundary on the camera ray t are separately represented as t.sub.n and t.sub.f.”)
sampling the ray to obtain the sampling point of the pixel; and ([0088] “In an embodiment of this application, the neural radiance field obtains the color information and the density information of a three-dimensional space sampling point. When the camera captures an image of the scene, a pixel on the obtained two-dimensional image actually corresponds to all continuous space sampling points on a camera ray emitted from the camera. Therefore, it is necessary to obtain a color value of this camera ray finally rendered on the two-dimensional image based on all space sampling points on this camera ray.”; [0089] “In addition, volume density (Volume Density) can be understood as a probability that a camera ray r is terminated when passing an infinitesimal particle at the location x of a space sampling point. This probability is differentiable, which means the opacity of this space sampling point. Because the space sampling points on a camera ray are continuous, the color value of the pixel corresponding to this camera ray on the two-dimensional image can be obtained by integration. FIG. 4 shows a schematic diagram of a camera ray. The camera ray (Ray) can be marked as r(t)=o+td. o represents the origin of the camera ray, d represents the angle of the camera ray, and the near boundary and the far boundary on the camera ray t are separately represented as t.sub.n and t.sub.f.”)
obtaining the position feature of the sampling point according to position information of the sampling point. ([0069] “The neural radiance field predicts an RGB color value (a color value) and density information (volume density) of the space sampling point based on the input three-dimensional coordinates and viewing direction of the space sampling point. Therefore, the three-dimensional coordinates and the viewing direction of the sampling point on the camera ray needs to be known.”; [0070] “In an embodiment, the three-dimensional coordinates x=(x, y, z) and the viewing direction d=(θ, ϕ) of the space sampling point may be preset. Specifically, because the position of the space sampling point determines the position of the pixel in the final two-dimensional plane image, the three-dimensional coordinates of the space sampling point can be set based on the position information of the pixel on the two-dimensional plane image. For example, pixel coordinates can be converted into the three-dimensional coordinates of the space sampling point on the camera ray in unified world coordinates based on internal and external parameters of the camera. Further, a viewing direction may be determined based on a preset shooting angle of the camera when shooting a scene. Alternatively, the viewing direction may be set in advance based on an observation angle of a character in an obtained reference video.”; [0087]-[0102])
Regarding claim 7, Wu discloses The image synthesis method according to claim 6, wherein the generating the synthetic image of the target image further comprises:
concatenating the spatial structure feature of a landmark of the one or more landmarks and the position feature of the sampling point associated with the landmark to obtain a second concatenated feature;
inputting the second concatenated feature into a multi-layer perceptron, to obtain color information and density information of the sampling point associated with the landmark; ([0070] “In an embodiment, the three-dimensional coordinates x=(x, y, z) and the viewing direction d=(θ, ϕ) of the space sampling point may be preset. Specifically, because the position of the space sampling point determines the position of the pixel in the final two-dimensional plane image, the three-dimensional coordinates of the space sampling point can be set based on the position information of the pixel on the two-dimensional plane image. For example, pixel coordinates can be converted into the three-dimensional coordinates of the space sampling point on the camera ray in unified world coordinates based on internal and external parameters of the camera. Further, a viewing direction may be determined based on a preset shooting angle of the camera when shooting a scene. Alternatively, the viewing direction may be set in advance based on an observation angle of a character in an obtained reference video.”; [0075] “In an embodiment, two temporal smoothing networks (Temporal Smoothing Network) may be separately used to filter a phonetic feature a and an expression parameter e. For example, performing temporal smoothing processing on the expression parameter e includes: in a time dimension, calculating the smooth expression parameter of the video frame at a moment t based on a linear combination of the expression parameter e of each video frame at a time step from t−T/2 to t+T/2. T is a time interval. A weight of the linear combination can be calculated using the expression parameter e as an input of the temporal smoothing network. The temporal smoothing network includes five one-dimensional convolutions, followed by a linear layer with softmax activation.”; [0077] “(2) Input the three-dimensional coordinates, the viewing direction, the smooth phonetic feature, the smooth expression parameter, and the head parameter to the preset single neural radiance field, and calculate the predicted color value and the volume density corresponding to the space sampling point.”; [0078] “In an embodiment, the single neural radiance field calculates a predicted color value c and a volume density σ of each space sampling point based on the three-dimensional coordinates, the viewing direction, the smooth phonetic feature, the smooth expression parameter, and the head parameter of the space sampling point. Specifically, the neural network of the single neural radiance field can be a multi-layer perceptron (Multi-Layer Perceptron, MLP), represented by an implicit function F.sub.θ:”;
PNG
media_image4.png
213
284
media_image4.png
Greyscale
PNG
media_image5.png
62
278
media_image5.png
Greyscale
[0079] “An input of the implicit function F.sub.θ (the single neural radiance field) includes three-dimensional coordinates x, a viewing direction d, a smooth phonetic feature a, a smooth expression parameter e, and a head parameter p. An output of the function F.sub.θ is the predicted color value c and the volume density σ corresponding to the space sampling point.”; [0082] “In a possible implementation, the neural radiance field can predict the volume density and an intermediate feature corresponding to the space sampling points based on the condition input and the three-dimensional coordinates x, and predict the predicted color value corresponding to the space sampling point based on the intermediate feature and the viewing direction d. Then, based on the predicted color value c and the volume density σ corresponding to the space sampling point, a complete image of the coordinated head-torso motions is generated, that is, the reconstructed video frame is generated. The single neural radiance field is trained based on the image reconstruction loss and the mouth emphasis loss. The mouth emphasis loss is calculated using a pre-obtained semantic segmentation map corresponding to the mouth area, and the intermediate feature is generated during the calculation process of the neural radiance field.”)
performing integration processing on the color information of the sampling point based on the density information to obtain a pixel value corresponding to the ray of the sampling point associated with the landmark; and
generating the synthetic image according to the pixel value. ([0087] “(3.1) Perform color integration on camera rays in the entire image area based on the predicted color value and the volume density, and predict a color value of a predicted object corresponding to each camera ray in the entire image area.”; [0088] “In an embodiment of this application, the neural radiance field obtains the color information and the density information of a three-dimensional space sampling point. When the camera captures an image of the scene, a pixel on the obtained two-dimensional image actually corresponds to all continuous space sampling points on a camera ray emitted from the camera. Therefore, it is necessary to obtain a color value of this camera ray finally rendered on the two-dimensional image based on all space sampling points on this camera ray.”; [0082] “In a possible implementation, the neural radiance field can predict the volume density and an intermediate feature corresponding to the space sampling points based on the condition input and the three-dimensional coordinates x, and predict the predicted color value corresponding to the space sampling point based on the intermediate feature and the viewing direction d. Then, based on the predicted color value c and the volume density σ corresponding to the space sampling point, a complete image of the coordinated head-torso motions is generated, that is, the reconstructed video frame is generated. The single neural radiance field is trained based on the image reconstruction loss and the mouth emphasis loss. The mouth emphasis loss is calculated using a pre-obtained semantic segmentation map corresponding to the mouth area, and the intermediate feature is generated during the calculation process of the neural radiance field.”)
Regarding claims 8, 12, 13, and 14, the claims are apparatus claims of method claims 1, 5, 6, and 7 respectively. The claims are similar in scope to claims 1, 5, 6, and 7 respectively and they are rejected under similar rationale as claims 1, 5, 6, and 7 respectively.
Regarding claims 15, 19, and 20, the claims are non-transitory computer-readable storage medium claims ([0007] of Wu) of method claims 1, 6, and 7 respectively. The claims are similar in scope to claims 1, 6, and 7 respectively and they are rejected under similar rationale as claims 1, 6, and 7 respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-3, 9-10, and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. (US 20240212252 A1) (hereinafter Wu) in view of Keller et al. (US 20230052645 A1) (hereinafter Keller).
Regarding Claim 2, Wu discloses The image synthesis method according to claim 1, wherein the processing further comprises:
performing ([0051] “The three-dimensional face reconstruction may refer to reconstructing a three-dimensional model of a face from one or more two-dimensional images. In an embodiment of this application, the two-dimensional image is a video frame of the training video. Therefore, the three-dimensional face reconstruction in this embodiment of this application refers to reconstructing the target user in the training video to obtain a three-dimensional face. The face shape representation includes a face shape and expression changes learned by the model from the three-dimensional face, and the expression parameter is determined based on the expression changes in the face shape representation.”; [0053]
“Further, the expression coefficient e may be used as the expression parameter of a reconstructed three-dimensional face shape.”;
PNG
media_image1.png
666
516
media_image1.png
Greyscale
)
Wu does not disclose performing .
Keller teaches performing hash grid encoding on the one or more landmarks to obtain the spatial structure feature of each of the one or more landmarks. ([0028] “FIG. 1B is a conceptual illustration of multiresolution hash encoding suitable for use in implementing some embodiments of the present disclosure. Operating at multiple resolutions allows the hash tables to represent both small and large features, while having a reliably small memory footprint and computational cost without requiring a task-specific data structure, in contrast with conventional parametric encodings.”; [0029] “A first step of the multiresolution hash encoding quantizes the input x to at least one resolution level to produce a set of quantized vertices 120. For a second step, each one of the integer coordinates x.sub.i,j in the set of quantized vertices 120 is hashed using a hash function 125, h: custom character.sup.d.fwdarw.custom-character.sub.T to produce a set of encoded coordinates 130. In an embodiment, a standard spatial hash function is used, having the form …”)
As both Wu and Keller are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wu to include performing hash grid encoding on the one or more landmarks to obtain the spatial structure feature of each of the one or more landmarks, in the context of neural network, according to the teaching of Kelly, in order to encode (map) inputs to the neural network into a higher dimensional space via a hash function accurately and efficiently (Abstract of Keller).
Regarding Claim 3, Wu discloses The image synthesis method according to claim 2, wherein the hash grid encoding further comprises:
performing a linear combination on the spatial structure feature of the first landmark according to the weight. ([0075] “In an embodiment, two temporal smoothing networks (Temporal Smoothing Network) may be separately used to filter a phonetic feature a and an expression parameter e. For example, performing temporal smoothing processing on the expression parameter e includes: in a time dimension, calculating the smooth expression parameter of the video frame at a moment t based on a linear combination of the expression parameter e of each video frame at a time step from t−T/2 to t+T/2. T is a time interval. A weight of the linear combination can be calculated using the expression parameter e as an input of the temporal smoothing network. The temporal smoothing network includes five one-dimensional convolutions, followed by a linear layer with softmax activation.”; [0148] “For example, the cloud training server 410 can use two temporal smoothing networks to filter a phonetic feature a and an expression parameter e respectively. For example, performing temporal smoothing processing on the expression parameter e includes: in a time dimension, calculating the smooth expression parameter of the video frame at a moment t based on a linear combination of the expression parameter e of each video frame at a time step from t−T/2 to t+T/2. A weight of the linear combination can be calculated using the expression parameter e as an input of the temporal smoothing network. The temporal smoothing network includes five one-dimensional convolutions, followed by a linear layer with softmax activation.”)
Wu does not disclose calculating a distance between a first landmark of the one or more landmarks and a first neighboring point of a plurality of neighboring points of the first landmark, and setting a weight of the first landmark and the first neighboring point based on the distance.
Keller teaches calculating a distance between a first landmark of the one or more landmarks and a first neighboring point of a plurality of neighboring points of the first landmark, and setting a weight of the first landmark and the first neighboring point based on the distance. ([0003] “Embodiments of the present disclosure relate to multiresolution hash encoding for neural networks. Neural network performance is improved in terms of training speed and/or accuracy by encoding (mapping) the input coordinates to the neural network into a higher dimensional space via a hash function. The input coordinates are used to identify a point within a d-dimensional space (e.g., 3D space). The point is quantized and a set of vertex coordinates corresponding to the point are input to a hash function. For example, for d=3, space may be partitioned into axis-aligned voxels of identical size and vertex coordinates of the eight corners of a voxel containing the point are input to the hash function to produce a set of encoded coordinates. The set of encoded coordinates is used to lookup D-dimensional feature vectors in a table of size T, where the feature vectors have been learned through training. The learned feature vectors are filtered (e.g., linearly interpolated, etc.) based on the coordinates of the point to compute a feature vector corresponding to the point. The feature vector is then input to the neural network model instead of providing the input coordinates directly to the neural network model. The quantizing, hash function computation, and feature vector lookup may be carried out independently on L different d-dimensional spaces (grids) at varying levels resolutions, before combining the results of each grid.”; [0024] “Encoding inputs for a neural network using the multiresolution hash function may improve the performance and accuracy for neural graphics primitive tasks. Neural graphics primitives, in which fully connected neural networks are used to encode an implicit function, can be costly to train and evaluate. For example, the neural network may map 2D positions to 3D colors (texture), 3D positions paired with view directions to 3D colors (radiance cache), 3D positions to distances from a query point to a closest point on a surface (signed distance functions), 3D positions paired with view directions to volumetric density and view-dependent color (combined radiance and density field).”; [0026] “FIG. 1A is a conceptual illustration of multiresolution vertex coordinates suitable for use in implementing some embodiments of the present disclosure. For example, a position x is an input to a neural network that generates a color for the position x within an image 100. When multiple encoding resolutions are used, the input x∈custom-character.sup.d is quantized at all L resolution levels (105, 110, and 115), where 2.sup.d integer coordinates are generated for each level, corresponding to the 2.sup.d vertices of the respective d-dimensional hypercube. FIG. 1A illustrates the case d=2: for each resolution level, the four corners of a grid aligned bounding box (grid cell) that encloses the position x are identified. Each grid cell 105, 110, and 115 that encloses x corresponds to a different resolution level L. A set of quantized vertices defining the corners of each of the grid cells 105, 110, and 115 may be input to a hash function to produce a set of encoded coordinates.”;
PNG
media_image6.png
279
382
media_image6.png
Greyscale
;
[0033] “Given a fully connected neural network m(y; Φ), the encoding of inputs x to produce y=ξ(x; θ) improves accuracy and training speed across a wide array of applications without incurring a notable performance overhead. The neural network not only has trainable weight parameters (Φ), but also trainable encoding parameters (θ). Although θ.sub.l is referred to as a “hash table”, there is no explicit collision handling such as probing, bucketing, or chaining. Rather, the neural network receiving the hash encoded input learns resilience with respect to collisions during training. Neural network performance is improved in terms of training speed and/or accuracy by encoding (mapping) inputs to the neural network into a higher dimensional space via the hash function.”)
Keller further teaches performing a linear combination on the spatial structure feature of the first landmark according to the weight. ([0032] “In a third step, the set of feature vectors 138 is filtered based on the coordinates of the point to compute a feature vector for the point at each resolution level l. In an embodiment, for each resolution level l, the corresponding 2.sup.d learned feature vectors are d-linearly interpolated according to the relative position of x within the hypercube (or grid cell) at resolution l to compute a single feature vector for each resolution. Linear interpolation provides an encoded input that is continuous which improves accuracy for the task compared using point sampling which selects a closest vertex to compute the feature vector at each resolution. In an embodiment, a higher-order interpolation function is used to filter the learned feature vectors. In an embodiment, the single feature vectors for the L resolutions are concatenated, resulting in a L.Math.D-dimensional value, y that is the encoded input to the neural network. In other embodiments, the single feature vectors are combined using a reduction or arithmetic operation.”; [0033] “Given a fully connected neural network m(y; Φ), the encoding of inputs x to produce y=ξ(x; θ) improves accuracy and training speed across a wide array of applications without incurring a notable performance overhead. The neural network not only has trainable weight parameters (Φ), but also trainable encoding parameters (θ). Although θ.sub.l is referred to as a “hash table”, there is no explicit collision handling such as probing, bucketing, or chaining. Rather, the neural network receiving the hash encoded input learns resilience with respect to collisions during training. Neural network performance is improved in terms of training speed and/or accuracy by encoding (mapping) inputs to the neural network into a higher dimensional space via the hash function.”)
As both Wu and Keller are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wu to include calculating a distance between a first landmark of the one or more landmarks and a first neighboring point of a plurality of neighboring points of the first landmark, and setting a weight of the first landmark and the first neighboring point based on the distance; and performing a linear combination on the spatial structure feature of the first landmark according to the weight, in the context of neural network, according to the teaching of Kelly, in order to encode (map) inputs to the neural network into a higher dimensional space via a hash function accurately and efficiently (Abstract of Keller).
Regarding claims 9 and 10, the claims are apparatus claims of method claims 2 and 3 respectively. The claims are similar in scope to claims 2 and 3 respectively and they are rejected under similar rationale as claims 2 and 3 respectively.
Regarding claims 16 and 17, the claims are non-transitory computer-readable storage medium claims ([0007] of Wu) of method claims 2 and 3 respectively. The claims are similar in scope to claims 2 and 3 respectively and they are rejected under similar rationale as claims 2 and 3 respectively.
Claims 4, 11, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. (US 20240212252 A1) (hereinafter Wu) in view of Chen et al. (US 20190035149 A1) (hereinafter Chen).
Regarding Claim 4, Wu discloses The image synthesis method according to claim 1, wherein
the processing the one or more landmarks further comprises:
concatenating the spatial structure feature of each of the N landmarks to obtain a first concatenated feature; and
performing spatial structure feature extraction on the first concatenated feature to obtain the spatial structure feature of each of the N landmarks. (
PNG
media_image7.png
356
333
media_image7.png
Greyscale
)
Wu does not disclose a quantity of the one or more landmarks is N.
Chen teaches a quantity of the one or more landmarks is N. ([0209] “To reconstruct a user's 3D face, we first analyze the input image and extract the shape features of the user's face. To achieve that, we detect the 2D facial landmarks automatically in our pipeline by integrating a 2D face landmark detector, which, in an example, can be provided by an open source image processing and computer vision library, e.g. DLib or OpenCV. In an example, the detector we adopted is implemented based on a regression forest algorithm [18]. It is able to detect NL=68 face landmarks from the image (see FIG. 6 for example), which characterise the positions and silhouettes of eyes, eyebrows, nose, mouth, lips, the jaw line, etc. This detector is proved to be reasonably robust against input images with different lighting conditions, head pose changes, and facial expression. The module can, however, be replaced by other more sophisticated 2D or 3D face landmark detectors or trackers, e.g. the 3D Constraint Local Model (CLM) based facial landmark detector [11].”)
As both Wu and Chen are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wu to include wherein a quantity of the one or more landmarks is N, and the processing the one or more landmarks further comprises: concatenating the spatial structure feature of each of the N landmarks to obtain a first concatenated feature; and performing spatial structure feature extraction on the first concatenated feature to obtain the spatial structure feature of each of the N landmarks, in the context of image synthesis and enhancement, according to the teaching of Chen, in order to generate a personalized head model of a user by detecting landmarks from an image (Abstract of Chen).
Regarding claims 11, the claim is the apparatus claim of method claim 4. The claim is similar in scope to claim 4 and is rejected under similar rationale as claims 4.
Regarding claims 18, the claim is the non-transitory computer-readable storage medium claims ([0007] of Wu) of method claim 4. The claim is similar in scope to claim 4 and is rejected under similar rationale as claim 4.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hyorim Park whose telephone number is (571)272-3859. The examiner can normally be reached Monday - Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Hyorim Park/Examiner, Art Unit 2615
/JAMES A THOMPSON/Primary Examiner, Art Unit 2615
Prosecution Insights