DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim 2 is rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 and 15 of U.S. Patent No. 12,249,031. Although the claims at issue are not identical, they are not patentably distinct from each other because they are generic to all that is recited in Patent 12,249,031 claims 1 and 15.
This will be described in reference to the table below; emphasis has been added in bold to the corresponding elements.
Instant Application 19/045638
US Patent 12,249,031
Claim 2:A computer-implemented method for generating an object based on output data, comprising:
training an autoencoder on a first set of training input data to identify a first set of latent variables and generate first set of output data;
training an hourglass predictor to return a second set of latent variables by converting a second set of training input data to the second set of latent variables and converting the second set of latent variables into a second set of output data at least substantially the same as a set of training target data, the set of training target data being generally of the same type of underlying object as the first set of training input data, and the second set of training input data is different from the first set of training input data;
using the hourglass predictor on a third set of input data to generate a third set of output data that is a comparable data format to the first set of output data; and
generating the object based on the third set of output data;
wherein the first set of training input data, the second set of training input data, the third set of input data, the set of training target data, the first set of output data, the second set of output data, and/or the third set of output data is a corresponding 3D mesh.
Claim 1:
A computer-implemented method for generating an object based on output data, comprising:
training an autoencoder on a first set of training input data to identify a first set of latent variables and generate first set of output data, where the autoencoder comprises a first encoder, and a first decoder, where the first encoder converts the first set of input data into a first set of latent variables, where the first decoder converts the first set of latent variables to the first set of output data, where the first set of output data is at least substantially the same as the first set of training input data;
training an hourglass predictor to return a second set of latent variables, where the hourglass predictor comprises a second encoder and the first decoder, where the second encoder converts a second set of training input data to the second set of latent variables, where the second set of latent variables has a comparable data format as the first set of latent variables, by the first decoder converts the second set of latent variables into a second set of output data at least substantially the same as a set of training target data, the set of training target data being generally of the same type of underlying object as the first set of training input data, and the second set of training input data is different from the first set of training input data; and
using the hourglass predictor on a third set of input data to generate a third set of output data, where the third set of input data and the second set of input data have the same type of underlying object and the same data format, where the third set of output data is a comparable data format to the first set of output data, and
generating the object based on the third set of output data, wherein the object is a dental restoration, an orthodontic appliance, an ear-related device or a proposed digital 2D image of a desired dental setup based on a pre-treatment digital 2D image.
Claim 15:
The method according to claim 1, wherein the first set of training input data, the second set of training input data, the third set of input data, the set of training target data, the first set of output data, the second set of output data, and the third set of output data is a corresponding 3D mesh, corresponding to a sampled matrix.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4, 6, 15, and 17-21 are rejected under 35 U.S.C. 103 as being unpatentable over Gallo et al. (US Pub. 2020/0202622), hereinafter Gallo, in view of Carr et al. (US Pub. 2019/0026631), hereinafter Carr.
Regarding claim 2, Gallo discloses a computer-implemented method for generating an object based on output data, comprising training an autoencoder on a first set of training input data to identify a first set of latent variables and generate first set of output data (Fig. 2; Paragraph [0018]: training engine 122 generates one or more machine learning models for performing mesh reconstruction using data-driven priors. Each machine learning model may learn priors related to vertices, edges, corners, faces, polygons, surfaces, shapes, and/or other attributes of two-dimensional (2D) and/or three-dimensional (3D) meshes. For example, each machine learning model may include a variational autoencoder (VAE) that learns to convert input meshes into latent vectors and reconstruct the meshes from the latent vectors; Paragraph [0023]: the machine learning model created by training engine 122 includes a VAE 200, which includes an encoder 202 and a decoder 206. For example, training engine 122 may input 2D and/or 3D training meshes 208 into encoder 202, and encoder 202 may “encode” training meshes 208 into latent vectors 204 (e.g., scalars, geometric vectors, tensors, and/or other geometric objects) within a latent space of lower dimensionality than training meshes 208. A decoder 206 in VAE 200 may “decode” latent vectors 204 into higher dimensionality reconstructed meshes 210 that substantially reproduce training meshes 208); using the hourglass predictor on a third set of input data to generate a third set of output data that is a comparable data format to the first set of output data (Fig. 2; Paragraph [0021]: execution engine 124 uses the machine learning model to perform inverse rendering of an object 260 captured in a set of images 258 into a corresponding mesh 216. For example, execution engine 124 may use the machine learning model to estimate a 3D mesh 216 of object from 2D images 258 that capture multiple views and/or multiple illuminations of object 260. Thus, images 258 may represent ground truth image observations of object 260; Paragraph [0032]-[0033]: execution engine 124 executes one or more portions of VAE 200 to produce a mesh 216 of an object 260 from a latent vector value 212 of a vector in the latent space of VAE 200 and images 258 of object 260. For example, execution engine 124 may input latent vector value 212 into decoder 206 to generate mesh 216 as a decoded representation of the vector… in one embodiment, execution engine 124 varies latent vector value 212 inputted into decoder 206 with fixed parameters to generate mesh 216 so that mesh 216 reproduces an object 260 captured in a set of images 258. In one or more embodiments, images 258 are associated with a set of assumptions that allow mesh 216 to be reconstructed base on images 258); and generating the object based on the third set of output data (Fig. 2; Paragraph [0021]: execution engine 124 uses the machine learning model to perform inverse rendering of an object 260 captured in a set of images 258 into a corresponding mesh 216. For example, execution engine 124 may use the machine learning model to estimate a 3D mesh 216 of object from 2D images 258 that capture multiple views and/or multiple illuminations of object 260. Thus, images 258 may represent ground truth image observations of object 260; Paragraph [0032]-[0033]: execution engine 124 executes one or more portions of VAE 200 to produce a mesh 216 of an object 260 from a latent vector value 212 of a vector in the latent space of VAE 200 and images 258 of object 260. For example, execution engine 124 may input latent vector value 212 into decoder 206 to generate mesh 216 as a decoded representation of the vector… in one embodiment, execution engine 124 varies latent vector value 212 inputted into decoder 206 with fixed parameters to generate mesh 216 so that mesh 216 reproduces an object 260 captured in a set of images 258. In one or more embodiments, images 258 are associated with a set of assumptions that allow mesh 216 to be reconstructed base on images 258); wherein the first set of training input data, the second set of training input data, the third set of input data, the set of training target data, the first set of output data, the second set of output data, and/or the third set of output data is a corresponding 3D mesh (Fig. 2; Paragraphs [0020]-[0024]: FIG. 2 is a more detailed illustration of training engine 122 and execution engine 124 of FIG. 1, according to various embodiments. In the embodiment shown, training engine 122 creates a machine learning model that learns to reconstruct a set of training meshes 208 by learning and/or encoding priors associated with training meshes 208. For example, the machine learning model may learn, from training meshes 208, a number of “basic types” related to vertices, edges, corners, faces, triangles, polygons, surfaces, shapes, and/or other attributes of 2D and/or 3D meshes… training meshes 208 may include collections of 2D and/or 3D points, representations of edges between pairs of points, and/or representations of triangles, polygons, and/or other shapes formed by the points and edges. Edges and/or shapes in training meshes 208 may be encoded in orderings of the points within training meshes 208. For example, a 2D mesh may be parameterized as an ordered set of points, with each adjacent pair of points in the ordered set connected by an edge between the points. In another example, a 3D mesh may be parameterized using one or more triangle strips, triangle fans, and/or other representations of triangles within the mesh; Paragraphs [0027]-[0029]: training engine 122 trains a different encoder that is not in VAE 200 to generate intermediate representations of ordered sets of points and/or other parameterizations of training meshes 208. In one embodiment, training engine 122 averages and/or otherwise aggregates the intermediate representations into a standardized value in the latent space, such that the standardized value defines the mean and standard deviation from which the output of encoder 202 is sampled…training engine 122 inputs the output sampled using the standardized value into decoder 206 to obtain reconstructed meshes 210 as output of decoder 206. Training engine 122 also computes an error between each training mesh and a corresponding reconstructed mesh as the minimum error between an ordering of points in the training mesh inputted into encoder 202 and the outputted ordering of points from decoder 206. For example, training engine 122 may compute the minimum error based on the intersection of points in the training mesh and reconstructed mesh, a Chamfer distance between the training mesh and reconstructed mesh, an earth mover's distance, and/or another measure of distance, similarity, and/or dissimilarity between the points, edges, shapes, and/or other attributes of the training mesh and reconstructed mesh…training engine 122 uses the error between each training mesh and the corresponding reconstructed mesh to update the parameters of encoder 202 and decoder 206. For example, training engine 122 may backpropagate the error across the layers and/or parameters of decoder 206 so that decoder 206 learns to decode the reconstructed mesh from the sampled output of encoder 202. Training engine 122 may also backpropagate the error across the layers and/or parameters of encoder 202 so that encoder 202 learns to produce a consistent latent vector representation (e.g., mean and standard deviation vectors) for different orderings of points in the corresponding training mesh; Paragraphs [0080]-[0084]: Processing operations specific to any of the functional execution units may be pipelined, which enables a new instruction to be issued for execution before a previous instruction has completed execution. Any combination of functional execution units within a given SM 710 may be provided. In various embodiments, the functional execution units may be configured to support a variety of different operations including integer and floating point arithmetic (e.g., addition and multiplication), comparison operations, Boolean operations (AND, OR, 50R), bit-shifting, and computation of various algebraic functions (e.g., planar interpolation and trigonometric, exponential, and logarithmic functions, etc.). Advantageously, the same functional execution unit can be configured to perform different operations… the matrix multiply inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D may be 16-bit floating point or 32-bit floating point matrices. Tensor Cores operate on 16-bit floating point input data with 32-bit floating point accumulation. The 16-bit floating point multiply requires 64 operations and results in a full precision product that is then accumulated using 32-bit floating point addition with the other intermediate products for a 4×4×4 matrix multiply. In practice, Tensor Cores are used to perform much larger two-dimensional or higher dimensional matrix operations, built up from these smaller elements. An API, such as CUDA 9 C++ API, exposes specialized matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use tensor cores from a CUDA-C++ program. At the CUDA level, the warp-level interface assumes 16×16 size matrices spanning all 32 threads of the warp).
Gallo does not explicitly disclose training an hourglass predictor to return a second set of latent variables by converting a second set of training input data to the second set of latent variables and converting the second set of latent variables into a second set of output data at least substantially the same as a set of training target data, the set of training target data being generally of the same type of underlying object as the first set of training input data, and the second set of training input data is different from the first set of training input data.
However, Carr teaches training variational autoencoders (Abstract), further comprising training an hourglass predictor to return a second set of latent variables by converting a second set of training input data to the second set of latent variables and converting the second set of latent variables into a second set of output data at least substantially the same as a set of training target data, the set of training target data being generally of the same type of underlying object as the first set of training input data, and the second set of training input data is different from the first set of training input data (Fig. 1; Paragraph [0129]: Panel B shows a factorized variational autoencoder (FVAE) framework 105 that solves the problems with linear tensor factorization and the straightforward combination of tensor factorization with deep learning discussed above. The FVAE framework builds a hierarchical prior model on top of a variational autoencoder (VAE) to combine the concepts of tensor factorization and VAEs. A VAE is a generative latent variable model that embeds each vector xit separately into a K-dimensional latent space. Similar to the straightforward combination of tensor factorization with deep learning, in the FVAE model, the matrix F representing latent spatial factors discussed above is replaced with a neural network that is a non-linear transform. Such a neural network is also referred to herein as the “decoder” 124 which takes a product of U and V and regenerates a tensor (that resembles observed data X). In order to learn such a decoder, a non-linear transform may be applied to training data 119, such as data in form of the tensor X, to get it into a space where regular matrix factorization can be performed. The encoder 120 is a neural network that performs such a non-linear transform to compress an element of the input tensor X into a lower-dimensional vector 112 of latent factors Z and predicts associated uncertainties. When learning the VAE, a constraint is enforced that any Z generated by the encoder must factor into vectors U 128 and V 126, i.e., that the output of the encoder factorizes. This permits generalization (e.g., to unobserved or missing data) like matrix factorization, which a traditional VAE's encoding of individual objects could not achieve. The learning process itself fits the input data from tensor X to the model to find hidden parameters U, V, and Z and corresponding neural network parameters. Illustratively, the output of the encoder 120, namely the vector Z 122 which factors into vectors U 128 and V 126, can be input into the decoder 124 to generate reconstructed data 125 similar to the input observations representing, e.g., the locations of landmarks on objects). Gallo and Carr are in the same specific field of endeavor and Carr teaches that this will allow for compact and accurate modeling of complex data (Paragraph [0014]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gallo with the features of above as taught by Carr so as to allow for compact and accurate modeling of complex data as presented by Carr.
Regarding claim 4, Gallo, in view of Carr teaches the method according to claim 2, Gallo discloses wherein the corresponding 3D mesh corresponds to a sampled matrix (Paragraphs [0027]-[0029]: training engine 122 trains a different encoder that is not in VAE 200 to generate intermediate representations of ordered sets of points and/or other parameterizations of training meshes 208. In one embodiment, training engine 122 averages and/or otherwise aggregates the intermediate representations into a standardized value in the latent space, such that the standardized value defines the mean and standard deviation from which the output of encoder 202 is sampled…training engine 122 inputs the output sampled using the standardized value into decoder 206 to obtain reconstructed meshes 210 as output of decoder 206. Training engine 122 also computes an error between each training mesh and a corresponding reconstructed mesh as the minimum error between an ordering of points in the training mesh inputted into encoder 202 and the outputted ordering of points from decoder 206. For example, training engine 122 may compute the minimum error based on the intersection of points in the training mesh and reconstructed mesh, a Chamfer distance between the training mesh and reconstructed mesh, an earth mover's distance, and/or another measure of distance, similarity, and/or dissimilarity between the points, edges, shapes, and/or other attributes of the training mesh and reconstructed mesh…training engine 122 uses the error between each training mesh and the corresponding reconstructed mesh to update the parameters of encoder 202 and decoder 206. For example, training engine 122 may backpropagate the error across the layers and/or parameters of decoder 206 so that decoder 206 learns to decode the reconstructed mesh from the sampled output of encoder 202. Training engine 122 may also backpropagate the error across the layers and/or parameters of encoder 202 so that encoder 202 learns to produce a consistent latent vector representation (e.g., mean and standard deviation vectors) for different orderings of points in the corresponding training mesh; Paragraphs [0080]-[0084]: Processing operations specific to any of the functional execution units may be pipelined, which enables a new instruction to be issued for execution before a previous instruction has completed execution. Any combination of functional execution units within a given SM 710 may be provided. In various embodiments, the functional execution units may be configured to support a variety of different operations including integer and floating point arithmetic (e.g., addition and multiplication), comparison operations, Boolean operations (AND, OR, 50R), bit-shifting, and computation of various algebraic functions (e.g., planar interpolation and trigonometric, exponential, and logarithmic functions, etc.). Advantageously, the same functional execution unit can be configured to perform different operations… the matrix multiply inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D may be 16-bit floating point or 32-bit floating point matrices. Tensor Cores operate on 16-bit floating point input data with 32-bit floating point accumulation. The 16-bit floating point multiply requires 64 operations and results in a full precision product that is then accumulated using 32-bit floating point addition with the other intermediate products for a 4×4×4 matrix multiply. In practice, Tensor Cores are used to perform much larger two-dimensional or higher dimensional matrix operations, built up from these smaller elements. An API, such as CUDA 9 C++ API, exposes specialized matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use tensor cores from a CUDA-C++ program. At the CUDA level, the warp-level interface assumes 16×16 size matrices spanning all 32 threads of the warp).
Regarding claim 6, Gallo, in view of Carr teaches the method according to claim 4, Gallo discloses wherein the initial three-dimensional mesh comprises a first set of vertices and edges and the planar mesh comprises a second set of vertices and edges (Fig. 2; Paragraphs [0020]-[0024]: FIG. 2 is a more detailed illustration of training engine 122 and execution engine 124 of FIG. 1, according to various embodiments. In the embodiment shown, training engine 122 creates a machine learning model that learns to reconstruct a set of training meshes 208 by learning and/or encoding priors associated with training meshes 208. For example, the machine learning model may learn, from training meshes 208, a number of “basic types” related to vertices, edges, corners, faces, triangles, polygons, surfaces, shapes, and/or other attributes of 2D and/or 3D meshes… training meshes 208 may include collections of 2D and/or 3D points, representations of edges between pairs of points, and/or representations of triangles, polygons, and/or other shapes formed by the points and edges. Edges and/or shapes in training meshes 208 may be encoded in orderings of the points within training meshes 208. For example, a 2D mesh may be parameterized as an ordered set of points, with each adjacent pair of points in the ordered set connected by an edge between the points. In another example, a 3D mesh may be parameterized using one or more triangle strips, triangle fans, and/or other representations of triangles within the mesh; Paragraphs [0080]-[0084]: Processing operations specific to any of the functional execution units may be pipelined, which enables a new instruction to be issued for execution before a previous instruction has completed execution. Any combination of functional execution units within a given SM 710 may be provided. In various embodiments, the functional execution units may be configured to support a variety of different operations including integer and floating point arithmetic (e.g., addition and multiplication), comparison operations, Boolean operations (AND, OR, 50R), bit-shifting, and computation of various algebraic functions (e.g., planar interpolation and trigonometric, exponential, and logarithmic functions, etc.). Advantageously, the same functional execution unit can be configured to perform different operations… the matrix multiply inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D may be 16-bit floating point or 32-bit floating point matrices. Tensor Cores operate on 16-bit floating point input data with 32-bit floating point accumulation. The 16-bit floating point multiply requires 64 operations and results in a full precision product that is then accumulated using 32-bit floating point addition with the other intermediate products for a 4×4×4 matrix multiply. In practice, Tensor Cores are used to perform much larger two-dimensional or higher dimensional matrix operations, built up from these smaller elements. An API, such as CUDA 9 C++ API, exposes specialized matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use tensor cores from a CUDA-C++ program. At the CUDA level, the warp-level interface assumes 16×16 size matrices spanning all 32 threads of the warp).
Regarding claim 15, Gallo, in view of Carr teaches the method according to claim 2, wherein: the autoencoder comprises a first encoder and a first decoder (Gallo: Fig. 2; Paragraph [0018]: training engine 122 generates one or more machine learning models for performing mesh reconstruction using data-driven priors. Each machine learning model may learn priors related to vertices, edges, corners, faces, polygons, surfaces, shapes, and/or other attributes of two-dimensional (2D) and/or three-dimensional (3D) meshes. For example, each machine learning model may include a variational autoencoder (VAE) that learns to convert input meshes into latent vectors and reconstruct the meshes from the latent vectors; Paragraph [0023]: the machine learning model created by training engine 122 includes a VAE 200, which includes an encoder 202 and a decoder 206. For example, training engine 122 may input 2D and/or 3D training meshes 208 into encoder 202, and encoder 202 may “encode” training meshes 208 into latent vectors 204 (e.g., scalars, geometric vectors, tensors, and/or other geometric objects) within a latent space of lower dimensionality than training meshes 208. A decoder 206 in VAE 200 may “decode” latent vectors 204 into higher dimensionality reconstructed meshes 210 that substantially reproduce training meshes 208); and the hourglass predictor comprises a second encoder and the first decoder (Carr: Fig. 1; Paragraph [0129]: Panel B shows a factorized variational autoencoder (FVAE) framework 105 that solves the problems with linear tensor factorization and the straightforward combination of tensor factorization with deep learning discussed above. The FVAE framework builds a hierarchical prior model on top of a variational autoencoder (VAE) to combine the concepts of tensor factorization and VAEs. A VAE is a generative latent variable model that embeds each vector xit separately into a K-dimensional latent space. Similar to the straightforward combination of tensor factorization with deep learning, in the FVAE model, the matrix F representing latent spatial factors discussed above is replaced with a neural network that is a non-linear transform. Such a neural network is also referred to herein as the “decoder” 124 which takes a product of U and V and regenerates a tensor (that resembles observed data X). In order to learn such a decoder, a non-linear transform may be applied to training data 119, such as data in form of the tensor X, to get it into a space where regular matrix factorization can be performed. The encoder 120 is a neural network that performs such a non-linear transform to compress an element of the input tensor X into a lower-dimensional vector 112 of latent factors Z and predicts associated uncertainties. When learning the VAE, a constraint is enforced that any Z generated by the encoder must factor into vectors U 128 and V 126, i.e., that the output of the encoder factorizes. This permits generalization (e.g., to unobserved or missing data) like matrix factorization, which a traditional VAE's encoding of individual objects could not achieve. The learning process itself fits the input data from tensor X to the model to find hidden parameters U, V, and Z and corresponding neural network parameters. Illustratively, the output of the encoder 120, namely the vector Z 122 which factors into vectors U 128 and V 126, can be input into the decoder 124 to generate reconstructed data 125 similar to the input observations representing, e.g., the locations of landmarks on objects).
Regarding claim 17, Gallo, in view of Carr teaches the method according to claim 15, Gallo discloses wherein the first decoder converts the first set of latent variables into the first set of output data (Fig. 2; Paragraph [0018]: training engine 122 generates one or more machine learning models for performing mesh reconstruction using data-driven priors. Each machine learning model may learn priors related to vertices, edges, corners, faces, polygons, surfaces, shapes, and/or other attributes of two-dimensional (2D) and/or three-dimensional (3D) meshes. For example, each machine learning model may include a variational autoencoder (VAE) that learns to convert input meshes into latent vectors and reconstruct the meshes from the latent vectors; Paragraph [0023]: the machine learning model created by training engine 122 includes a VAE 200, which includes an encoder 202 and a decoder 206. For example, training engine 122 may input 2D and/or 3D training meshes 208 into encoder 202, and encoder 202 may “encode” training meshes 208 into latent vectors 204 (e.g., scalars, geometric vectors, tensors, and/or other geometric objects) within a latent space of lower dimensionality than training meshes 208. A decoder 206 in VAE 200 may “decode” latent vectors 204 into higher dimensionality reconstructed meshes 210 that substantially reproduce training meshes 208).
Regarding claim 18, Gallo, in view of Carr teaches the method according to claim 2, Gallo discloses wherein the first set of output data is at least substantially the same as the first set of training input data (Fig. 2; Paragraph [0018]: training engine 122 generates one or more machine learning models for performing mesh reconstruction using data-driven priors. Each machine learning model may learn priors related to vertices, edges, corners, faces, polygons, surfaces, shapes, and/or other attributes of two-dimensional (2D) and/or three-dimensional (3D) meshes. For example, each machine learning model may include a variational autoencoder (VAE) that learns to convert input meshes into latent vectors and reconstruct the meshes from the latent vectors; Paragraph [0023]: the machine learning model created by training engine 122 includes a VAE 200, which includes an encoder 202 and a decoder 206. For example, training engine 122 may input 2D and/or 3D training meshes 208 into encoder 202, and encoder 202 may “encode” training meshes 208 into latent vectors 204 (e.g., scalars, geometric vectors, tensors, and/or other geometric objects) within a latent space of lower dimensionality than training meshes 208. A decoder 206 in VAE 200 may “decode” latent vectors 204 into higher dimensionality reconstructed meshes 210 that substantially reproduce training meshes 208).
Regarding claim 19, Gallo, in view of Carr teaches the method according to claim 15, Carr discloses wherein the second encoder converts the second set of training unput data to the second set of latent variables (Fig. 1; Paragraph [0129]: Panel B shows a factorized variational autoencoder (FVAE) framework 105 that solves the problems with linear tensor factorization and the straightforward combination of tensor factorization with deep learning discussed above. The FVAE framework builds a hierarchical prior model on top of a variational autoencoder (VAE) to combine the concepts of tensor factorization and VAEs. A VAE is a generative latent variable model that embeds each vector xit separately into a K-dimensional latent space. Similar to the straightforward combination of tensor factorization with deep learning, in the FVAE model, the matrix F representing latent spatial factors discussed above is replaced with a neural network that is a non-linear transform. Such a neural network is also referred to herein as the “decoder” 124 which takes a product of U and V and regenerates a tensor (that resembles observed data X). In order to learn such a decoder, a non-linear transform may be applied to training data 119, such as data in form of the tensor X, to get it into a space where regular matrix factorization can be performed. The encoder 120 is a neural network that performs such a non-linear transform to compress an element of the input tensor X into a lower-dimensional vector 112 of latent factors Z and predicts associated uncertainties. When learning the VAE, a constraint is enforced that any Z generated by the encoder must factor into vectors U 128 and V 126, i.e., that the output of the encoder factorizes. This permits generalization (e.g., to unobserved or missing data) like matrix factorization, which a traditional VAE's encoding of individual objects could not achieve. The learning process itself fits the input data from tensor X to the model to find hidden parameters U, V, and Z and corresponding neural network parameters. Illustratively, the output of the encoder 120, namely the vector Z 122 which factors into vectors U 128 and V 126, can be input into the decoder 124 to generate reconstructed data 125 similar to the input observations representing, e.g., the locations of landmarks on objects).
Regarding claim 20, Gallo, in view of Carr teaches the method according to claim 2, Carr discloses wherein the second set of latent variables has a comparable data format as the first set of latent variables (Fig. 1; Paragraph [0129]: Panel B shows a factorized variational autoencoder (FVAE) framework 105 that solves the problems with linear tensor factorization and the straightforward combination of tensor factorization with deep learning discussed above. The FVAE framework builds a hierarchical prior model on top of a variational autoencoder (VAE) to combine the concepts of tensor factorization and VAEs. A VAE is a generative latent variable model that embeds each vector xit separately into a K-dimensional latent space. Similar to the straightforward combination of tensor factorization with deep learning, in the FVAE model, the matrix F representing latent spatial factors discussed above is replaced with a neural network that is a non-linear transform. Such a neural network is also referred to herein as the “decoder” 124 which takes a product of U and V and regenerates a tensor (that resembles observed data X). In order to learn such a decoder, a non-linear transform may be applied to training data 119, such as data in form of the tensor X, to get it into a space where regular matrix factorization can be performed. The encoder 120 is a neural network that performs such a non-linear transform to compress an element of the input tensor X into a lower-dimensional vector 112 of latent factors Z and predicts associated uncertainties. When learning the VAE, a constraint is enforced that any Z generated by the encoder must factor into vectors U 128 and V 126, i.e., that the output of the encoder factorizes. This permits generalization (e.g., to unobserved or missing data) like matrix factorization, which a traditional VAE's encoding of individual objects could not achieve. The learning process itself fits the input data from tensor X to the model to find hidden parameters U, V, and Z and corresponding neural network parameters. Illustratively, the output of the encoder 120, namely the vector Z 122 which factors into vectors U 128 and V 126, can be input into the decoder 124 to generate reconstructed data 125 similar to the input observations representing, e.g., the locations of landmarks on objects).
Regarding claim 21, Gallo, in view of Carr teaches the method according to claim 2, Gallo discloses wherein the third set of input data and the second set of input data have the same type of underlying object and the same data format (Fig. 2; Paragraph [0021]: execution engine 124 uses the machine learning model to perform inverse rendering of an object 260 captured in a set of images 258 into a corresponding mesh 216. For example, execution engine 124 may use the machine learning model to estimate a 3D mesh 216 of object from 2D images 258 that capture multiple views and/or multiple illuminations of object 260. Thus, images 258 may represent ground truth image observations of object 260; Paragraph [0032]-[0033]: execution engine 124 executes one or more portions of VAE 200 to produce a mesh 216 of an object 260 from a latent vector value 212 of a vector in the latent space of VAE 200 and images 258 of object 260. For example, execution engine 124 may input latent vector value 212 into decoder 206 to generate mesh 216 as a decoded representation of the vector… in one embodiment, execution engine 124 varies latent vector value 212 inputted into decoder 206 with fixed parameters to generate mesh 216 so that mesh 216 reproduces an object 260 captured in a set of images 258. In one or more embodiments, images 258 are associated with a set of assumptions that allow mesh 216 to be reconstructed base on images 258).
Claims 3, 5, 7-14, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Gallo, in view of Carr, and further in view of Apte et al. (US Pub. 2019/0172555), hereinafter Apte.
Regarding claim 3, Gallo, in view of Carr teaches the method according to claim 2.
Gallo, in view of Carr does not explicitly disclose wherein the object is a dental restoration, an orthodontic appliance, an ear-related device, and/or a proposed digital 2D image of a desired dental setup based on a pre-treatment digital 2D image.
However, Apte teaches training autoencoders (Paragraph [0247]), further comprising wherein the object is a dental restoration, an orthodontic appliance, an ear-related device, and/or a proposed digital 2D image of a desired dental setup based on a pre-treatment digital 2D image (Paragraph [0254]: therapies promoted by the therapy model of Block S150 can include one or more of: different forms of therapy having different therapy orientations (e.g., motivational, increase energy level, reduce weight gain, improve diet, psychoeducational, cognitive behavioral, biological, physical, mindfulness-related, relaxation-related, dialectical behavioral, acceptance-related, commitment-related, etc.) configured to address a variety of factors contributing to an adverse states due to a microbiome that is altered by an oral health issue or a microbiome that is caused by or indicative of an oral health issue; weight management interventions (e.g., to prevent adverse weight-related (e.g., weight gain or loss) side effects due to dental decay or gingivitis, or a therapy to prevent, mitigate, or reduce the frequency or severity of dental decay or gingivitis); gingival graft; dental restoration; application of dental sealants; physical therapy; rehabilitation measures; and any other suitable therapeutic measure). Apte teaches that this will allow for compact and accurate predictive power of modeling of data (Paragraph [0168]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Gallo, in view of Carr with the features of above as taught by Apte so as to allow for compact and accurate modeling of complex data as presented by Apte.
Regarding claim 5, Gallo, in view of Carr, and further in view of Apte teaches the method according to claim 3, Gallo discloses wherein the sampled matrix is generated by transforming an initial three-dimensional mesh into a planar mesh (Fig. 2; Paragraphs [0020]-[0024]: FIG. 2 is a more detailed illustration of training engine 122 and execution engine 124 of FIG. 1, according to various embodiments. In the embodiment shown, training engine 122 creates a machine learning model that learns to reconstruct a set of training meshes 208 by learning and/or encoding priors associated with training meshes 208. For example, the machine learning model may learn, from training meshes 208, a number of “basic types” related to vertices, edges, corners, faces, triangles, polygons, surfaces, shapes, and/or other attributes of 2D and/or 3D meshes… training meshes 208 may include collections of 2D and/or 3D points, representations of edges between pairs of points, and/or representations of triangles, polygons, and/or other shapes formed by the points and edges. Edges and/or shapes in training meshes 208 may be encoded in orderings of the points within training meshes 208. For example, a 2D mesh may be parameterized as an ordered set of points, with each adjacent pair of points in the ordered set connected by an edge between the points. In another example, a 3D mesh may be parameterized using one or more triangle strips, triangle fans, and/or other representations of triangles within the mesh; Paragraphs [0080]-[0084]: Processing operations specific to any of the functional execution units may be pipelined, which enables a new instruction to be issued for execution before a previous instruction has completed execution. Any combination of functional execution units within a given SM 710 may be provided. In various embodiments, the functional execution units may be configured to support a variety of different operations including integer and floating point arithmetic (e.g., addition and multiplication), comparison operations, Boolean operations (AND, OR, 50R), bit-shifting, and computation of various algebraic functions (e.g., planar interpolation and trigonometric, exponential, and logarithmic functions, etc.). Advantageously, the same functional execution unit can be configured to perform different operations… the matrix multiply inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D may be 16-bit floating point or 32-bit floating point matrices. Tensor Cores operate on 16-bit floating point input data with 32-bit floating point accumulation. The 16-bit floating point multiply requires 64 operations and results in a full precision product that is then accumulated using 32-bit floating point addition with the other intermediate products for a 4×4×4 matrix multiply. In practice, Tensor Cores are used to perform much larger two-dimensional or higher dimensional matrix operations, built up from these smaller elements. An API, such as CUDA 9 C++ API, exposes specialized matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use tensor cores from a CUDA-C++ program. At the CUDA level, the warp-level interface assumes 16×16 size matrices spanning all 32 threads of the warp).
Regarding claim 7, Gallo, in view of Carr, and further in view of Apte teaches the method according to claim 5, Gallo discloses wherein at least one vertex of the second set of vertices is a transformation of a vertex from the first set of vertices and comprises a value from the vertex from the first set of vertices (Fig. 2; Paragraphs [0020]-[0024]: FIG. 2 is a more detailed illustration of training engine 122 and execution engine 124 of FIG. 1, according to various embodiments. In the embodiment shown, training engine 122 creates a machine learning model that learns to reconstruct a set of training meshes 208 by learning and/or encoding priors associated with training meshes 208. For example, the machine learning model may learn, from training meshes 208, a number of “basic types” related to vertices, edges, corners, faces, triangles, polygons, surfaces, shapes, and/or other attributes of 2D and/or 3D meshes… training meshes 208 may include collections of 2D and/or 3D points, representations of edges between pairs of points, and/or representations of triangles, polygons, and/or other shapes formed by the points and edges. Edges and/or shapes in training meshes 208 may be encoded in orderings of the points within training meshes 208. For example, a 2D mesh may be parameterized as an ordered set of points, with each adjacent pair of points in the ordered set connected by an edge between the points. In another example, a 3D mesh may be parameterized using one or more triangle strips, triangle fans, and/or other representations of triangles within the mesh; Paragraphs [0080]-[0084]: Processing operations specific to any of the functional execution units may be pipelined, which enables a new instruction to be issued for execution before a previous instruction has completed execution. Any combination of functional execution units within a given SM 710 may be provided. In various embodiments, the functional execution units may be configured to support a variety of different operations including integer and floating point arithmetic (e.g., addition and multiplication), comparison operations, Boolean operations (AND, OR, 50R), bit-shifting, and computation of various algebraic functions (e.g., planar interpolation and trigonometric, exponential, and logarithmic functions, etc.). Advantageously, the same functional execution unit can be configured to perform different operations… the matrix multiply inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D may be 16-bit floating point or 32-bit floating point matrices. Tensor Cores operate on 16-bit floating point input data with 32-bit floating point accumulation. The 16-bit floating point multiply requires 64 operations and results in a full precision product that is then accumulated using 32-bit floating point addition with the other intermediate products for a 4×4×4 matrix multiply. In practice, Tensor Cores are used to perform much larger two-dimensional or higher dimensional matrix operations, built up from these smaller elements. An API, such as CUDA 9 C++ API, exposes specialized matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use tensor cores from a CUDA-C++ program. At the CUDA level, the warp-level interface assumes 16×16 size matrices spanning all 32 threads of the warp).
Regarding claim 8, Gallo, in view of Carr, and further in view of Apte teaches the method according to claim 5, Gallo discloses wherein at least one edge of the second set of edges is a transformation of an edge from the first set of edges and comprises a value of the edge from the first set of edges (Fig. 2; Paragraphs [0020]-[0024]: FIG. 2 is a more detailed illustration of training engine 122 and execution engine 124 of FIG. 1, according to various embodiments. In the embodiment shown, training engine 122 creates a machine learning model that learns to reconstruct a set of training meshes 208 by learning and/or encoding priors associated with training meshes 208. For example, the machine learning model may learn, from training meshes 208, a number of “basic types” related to vertices, edges, corners, faces, triangles, polygons, surfaces, shapes, and/or other attributes of 2D and/or 3D meshes… training meshes 208 may include collections of 2D and/or 3D points, representations of edges between pairs of points, and/or representations of triangles, polygons, and/or other shapes formed by the points and edges. Edges and/or shapes in training meshes 208 may be encoded in orderings of the points within training meshes 208. For example, a 2D mesh may be parameterized as an ordered set of points, with each adjacent pair of points in the ordered set connected by an edge between the points. In another example, a 3D mesh may be parameterized using one or more triangle strips, triangle fans, and/or other representations of triangles within the mesh; Paragraphs [0080]-[0084]: Processing operations specific to any of the functional execution units may be pipelined, which enables a new instruction to be issued for execution before a previous instruction has completed execution. Any combination of functional execution units within a given SM 710 may be provided. In various embodiments, the functional execution units may be configured to support a variety of different operations including integer and floating point arithmetic (e.g., addition and multiplication), comparison operations, Boolean operations (AND, OR, 50R), bit-shifting, and computation of various algebraic functions (e.g., planar interpolation and trigonometric, exponential, and logarithmic functions, etc.). Advantageously, the same functional execution unit can be configured to perform different operations… the matrix multiply inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D may be 16-bit floating point or 32-bit floating point matrices. Tensor Cores operate on 16-bit floating point input data with 32-bit floating point accumulation. The 16-bit floating point multiply requires 64 operations and results in a full precision product that is then accumulated using 32-bit floating point addition with the other intermediate products for a 4×4×4 matrix multiply. In practice, Tensor Cores are used to perform much larger two-dimensional or higher dimensional matrix operations, built up from these smaller elements. An API, such as CUDA 9 C++ API, exposes specialized matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use tensor cores from a CUDA-C++ program. At the CUDA level, the warp-level interface assumes 16×16 size matrices spanning all 32 threads of the warp).
Regarding claim 9, Gallo, in view of Carr, and further in view of Apte teaches the method according to claim 5, Gallo discloses further comprising: sampling the planar mesh to generate a plurality of samples (Paragraphs [0027]-[0029]: training engine 122 trains a different encoder that is not in VAE 200 to generate intermediate representations of ordered sets of points and/or other parameterizations of training meshes 208. In one embodiment, training engine 122 averages and/or otherwise aggregates the intermediate representations into a standardized value in the latent space, such that the standardized value defines the mean and standard deviation from which the output of encoder 202 is sampled…training engine 122 inputs the output sampled using the standardized value into decoder 206 to obtain reconstructed meshes 210 as output of decoder 206. Training engine 122 also computes an error between each training mesh and a corresponding reconstructed mesh as the minimum error between an ordering of points in the training mesh inputted into encoder 202 and the outputted ordering of points from decoder 206. For example, training engine 122 may compute the minimum error based on the intersection of points in the training mesh and reconstructed mesh, a Chamfer distance between the training mesh and reconstructed mesh, an earth mover's distance, and/or another measure of distance, similarity, and/or dissimilarity between the points, edges, shapes, and/or other attributes of the training mesh and reconstructed mesh…training engine 122 uses the error between each training mesh and the corresponding reconstructed mesh to update the parameters of encoder 202 and decoder 206. For example, training engine 122 may backpropagate the error across the layers and/or parameters of decoder 206 so that decoder 206 learns to decode the reconstructed mesh from the sampled output of encoder 202. Training engine 122 may also backpropagate the error across the layers and/or parameters of encoder 202 so that encoder 202 learns to produce a consistent latent vector representation (e.g., mean and standard deviation vectors) for different orderings of points in the corresponding training mesh).
Regarding claim 10, Gallo, in view of Carr, and further in view of Apte teaches the method according to claim 9, Gallo discloses wherein at least one sample from the plurality of samples comprises a three-dimensional coordinate comprising three numerical values representing a point in a three-dimensional space (Paragraphs [0027]-[0029]: training engine 122 trains a different encoder that is not in VAE 200 to generate intermediate representations of ordered sets of points and/or other parameterizations of training meshes 208. In one embodiment, training engine 122 averages and/or otherwise aggregates the intermediate representations into a standardized value in the latent space, such that the standardized value defines the mean and standard deviation from which the output of encoder 202 is sampled…training engine 122 inputs the output sampled using the standardized value into decoder 206 to obtain reconstructed meshes 210 as output of decoder 206. Training engine 122 also computes an error between each training mesh and a corresponding reconstructed mesh as the minimum error between an ordering of points in the training mesh inputted into encoder 202 and the outputted ordering of points from decoder 206. For example, training engine 122 may compute the minimum error based on the intersection of points in the training mesh and reconstructed mesh, a Chamfer distance between the training mesh and reconstructed mesh, an earth mover's distance, and/or another measure of distance, similarity, and/or dissimilarity between the points, edges, shapes, and/or other attributes of the training mesh and reconstructed mesh…training engine 122 uses the error between each training mesh and the corresponding reconstructed mesh to update the parameters of encoder 202 and decoder 206. For example, training engine 122 may backpropagate the error across the layers and/or parameters of decoder 206 so that decoder 206 learns to decode the reconstructed mesh from the sampled output of encoder 202. Training engine 122 may also backpropagate the error across the layers and/or parameters of encoder 202 so that encoder 202 learns to produce a consistent latent vector representation (e.g., mean and standard deviation vectors) for different orderings of points in the corresponding training mesh).
Regarding claim 11, Gallo, in view of Carr, and further in view of Apte teaches the method according to claim 10, Gallo discloses wherein the three numerical values are derived and/or taken directly from the initial three-dimensional mesh (Paragraphs [0027]-[0029]: training engine 122 trains a different encoder that is not in VAE 200 to generate intermediate representations of ordered sets of points and/or other parameterizations of training meshes 208. In one embodiment, training engine 122 averages and/or otherwise aggregates the intermediate representations into a standardized value in the latent space, such that the standardized value defines the mean and standard deviation from which the output of encoder 202 is sampled…training engine 122 inputs the output sampled using the standardized value into decoder 206 to obtain reconstructed meshes 210 as output of decoder 206. Training engine 122 also computes an error between each training mesh and a corresponding reconstructed mesh as the minimum error between an ordering of points in the training mesh inputted into encoder 202 and the outputted ordering of points from decoder 206. For example, training engine 122 may compute the minimum error based on the intersection of points in the training mesh and reconstructed mesh, a Chamfer distance between the training mesh and reconstructed mesh, an earth mover's distance, and/or another measure of distance, similarity, and/or dissimilarity between the points, edges, shapes, and/or other attributes of the training mesh and reconstructed mesh…training engine 122 uses the error between each training mesh and the corresponding reconstructed mesh to update the parameters of encoder 202 and decoder 206. For example, training engine 122 may backpropagate the error across the layers and/or parameters of decoder 206 so that decoder 206 learns to decode the reconstructed mesh from the sampled output of encoder 202. Training engine 122 may also backpropagate the error across the layers and/or parameters of encoder 202 so that encoder 202 learns to produce a consistent latent vector representation (e.g., mean and standard deviation vectors) for different orderings of points in the corresponding training mesh).
Regarding claim 12, Gallo, in view of Carr, and further in view of Apte teaches the method according to claim 9, Gallo discloses wherein at least one sample from the plurality of samples comprises a coordinate comprising a numerical value representing a position of the sample relative to other samples of the plurality of samples (Paragraphs [0027]-[0029]: training engine 122 trains a different encoder that is not in VAE 200 to generate intermediate representations of ordered sets of points and/or other parameterizations of training meshes 208. In one embodiment, training engine 122 averages and/or otherwise aggregates the intermediate representations into a standardized value in the latent space, such that the standardized value defines the mean and standard deviation from which the output of encoder 202 is sampled…training engine 122 inputs the output sampled using the standardized value into decoder 206 to obtain reconstructed meshes 210 as output of decoder 206. Training engine 122 also computes an error between each training mesh and a corresponding reconstructed mesh as the minimum error between an ordering of points in the training mesh inputted into encoder 202 and the outputted ordering of points from decoder 206. For example, training engine 122 may compute the minimum error based on the intersection of points in the training mesh and reconstructed mesh, a Chamfer distance between the training mesh and reconstructed mesh, an earth mover's distance, and/or another measure of distance, similarity, and/or dissimilarity between the points, edges, shapes, and/or other attributes of the training mesh and reconstructed mesh…training engine 122 uses the error between each training mesh and the corresponding reconstructed mesh to update the parameters of encoder 202 and decoder 206. For example, training engine 122 may backpropagate the error across the layers and/or parameters of decoder 206 so that decoder 206 learns to decode the reconstructed mesh from the sampled output of encoder 202. Training engine 122 may also backpropagate the error across the layers and/or parameters of encoder 202 so that encoder 202 learns to produce a consistent latent vector representation (e.g., mean and standard deviation vectors) for different orderings of points in the corresponding training mesh).
Regarding claim 13, Gallo, in view of Carr, and further in view of Apte teaches the method according to claim 9, Gallo discloses further comprising: generating the sampled matrix based on the plurality of samples (Paragraphs [0027]-[0029]: training engine 122 trains a different encoder that is not in VAE 200 to generate intermediate representations of ordered sets of points and/or other parameterizations of training meshes 208. In one embodiment, training engine 122 averages and/or otherwise aggregates the intermediate representations into a standardized value in the latent space, such that the standardized value defines the mean and standard deviation from which the output of encoder 202 is sampled…training engine 122 inputs the output sampled using the standardized value into decoder 206 to obtain reconstructed meshes 210 as output of decoder 206. Training engine 122 also computes an error between each training mesh and a corresponding reconstructed mesh as the minimum error between an ordering of points in the training mesh inputted into encoder 202 and the outputted ordering of points from decoder 206. For example, training engine 122 may compute the minimum error based on the intersection of points in the training mesh and reconstructed mesh, a Chamfer distance between the training mesh and reconstructed mesh, an earth mover's distance, and/or another measure of distance, similarity, and/or dissimilarity between the points, edges, shapes, and/or other attributes of the training mesh and reconstructed mesh…training engine 122 uses the error between each training mesh and the corresponding reconstructed mesh to update the parameters of encoder 202 and decoder 206. For example, training engine 122 may backpropagate the error across the layers and/or parameters of decoder 206 so that decoder 206 learns to decode the reconstructed mesh from the sampled output of encoder 202. Training engine 122 may also backpropagate the error across the layers and/or parameters of encoder 202 so that encoder 202 learns to produce a consistent latent vector representation (e.g., mean and standard deviation vectors) for different orderings of points in the corresponding training mesh; Paragraphs [0080]-[0084]: Processing operations specific to any of the functional execution units may be pipelined, which enables a new instruction to be issued for execution before a previous instruction has completed execution. Any combination of functional execution units within a given SM 710 may be provided. In various embodiments, the functional execution units may be configured to support a variety of different operations including integer and floating point arithmetic (e.g., addition and multiplication), comparison operations, Boolean operations (AND, OR, 50R), bit-shifting, and computation of various algebraic functions (e.g., planar interpolation and trigonometric, exponential, and logarithmic functions, etc.). Advantageously, the same functional execution unit can be configured to perform different operations… the matrix multiply inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D may be 16-bit floating point or 32-bit floating point matrices. Tensor Cores operate on 16-bit floating point input data with 32-bit floating point accumulation. The 16-bit floating point multiply requires 64 operations and results in a full precision product that is then accumulated using 32-bit floating point addition with the other intermediate products for a 4×4×4 matrix multiply. In practice, Tensor Cores are used to perform much larger two-dimensional or higher dimensional matrix operations, built up from these smaller elements. An API, such as CUDA 9 C++ API, exposes specialized matrix load, matrix multiply and accumulate, and matrix store operations to efficiently use tensor cores from a CUDA-C++ program. At the CUDA level, the warp-level interface assumes 16×16 size matrices spanning all 32 threads of the warp).
Regarding claim 14, Gallo, in view of Carr, and further in view of Apte teaches the method according to claim 13, Gallo discloses further comprising: representing the sampled matrix as a corresponding 3D mesh (Paragraph [0055]: Training engine 122 then trains 406 a decoder in the VAE to reconstruct the training mesh from the standardized value. Training engine 122 also optionally trains 408 an encoder in the VAE to generate the standardized value from each ordering of points in the training mesh. For example, training engine 122 may calculate an error between the reconstructed mesh and the training mesh as the minimum error between an ordering of points in the training mesh inputted into the encoder and the outputted ordering of points from the decoder. The error may be based on an intersection of points between the training mesh and reconstructed mesh, a distance metric between the training mesh and reconstructed mesh, and/or other measures of similarity or dissimilarity between the training mesh and reconstructed mesh. Training engine 122 may backpropagate the error across the layers and/or parameters of the decoder so that the decoder learns to reconstruct the training mesh based on the standardized value. Training engine 122 may also backpropagate the error across the layers and/or parameters of the encoder so that the encoder learns to output the standardized value for different orderings of points in the corresponding training mesh).
Regarding claim 16, Gallo, in view of Carr, and further in view of Apte teaches the method according to claim 14, Gallo discloses wherein the first encoder converts the first set of input data into the first set of latent variables (Fig. 2; Paragraph [0018]: training engine 122 generates one or more machine learning models for performing mesh reconstruction using data-driven priors. Each machine learning model may learn priors related to vertices, edges, corners, faces, polygons, surfaces, shapes, and/or other attributes of two-dimensional (2D) and/or three-dimensional (3D) meshes. For example, each machine learning model may include a variational autoencoder (VAE) that learns to convert input meshes into latent vectors and reconstruct the meshes from the latent vectors; Paragraph [0023]: the machine learning model created by training engine 122 includes a VAE 200, which includes an encoder 202 and a decoder 206. For example, training engine 122 may input 2D and/or 3D training meshes 208 into encoder 202, and encoder 202 may “encode” training meshes 208 into latent vectors 204 (e.g., scalars, geometric vectors, tensors, and/or other geometric objects) within a latent space of lower dimensionality than training meshes 208. A decoder 206 in VAE 200 may “decode” latent vectors 204 into higher dimensionality reconstructed meshes 210 that substantially reproduce training meshes 208).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW D SALVUCCI whose telephone number is (571)270-5748. The examiner can normally be reached M-F: 7:30-4:00PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XIAO WU can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW SALVUCCI/Primary Examiner, Art Unit 2613
