DETAILED ACTION
The Office Action is in response to the Applicants' communication filed on September 1, 2026, which amends the independent claims 1, 7 and 13-14, amends the dependent claims 2-6, 8-12, 16 and 19, cancels the dependent claim 15, adds new dependent claim 21, and presents arguments, is hereby acknowledged. Claims 1-14 and 16-21 are currently pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s arguments filed on September 1, 2026, have been fully considered.
Applicant argues that by this response, the independent 13 is hereby amended to add a new limitation “non-transitory” in order to overcome the 35 U.S.C. §101 rejection.
Examiner replies that the 101 rejections to claim 13 and 15 are withdrawn, claim 15 has been canceled.
Applicant argues that by this response, the independent claims 1, 7 and 13-14 are hereby amended to add a new limitation “adjusting, at the rendering device, a three-dimensional position of at least one vertex of the 3D mesh by interpolating the three-dimensional position of the at least one vertex according to the corresponding offset vectors and a playback time moment while preserving the texture as retrieved from the frame N, wherein the playback time moment is between N and N+1, wherein N+ I is a frame number corresponding to a frame N+ I immediately following the frame Nin the vide” in order to overcome the 35 U.S.C. §103 rejection.
Examiner replies that the amended claims with new limitation may overcome the cited portions of the prior arts. However, a newly found art, Vogt, etc. (US 20060161572 A1) teaches that adjusting, at the rendering device, a three-dimensional position of at least one vertex of the 3D mesh by interpolating the three-dimensional position of the at least one vertex according to the corresponding offset vectors (See Vogt: Figs. 1-2, and [0070], “The vertex information of the first static 3D key-model is put into the position, normal, color, and texture attributes of the OpenGL vertex. The information of the vertex of the second static 3D key-model is put into four of the remaining texture coordinates of the same OpenGL vertex. The actual interpolation of the two vertices is achieved in the vertex shader program on the GPU. See FIG. 2, which shows a schematic representation of the usage, in accordance with principles of the present invention, of a GPU vertex shader for the generation of a dynamic 3D object from the input of two different static 3D key-models of the object. The "keyFrameBlend" parameter controls the linear interpolation between both static 3D key-models. See Table 1 below, which provides an example of Cg code for the linear interpolation of static 3D key-models on programmable vertex shaders. TABLE-US-00001 TABLE 1 struct vert { float4 position : POSITION; float3 normal : NORMAL; float3 color : COLOR; float3 texCoord : TEXCOORDO; }; vert main( float4 position1 : POSITION, float4 normal1 : NORMAL float4 color1 : COLOR float4 texCoord1 : TEXCOORD0, float4 position2 : TEXCOORD1, float4 normal2 : TEXCOORD2, float4 color2 : TEXCOORD3, float4 texCoord2 : TEXCOORD4, uniform float keyFrameBlend, uniform float4.times.4 ModelViewProj ) { vert OUT; OUT.position = mul { ModelViewProj, ( lerp(position1, position2, keyFrameBlend ) ) ); OUT.normal = normalize( lerp(normal1, normal2, keyFrameBlend) ) ); OUT.color = lerp(color1, color2, keyFrameBlend ); OUT.texCoord = lerp( texCoord1, texCoord2, keyFrameBlend ); return OUT”; and [0069], “A preferred embodiment of the present invention includes the above-mentioned real-time interpolation of corresponding vertex coordinates in between the static 3D key-models, and the extension of the information for all vertices in each static 3D key-model in the modeling-phase to include other attributes, with GPU acceleration. The real-time interpolation is implemented in a vertex shader program of a programmable GPU. By utilizing the multi-texturing capabilities of current graphics hardware, there is assigned to each vertex the information of two corresponding vertices of two static 3D key-models”) and a playback time moment while preserving the texture as retrieved from the frame N (See Vogt: Figs. 1-2, and [0003], “As used herein, a dynamic 3D object is defined as a three-dimensional virtual object that dynamically changes its shape through deformations or relative motions of its parts. The definition of dynamic can be extended to not only include geometrical changes but also changes in appearance, color, texture, and other attributes of the object. Examples of a dynamic 3D object are a virtual person walking or a virtual heart beating. A static 3D key-model is herein defined as a model of a dynamic 3D object in a fixed pose with a particular set of attributes. Typical representations are static wireframe or triangulated surface models, which can be stored, e.g., in form of VRML files”; [0016], “In accordance with another aspect of the invention, the step of entering first and second vertex information comprises entering attributes, including any of color, texture, and normal vector information”; and [0087], “Several 3D models of that little comic-like figure were created in different poses with Maya and were imported as static 3D key-models into the AR system in accordance with the present invention. The interpolation between any two of those static 3D key-modes, that is, implementation layer (a), is accomplished by the GPU vertex shader program as depicted in FIG. 2 and Table 1. The dynamics in between both static 3D key-models is achieved by slowly increasing the parameter keyFrameBlend from 0.0 to 1.0 at each new rendering process”), wherein the playback time moment is between N and N+1, wherein N+ I is a frame number corresponding to a frame N+ I immediately following the frame Nin the vide (See Vogt: Figs. 1-2, and [0069], “A preferred embodiment of the present invention includes the above-mentioned real-time interpolation of corresponding vertex coordinates in between the static 3D key-models, and the extension of the information for all vertices in each static 3D key-model in the modeling-phase to include other attributes, with GPU acceleration. The real-time interpolation is implemented in a vertex shader program of a programmable GPU. By utilizing the multi-texturing capabilities of current graphics hardware, there is assigned to each vertex the information of two corresponding vertices of two static 3D key-models”; and [0087], “Several 3D models of that little comic-like figure were created in different poses with Maya and were imported as static 3D key-models into the AR system in accordance with the present invention. The interpolation between any two of those static 3D key-modes, that is, implementation layer (a), is accomplished by the GPU vertex shader program as depicted in FIG. 2 and Table 1. The dynamics in between both static 3D key-models is achieved by slowly increasing the parameter keyFrameBlend from 0.0 to 1.0 at each new rendering process”. Note that blending th4 N and N+1 frame according to the same playback time fraction between 0 and 1 while only the vertex positions are adjusted, and this is mapped to the texture preserved.). The remaining arguments of the applicant are mooted in light of the newly found art.
Examiner respectfully further replies that the Applicant's arguments have been fully considered and a new ground of rejections have been made. Accordingly, new grounds of rejection are set forth below. Since the new grounds of rejection are necessitated by Applicant's amendments to the claims, the present action is made final.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-14 and 16-21 are rejected under 35 U.S.C. 103 as being unpatentable over Kim, etc. (US 20200265294 A1) in view of Cower (US 20210182540 A1), further in view of Graziosi, etc. (US 20220108483 A1) and Vogt, etc. (US 20060161572 A1).
Regarding claim 1, Kim teaches that a method for displaying a three-dimensional video comprising a plurality of frames, at a different frame rate from which the video was stored or acquired, the method (See Kim: Figs. 1-2, and [0032], "FIG. 1 is an illustration of an environment 100 in an example implementation that is operable to employ digital systems and techniques as described herein. The illustrated environment 100 includes a computing device 102 connected to a network 104. The computing device 102 may be configured as a desktop computer, a laptop computer, a mobile device (e.g., assuming a handheld configuration such as a tablet or mobile phone), and so forth. Thus, the computing device 102 may range from a full resource device with substantial memory and processor resources (e.g., personal computers, game consoles) to a low-resource device with limited memory and/or processing resources (e.g., mobile devices). Additionally, the computing device 102 may be representative of a plurality of different devices, such as multiple servers utilized by a business to perform operations "over the cloud.""; [0003], "Systems and techniques are described for object animation using generative neural networks. A computing device implements an animation system which receives a digital video having an animated object as an input. The animation system includes a meshing module, a warping module, and a training module. The meshing module obtains a mesh of the animated object from a first frame of the digital video. This mesh has a plurality of vertices which correspond to features of the object in the first frame. The animation system selects a second frame from the digital video which also has the animated object but in a different orientation than in the first frame. For example, the object might be standing with its arms at its sides in the first frame and the object may have its right arm raised with its right elbow bent in the second frame"; and [0036], "FIG. 2 depicts a system 200 in an example implementation showing operation of an animation system 110. As described above, the animation system 110 receives as an input a digital video having frames of an animated object, and the animation system 110 employs the neural network 122 to output a reproduction of the animated object. In more general terms, the animated object in the digital video is an object. In one or more implementations, the neural network 122 can include an encoder to analyze an image of an object and a decoder to predict vertex offsets for vertices of a mesh of the object") comprising:
receiving, at a video reader and decoder, a frame N corresponding to a frame number N in the video (See Kim: Figs. 1-2, and [0024], "The selection module selects a second frame from the digital video which also has the animated object but in a different orientation than in the first frame. In the first frame, the object might be standing with its arms at its sides, but the object may have its right arm raised with its right elbow bent in the second frame. The identification module identifies features of the animated object in the second frame using an encoder of a neural network"; and [0029], "The described systems and techniques improve digital animation by allowing digital animators to transfer an animation of an object to another object. This improvement also significantly increases the efficiency of digital animation. These systems and techniques improve a user experience for digital animators by eliminating the tedious task of animating an object to have features of another animated object. Additionally, the systems and techniques can improve computational efficiency by using neural networks to transfer animation from an object to another object instead of requiring computations to render animation definitions for every object requiring animation". Note that the system receives and processes frames from the digital video sequentially with frame N (such as the first or second frame) in the meshing module using neural networks);
retrieving, at the video reader and decoder, a 3D mesh including a plurality of vertices, a texture, and one or more offset vectors associated with each vertex of the 3D mesh in the frame N (See Kim: Figs. 1-5, and [0025], "The animation system also includes a mapping module and a prediction module. The mapping module maps the identified features of the object in the second frame to the vertices of the combined mesh. Next, the prediction module generates a vertex offset for the mesh using a decoder of the neural network. This vertex offset is a prediction of how the combined mesh is to be deformed to reproduce the animated object as it appears in the second frame"; [0046], "In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video"; and [0052], "FIG. 5 is an illustration depicting a representation 500 of a rendering of a warped mesh as an image of the object. The representation 500 is illustrated to include a rendering module 502 implemented to render an image 504 of a warped mesh. In one example, the rendering module 502 may be a Neural Renderer. The rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 504 of the object 406". Note that the meshing module generate mesh with plurality of vertices corresponding to object features in the first frame (N));
transmitting, to a rendering device, the 3D mesh, the texture, and the offset vectors associated with the frame N (See Kim: Figs. 1-3, and [0035], "According to various implementations, such a machine-learning model uses supervised learning, semi-supervised learning, unsupervised learning, or reinforcement learning. For example, the machine-learning model can include, but is not limited to, clustering, decision trees, support vector machines, linear regression, logistic regression, Bayesian networks, random forest learning, dimensionality reduction algorithms, boosting algorithms, artificial neural networks (e.g., fully-connected neural networks, deep convolutional neural networks, or recurrent neural networks), deep learning, etc. Thus, a machine-learning model makes high-level abstractions in data by generating data-driven predictions or decisions from the known input data. In this manner, the animation system 110 receives a digital video having an animated object as an input, and the animation system 110 employs the neural network 122 to output a reproduction of the animated object. In one or more implementations, the animation system 110 may output the reproduction of the animated object without having any information about the animated object in the digital video"; [0045], "The warping module 212 warps the mesh based on the vertex offset (block 314) and the training module 214 trains the neural network based on a comparison of the warped mesh with the object from the second frame (block 316). For example, the training module 214 may use the loss function 218 to train the neural network to improve the quality of the reproduction of the object in the second frame 116"; and [0046], "In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video". Note that the mesh and vertex offset are passed to the warping and rendering module (neural renderer) to generate the output images);
adjusting, at the rendering device, a three-dimensional position of at least one vertex of the 3D mesh by interpolating the three-dimensional position of the at least one vertex according to the corresponding offset vectors and a playback time moment (See Kim: Figs. 1-2, and [0038], "The animation system 110 is also illustrated to include a selection module 204. The selection module 204 is implemented to select a second frame 116 from the digital video having the object. An identification module 206 identifies features of the object from the second frame 116 as part of the neural network 122. The neural network 122 includes a mapping module 208 and a prediction module 210. When implemented, the mapping module 208 maps the identified features of the object in the second frame 116 to the vertices of the mesh, and the prediction module 210 generates vertex offsets based on this mapping. Finally, a warping module 212 of the animation system 110 warps the mesh based on the vertex offsets to reproduce the object as it appears in the second frame 116"; [0047], "In one or more implementations, regularization may help avoid overfitting as part of training the neural network 122. Regularization is a form of regression which regulates or minimizes coefficients of the lost function 218, and this helps prevent the neural network 122 from fitting noise in training. This also improves generalization in machine learning. For example, Laplacian regularization may be used to calculate distances between a Laplacian coordinate of a vertex before and after warping the mesh based on the predicted vertex offset. In another example, edge-length regularization may be used to penalize differences in edge lengths before and after warping the mesh"; Figs. 4 and 9, and [0064], "In one example, the animation system 904 may include an encoder and a decoder of a neural network to predict vertex offsets for warping the combined mesh 902 to match features of the object 406 from the second frame 116. In another example, the animation system 904 may include a Neural Renderer to receive faces, textures, initial vertex positions, and the predicted vertex offsets to warp the combined mesh 902 and render the warped mesh as an image of the object 406. For example, the neural network may be trained using a loss function based on a comparison of the rendered image of the object and the object 406. In an example, the animation system 904 may include a generator and a discriminator of a generative adversarial network. For example, the generator may generate a refined image from the rendered image and the discriminator may receive the refined image and a ground truth to determine whether the refined image is real or fake. In one or more implementations, the generator can be trained by backpropagation of a result of the discriminator's determination whether the refined image is considered real or fake. In this way, the animation system 904 may include a Neural Renderer to render the warped mesh as an image of the object 406 such that the image of the object is a reproduction of the object 406, and the animation system may include a generative adversarial network to refine the image of the object at a high frequency to correct the warping effects that make the rendered image of the object distinguishable from the object 406. In other words, the refined image can correct high frequency features of the rendered image which make it distinguishable from the object 406. Thus, the animation system 904 can transfer the motions of an object in a digital video to another object without needing any information about the definitions describing the motions of the object in the video". Note that warping adjusts vertices by offsets to match target appearance, and warps mesh based on the vertex offset to reproduce objects in the second frame N+1, enabling intermediate deformations for animation sequences) while preserving the texture as retrieved from the frame N, wherein the playback time moment is between N and N+1, wherein N+1 is a frame number corresponding to a frame N+1 immediately following the frame N in the video; and
rendering the adjusted 3D mesh and the texture according to one or more camera parameters associated with a display device, wherein the display device is configured to display the video (See Kim: Figs. 4-6, and [0052], "FIG. 5 is an illustration depicting a representation 500 of a rendering of a warped mesh as an image of the object. The representation 500 is illustrated to include a rendering module 502 implemented to render an image 504 of a warped mesh. In one example, the rendering module 502 may be a Neural Renderer. The rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 504 of the object 406"; [0055], "In one or more implementations, the animation system 110 generates the mesh 404 of the object as it appears in the first frame 114 from the first mesh 604 and the second mesh 606 using the layering dimension which is an output of the layering module 602. In one example, the output of the layering module 602 is an input to the rendering module 502. Thus, the rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 608 of the object 406"; and Figs. 1-2, and [0033], "The illustrated environment 100 also includes a display device 106 that is communicatively coupled to the computing device 102 via a wired or a wireless connection. A variety of device configurations may be used to implement the computing device 102 and/or the display device 106. The computing device 102 includes a storage device 108 and an animation system 110. The storage device 108 is illustrated to include digital content 112. An example of the digital content 112 is a digital video such as a digital video having an animated object interacting with an environment. The digital video is illustrated as frames 114-120 shown on the display device 106. The frames 114-120 are snapshots of the object as it moves in the digital video. Thus, the digital video may be observed as a series of frames 114-120 on a display of the display device 106. For example, in frame 114, the object is standing with its arms at its side, but in frame 116, the object's right arm is raised as if to direct the viewer's attention to something of interest. The object appears relaxed with its left hand partially in its pants pocket in frame 118, and the object appears to be giving a thumb's up with its left hand in frame 120. Thus, when frames 114-120 are displayed in succession, the object's various poses in each of the frames 114-120 may appear as the animation of the object in the digital video". Note that the neural 3D renderer renders the warped mesh, and outputs the rendered images to the display device).
However, Kim fails to explicitly disclose that at a different frame rate from which the video was stored or acquired; a three-dimensional position of at least one vertex of the 3D mesh by interpolating the three-dimensional position of the at least one vertex ; while preserving the texture as retrieved from the frame N; wherein the playback time moment is between N and N+1, wherein N+1 is a frame number corresponding to a frame N+1 immediately following the frame N in the video; and according to one or more camera parameters associated with a display device.
However, Cower teaches that at a different frame rate from which the video was stored or acquired (See Cower: Fig. 2, and [0050], "In some embodiments, the decoder 202 receives a set of encoded video frames via the communication unit 239. The decoder 202 decodes the video frames, for example, by decrypting (e.g., when the video is encrypted) and decompressing the encoded video frames. Once the decoder 202 decodes the video frames, in conventional video processing, the video frames are typically rendered for display. However, per techniques described herein, the decoder 202 skips scheduling of the video frames for presentation, e.g., when the frame rate of the received video is lower than a frame rate for the video application 103. As a result, instead of scheduling the video frames for presentation, the decoder 202 provides each of the decoded video frames to the video analyzer 204. For example, the set of video frames includes a first video frame and a second video frame, where the second video frame is subsequent to the first video frame, e.g., is associated with a timestamp that is later than a timestamp associated with the first video frame, such that the second video frame occupies a later position in a chronological sequence of frames than the first video frame. In some embodiments, the second video frame directly follows the first video frame, for example, the two frames are identified for sequential display with no other frames in between"); and
wherein the playback time moment is between N and N+1, wherein N+1 is a frame number corresponding to a frame N+1 immediately following the frame N in the video (Se Cower: Fig. 8, and [0086], "Frame sequence 820 includes upsampled video and includes frames 811,822,823,824,815,826,827,828, and 819, which are displayed at a higher framerate. From the received video stream that includes frames 811,815, and 819, additional frames are obtained by interpolation, as described herein. The additional frames in frame sequence 820 are 822, 823, 824, 826, 827, and 828, and are obtained by interpolation and added to the sequence according to the techniques described herein. As a result of using interpolated frames, the frame sequence 820 can be displayed at a higher framerate (since intermediate frames 822-824 and 826-828 are available) with no jumpiness while the bandwidth utilized to receive the video remains the same as that for frame sequence 810"; and [0089], "In some embodiments, the number of intermediate frames that are generated between the first frame and the second frame is based on a difference in the frame rate of the received video and the frame rate at which the video is to be played back. For example, if the video is received at 10 FPS and is to be played by at 30 FPS, 2 intermediate frames are generated for each pair of consecutive frames of the video. In this instance, if the original video has 100 frames (corresponding to 10 seconds of video), 200 intermediate frames are generated to obtain a video of 300 frames which can be played back at 30 FPS for 10 seconds of video. The frame rate for video playback may be selected based on various factors, e.g., the refresh rate of display 241, the computational capabilities of device 200, the rate of motion in the received video, etc.").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Kim to have at a different frame rate from which the video was stored or acquired; wherein the playback time moment is between N and N+1, wherein N+1 is a frame number corresponding to a frame N+1 immediately following the frame N in the video as taught by Cower in order to use reduced computational power to display video with a perceived higher frame rate (See Cower: Fig. 1, and [0029], "The various embodiments described below have several advantages. First, the processing is performed by the user device that displays the video. As a result, the video application uses reduced computational power to display video with a perceived higher frame rate. Second, the embodiments also provide higher frame rates than received video frame rate, even when the video is received with the use of end-to-end encryption between a sender device and a receiver device that displays the video. Third, the interpolation is computationally efficient because the structure of the video frames is interpolated and not the texture"). Kim teaches a method and system that may animate the objects in the 3D video playback with dynamic temporal vertex interpolation using deformation and or offset vector via neural networks for frame rate conversion; while Cower teaches a system and method that explicitly adjust the number of frames and the time amount (time stamps) during playback when the frame rate and display refresh rate are different. Therefore, it is obvious to one of ordinary skill in the art to modify Kim by Cower to adjust the 3D mesh based on the frame rate and playback refresh rate. The motivation to modify Kim by Cower is "Use of known technique to improve similar devices (methods, or products) in the same way".
However, Kim, modified by Cower, fails to explicitly disclose that a three-dimensional position of at least one vertex of the 3D mesh by interpolating the three-dimensional position of the at least one vertex ; while preserving the texture as retrieved from the frame N; and according to one or more camera parameters associated with a display device.
However, Graziosi teaches that according to one or more camera parameters associated with a display device (See (Graziosi: Fig. 1, and [0041], "To address voxelization, mesh scaling and offset information is sent in Atlas Adaptation Parameter Set (AAPS). Available camera parameters are able to be used. Alternatively, new syntax elements are introduced for voxelization (where only scaling and offset are used). The following is exemplary syntax").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Kim to have according to one or more camera parameters associated with a display device as taught by Graziosi in order to enable progressive mesh coding by separating sets of vertices in layers and generating levels of detail for mesh connectivity (See Graziosi: Fig. 1, and [0099], "In operation, the video based mesh compression method enables more efficient and more accurate 3D content encoding compared to previous implementations"). Kim teaches a method and system that may animate the objects in the 3D video playback with dynamic temporal vertex interpolation using deformation and or offset vector via neural networks for frame rate conversion; while Graziosi teaches a video based mesh compression method and system that use the camera parameters together with the voxelization, mesh scaling and offset information in video mesh coding and decoding in order to enable progressive encoding more efficient. Therefore, it is obvious to one of ordinary skill in the art to modify Kim by Graziosi to use the camera parameters in 3D video animation rendering (actually camera/view parameters are inherent in 3D rendering). The motivation to modify Kim by Graziosi is "Use of known technique to improve similar devices (methods, or products) in the same way".
However, Kim, modified by Cower and Graziosi, fails to explicitly disclose that a three-dimensional position of at least one vertex of the 3D mesh by interpolating the three-dimensional position of the at least one vertex ; and while preserving the texture as retrieved from the frame N.
However, Vogt teaches that adjusting, at the rendering device, a three-dimensional position of at least one vertex of the 3D mesh by interpolating the three-dimensional position of the at least one vertex according to the corresponding offset vectors (See Vogt: Figs. 1-2, and [0070], “The vertex information of the first static 3D key-model is put into the position, normal, color, and texture attributes of the OpenGL vertex. The information of the vertex of the second static 3D key-model is put into four of the remaining texture coordinates of the same OpenGL vertex. The actual interpolation of the two vertices is achieved in the vertex shader program on the GPU. See FIG. 2, which shows a schematic representation of the usage, in accordance with principles of the present invention, of a GPU vertex shader for the generation of a dynamic 3D object from the input of two different static 3D key-models of the object. The "keyFrameBlend" parameter controls the linear interpolation between both static 3D key-models. See Table 1 below, which provides an example of Cg code for the linear interpolation of static 3D key-models on programmable vertex shaders. TABLE-US-00001 TABLE 1 struct vert { float4 position : POSITION; float3 normal : NORMAL; float3 color : COLOR; float3 texCoord : TEXCOORDO; }; vert main( float4 position1 : POSITION, float4 normal1 : NORMAL float4 color1 : COLOR float4 texCoord1 : TEXCOORD0, float4 position2 : TEXCOORD1, float4 normal2 : TEXCOORD2, float4 color2 : TEXCOORD3, float4 texCoord2 : TEXCOORD4, uniform float keyFrameBlend, uniform float4.times.4 ModelViewProj ) { vert OUT; OUT.position = mul { ModelViewProj, ( lerp(position1, position2, keyFrameBlend ) ) ); OUT.normal = normalize( lerp(normal1, normal2, keyFrameBlend) ) ); OUT.color = lerp(color1, color2, keyFrameBlend ); OUT.texCoord = lerp( texCoord1, texCoord2, keyFrameBlend ); return OUT”; and [0069], “A preferred embodiment of the present invention includes the above-mentioned real-time interpolation of corresponding vertex coordinates in between the static 3D key-models, and the extension of the information for all vertices in each static 3D key-model in the modeling-phase to include other attributes, with GPU acceleration. The real-time interpolation is implemented in a vertex shader program of a programmable GPU. By utilizing the multi-texturing capabilities of current graphics hardware, there is assigned to each vertex the information of two corresponding vertices of two static 3D key-models”); and
while preserving the texture as retrieved from the frame N (See Vogt: Figs. 1-2, and [0003], “As used herein, a dynamic 3D object is defined as a three-dimensional virtual object that dynamically changes its shape through deformations or relative motions of its parts. The definition of dynamic can be extended to not only include geometrical changes but also changes in appearance, color, texture, and other attributes of the object. Examples of a dynamic 3D object are a virtual person walking or a virtual heart beating. A static 3D key-model is herein defined as a model of a dynamic 3D object in a fixed pose with a particular set of attributes. Typical representations are static wireframe or triangulated surface models, which can be stored, e.g., in form of VRML files”; [0016], “In accordance with another aspect of the invention, the step of entering first and second vertex information comprises entering attributes, including any of color, texture, and normal vector information”; and [0087], “Several 3D models of that little comic-like figure were created in different poses with Maya and were imported as static 3D key-models into the AR system in accordance with the present invention. The interpolation between any two of those static 3D key-modes, that is, implementation layer (a), is accomplished by the GPU vertex shader program as depicted in FIG. 2 and Table 1. The dynamics in between both static 3D key-models is achieved by slowly increasing the parameter keyFrameBlend from 0.0 to 1.0 at each new rendering process”), wherein the playback time moment is between N and N+1, wherein N+ I is a frame number corresponding to a frame N+ I immediately following the frame Nin the vide (See Vogt: Figs. 1-2, and [0069], “A preferred embodiment of the present invention includes the above-mentioned real-time interpolation of corresponding vertex coordinates in between the static 3D key-models, and the extension of the information for all vertices in each static 3D key-model in the modeling-phase to include other attributes, with GPU acceleration. The real-time interpolation is implemented in a vertex shader program of a programmable GPU. By utilizing the multi-texturing capabilities of current graphics hardware, there is assigned to each vertex the information of two corresponding vertices of two static 3D key-models”; and [0087], “Several 3D models of that little comic-like figure were created in different poses with Maya and were imported as static 3D key-models into the AR system in accordance with the present invention. The interpolation between any two of those static 3D key-modes, that is, implementation layer (a), is accomplished by the GPU vertex shader program as depicted in FIG. 2 and Table 1. The dynamics in between both static 3D key-models is achieved by slowly increasing the parameter keyFrameBlend from 0.0 to 1.0 at each new rendering process”. Note that blending th4 N and N+1 frame according to the same playback time fraction between 0 and 1 while only the vertex positions are adjusted, and this is mapped to the texture preserved.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Kim to have a three-dimensional position of at least one vertex of the 3D mesh by interpolating the three-dimensional position of the at least one vertex ; and while preserving the texture as retrieved from the frame N as taught by Vogt in order to provide appropriate occlusion of the virtual objects by the objects from a real world and appropriate occlusion of the objects from a real world by the virtual objects (See Vogt: Fig. 1, and [0049], " In accordance with another aspect of the invention, a system includes a system for providing appropriate occlusion of the virtual objects by the objects from a real world and appropriate occlusion of the objects from a real world by the virtual objects"). Kim teaches a method and system that may animate the objects in the 3D video playback with dynamic temporal vertex interpolation using deformation and or offset vector via neural networks for frame rate conversion; while Vogt teaches a video based mesh compression method and system that treat the texture, especially texture coordinates as one of the vertex attributes that are interpolated between two static 3D key-models, just like color, textures, and other attributes. Therefore, it is obvious to one of ordinary skill in the art to modify Kim by Vogt to use the linear interpolation of vertex position in the vertex shader controlled by to a scalar blender factor between 0 and 1 and to preserve the texture of the frame N.. The motivation to modify Kim by Vogt is "Use of known technique to improve similar devices (methods, or products) in the same way".
Regarding claim 2, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 1 as outlined above. Further, Kim teaches that the method of claim 1, wherein the offset vector are determined by adjusting the offset vectors to reduce a measurement representing a mismatch between the frame N and the frame N+I (See Kim: Figs. 1-2, and [0039], " The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections ").
Regarding claim 3, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 1 as outlined above. Further, Kim teaches that the method of claim 1, wherein the offset vectors are predicted by a neural network trained on a plurality of pairs of frames of the video, wherein a training objective of the neural network is a measurement representing a mismatch between an offset-adjusted first frame of a pair of frames from the plurality of frames and a second frame of the pair of frames (See Kim: Figs. 1-4, and [0004], " The animation system identifies features of the object in the second frame using an encoder of a neural network and maps the identified features to the vertices of the mesh using a decoder of the neural network. The mapping generates a vertex offset for the mesh which predicts how the mesh is to be deformed to reproduce the animated object as it appears in the second frame. The computing device implements the warping module to warp the mesh based on the vertex offset and then a Neural Renderer renders the warped mesh as an image. The training module compares the rendered image to the second frame and trains the neural network with a loss function based on the comparison"; [0039], “The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections”; and [0049], “FIG. 4 is an illustration depicting a representation 400 of a mesh from a first frame of a digital video and an object from a second frame of the digital video. As shown in FIG. 4, the representation 400 includes a digital video 402 having frames 114-120 that include an animated object. The representation 400 also includes a mesh 404 of the object as it appears in the first frame 114 and the object 406 as it appears in the second frame 116. The mesh 404 includes a plurality of vertices which correspond to features of the object in the first frame 114, and some features of the object 406 are different than features of the object in the first frame 114. For example, the object in the first frame 114 may be standing with its hands at its sides and the object 406 may have its right arm raised as if to direct the viewer's attention to something of interest”).
Regarding claim 4, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 1 as outlined above. Further, Kim teaches that the method of claim 1, wherein the offset vectors are determined such that rasterized projections of the offset vectors on a set of two-dimensional views of a scene in the video match one or more optical flow fields between the frame N and the frame N+1 of the scene for the two-dimensional views (See Kim: Fig. 9, and [0064], "In one example, the animation system 904 may include an encoder and a decoder of a neural network to predict vertex offsets for warping the combined mesh 902 to match features of the object 406 from the second frame 116. In another example, the animation system 904 may include a Neural Renderer to receive faces, textures, initial vertex positions, and the predicted vertex offsets to warp the combined mesh 902 and render the warped mesh as an image of the object 406. For example, the neural network may be trained using a loss function based on a comparison of the rendered image of the object and the object 406. In an example, the animation system 904 may include a generator and a discriminator of a generative adversarial network. For example, the generator may generate a refined image from the rendered image and the discriminator may receive the refined image and a ground truth to determine whether the refined image is real or fake. In one or more implementations, the generator can be trained by backpropagation of a result of the discriminator's determination whether the refined image is considered real or fake. In this way, the animation system 904 may include a Neural Renderer to render the warped mesh as an image of the object 406 such that the image of the object is a reproduction of the object 406, and the animation system may include a generative adversarial network to refine the image of the object at a high frequency to correct the warping effects that make the rendered image of the object distinguishable from the object 406. In other words, the refined image can correct high frequency features of the rendered image which make it distinguishable from the object 406. Thus, the animation system 904 can transfer the motions of an object in a digital video to another object without needing any information about the definitions describing the motions of the object in the video").
Regarding claim 5, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 1 as outlined above. Further, Kim teaches that the method of claim 1, wherein the offset vectors are determined by:
comparing a rendered image of an offset-adjusted frame N, defined as the frame N with the offset vectors added to the vertices of the 3D mesh, with a rendered image of the frame N+ l (See Kim: Figs. 1-2, and [0046], "In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video");
computing a loss function based on the comparison (See Kim: Figs. 1-2, and [0039], "The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network "learns" how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections"); and
adjusting the offset vectors based on the loss function (See Kim: Figs. 1-2, and [0039], "The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network "learns" how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections"; and Fig. 7, and [0059], "The animation system 702 includes a generative adversarial network which is illustrated as a generator module 710 and a discriminator module 712. Once trained, the generative adversarial network is configured to generate refined images that are indistinguishable from the object 406 in the second frame 116. The generator module 710 receives the rendered image 608 from the rendering module 708 as an input, and the generator module 710 outputs a refined image of the object 406 in the second frame 116. In this way, the generator module 710 is implemented to generate refined image candidates which are then evaluated by the discriminator module 712, e.g., to determine whether the candidates are real or fake 714. A goal of the generator module 710 is therefore to generate a candidate that is considered real by the discriminator module 712, e.g., through comparison to a ground truth. Accordingly, the generator module 710 is trained as part of adversarial back-and-forth communication between the generative and discriminator modules in order to generate "real" candidates". Note that the training continues until the refined images sufficiently matching the target object images).
Regarding claim 6, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 5 as outlined above. Further, Kim teaches that the method of claim 5 wherein the offset vectors are predicted by a neural network trained using the computed loss function (See Kim: Figs. 1-3, and [0045], " The warping module 212 warps the mesh based on the vertex offset (block 314) and the training module 214 trains the neural network based on a comparison of the warped mesh with the object from the second frame (block 316). For example, the training module 214 may use the loss function 218 to train the neural network to improve the quality of the reproduction of the object in the second frame 116").
Regarding claim 7, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 1 as outlined above. Further, Kim, Cower, Graziosi and Vogt teach that a system for displaying a three-dimensional video comprising a plurality of frames, at a different frame rate from which the video was stored or acquired (See Cower: Fig. 2, and [0050], "In some embodiments, the decoder 202 receives a set of encoded video frames via the communication unit 239. The decoder 202 decodes the video frames, for example, by decrypting (e.g., when the video is encrypted) and decompressing the encoded video frames. Once the decoder 202 decodes the video frames, in conventional video processing, the video frames are typically rendered for display. However, per techniques described herein, the decoder 202 skips scheduling of the video frames for presentation, e.g., when the frame rate of the received video is lower than a frame rate for the video application 103. As a result, instead of scheduling the video frames for presentation, the decoder 202 provides each of the decoded video frames to the video analyzer 204. For example, the set of video frames includes a first video frame and a second video frame, where the second video frame is subsequent to the first video frame, e.g., is associated with a timestamp that is later than a timestamp associated with the first video frame, such that the second video frame occupies a later position in a chronological sequence of frames than the first video frame. In some embodiments, the second video frame directly follows the first video frame, for example, the two frames are identified for sequential display with no other frames in between"), the system (See Kim: Figs. 1-2, and [0032], "FIG. 1 is an illustration of an environment 100 in an example implementation that is operable to employ digital systems and techniques as described herein. The illustrated environment 100 includes a computing device 102 connected to a network 104. The computing device 102 may be configured as a desktop computer, a laptop computer, a mobile device (e.g., assuming a handheld configuration such as a tablet or mobile phone), and so forth. Thus, the computing device 102 may range from a full resource device with substantial memory and processor resources (e.g., personal computers, game consoles) to a low-resource device with limited memory and/or processing resources (e.g., mobile devices). Additionally, the computing device 102 may be representative of a plurality of different devices, such as multiple servers utilized by a business to perform operations "over the cloud.""; [0003], "Systems and techniques are described for object animation using generative neural networks. A computing device implements an animation system which receives a digital video having an animated object as an input. The animation system includes a meshing module, a warping module, and a training module. The meshing module obtains a mesh of the animated object from a first frame of the digital video. This mesh has a plurality of vertices which correspond to features of the object in the first frame. The animation system selects a second frame from the digital video which also has the animated object but in a different orientation than in the first frame. For example, the object might be standing with its arms at its sides in the first frame and the object may have its right arm raised with its right elbow bent in the second frame"; and [0036], "FIG. 2 depicts a system 200 in an example implementation showing operation of an animation system 110. As described above, the animation system 110 receives as an input a digital video having frames of an animated object, and the animation system 110 employs the neural network 122 to output a reproduction of the animated object. In more general terms, the animated object in the digital video is an object. In one or more implementations, the neural network 122 can include an encoder to analyze an image of an object and a decoder to predict vertex offsets for vertices of a mesh of the object") comprising:
a video reader and decoder configured to receive a frame N corresponding to a frame number N in the video (See Kim: Figs. 1-2, and [0024], "The selection module selects a second frame from the digital video which also has the animated object but in a different orientation than in the first frame. In the first frame, the object might be standing with its arms at its sides, but the object may have its right arm raised with its right elbow bent in the second frame. The identification module identifies features of the animated object in the second frame using an encoder of a neural network"; and [0029], "The described systems and techniques improve digital animation by allowing digital animators to transfer an animation of an object to another object. This improvement also significantly increases the efficiency of digital animation. These systems and techniques improve a user experience for digital animators by eliminating the tedious task of animating an object to have features of another animated object. Additionally, the systems and techniques can improve computational efficiency by using neural networks to transfer animation from an object to another object instead of requiring computations to render animation definitions for every object requiring animation". Note that the system receives and processes frames from the digital video sequentially with frame N (such as the first or second frame) in the meshing module using neural networks);
the video reader and decoder further configured to retrieve a 3D mesh including a plurality of vertices, a texture and one or more offset vectors associated with each vertex of the 3D mesh in the frame N (See Kim: Figs. 1-5, and [0025], "The animation system also includes a mapping module and a prediction module. The mapping module maps the identified features of the object in the second frame to the vertices of the combined mesh. Next, the prediction module generates a vertex offset for the mesh using a decoder of the neural network. This vertex offset is a prediction of how the combined mesh is to be deformed to reproduce the animated object as it appears in the second frame"; [0046], "In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video"; and [0052], "FIG. 5 is an illustration depicting a representation 500 of a rendering of a warped mesh as an image of the object. The representation 500 is illustrated to include a rendering module 502 implemented to render an image 504 of a warped mesh. In one example, the rendering module 502 may be a Neural Renderer. The rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 504 of the object 406". Note that the meshing module generate mesh with plurality of vertices corresponding to object features in the first frame (N)), and
transmit, to a rendering device, the 3D mesh, the texture, and the offset vectors associated with the frame N (See Kim: Figs. 1-3, and [0035], "According to various implementations, such a machine-learning model uses supervised learning, semi-supervised learning, unsupervised learning, or reinforcement learning. For example, the machine-learning model can include, but is not limited to, clustering, decision trees, support vector machines, linear regression, logistic regression, Bayesian networks, random forest learning, dimensionality reduction algorithms, boosting algorithms, artificial neural networks (e.g., fully-connected neural networks, deep convolutional neural networks, or recurrent neural networks), deep learning, etc. Thus, a machine-learning model makes high-level abstractions in data by generating data-driven predictions or decisions from the known input data. In this manner, the animation system 110 receives a digital video having an animated object as an input, and the animation system 110 employs the neural network 122 to output a reproduction of the animated object. In one or more implementations, the animation system 110 may output the reproduction of the animated object without having any information about the animated object in the digital video"; [0045], "The warping module 212 warps the mesh based on the vertex offset (block 314) and the training module 214 trains the neural network based on a comparison of the warped mesh with the object from the second frame (block 316). For example, the training module 214 may use the loss function 218 to train the neural network to improve the quality of the reproduction of the object in the second frame 116"; and [0046], "In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video". Note that the mesh and vertex offset are passed to the warping and rendering module (neural renderer) to generate the output images);
the rendering device configured to adjust a three-dimensional position of at least one vertex of the 3D mesh by interpolating the three-dimensional position of the at least one vertex according to the corresponding offset vectors (See Vogt: Figs. 1-2, and [0070], “The vertex information of the first static 3D key-model is put into the position, normal, color, and texture attributes of the OpenGL vertex. The information of the vertex of the second static 3D key-model is put into four of the remaining texture coordinates of the same OpenGL vertex. The actual interpolation of the two vertices is achieved in the vertex shader program on the GPU. See FIG. 2, which shows a schematic representation of the usage, in accordance with principles of the present invention, of a GPU vertex shader for the generation of a dynamic 3D object from the input of two different static 3D key-models of the object. The "keyFrameBlend" parameter controls the linear interpolation between both static 3D key-models. See Table 1 below, which provides an example of Cg code for the linear interpolation of static 3D key-models on programmable vertex shaders. TABLE-US-00001 TABLE 1 struct vert { float4 position : POSITION; float3 normal : NORMAL; float3 color : COLOR; float3 texCoord : TEXCOORDO; }; vert main( float4 position1 : POSITION, float4 normal1 : NORMAL float4 color1 : COLOR float4 texCoord1 : TEXCOORD0, float4 position2 : TEXCOORD1, float4 normal2 : TEXCOORD2, float4 color2 : TEXCOORD3, float4 texCoord2 : TEXCOORD4, uniform float keyFrameBlend, uniform float4.times.4 ModelViewProj ) { vert OUT; OUT.position = mul { ModelViewProj, ( lerp(position1, position2, keyFrameBlend ) ) ); OUT.normal = normalize( lerp(normal1, normal2, keyFrameBlend) ) ); OUT.color = lerp(color1, color2, keyFrameBlend ); OUT.texCoord = lerp( texCoord1, texCoord2, keyFrameBlend ); return OUT”; and [0069], “A preferred embodiment of the present invention includes the above-mentioned real-time interpolation of corresponding vertex coordinates in between the static 3D key-models, and the extension of the information for all vertices in each static 3D key-model in the modeling-phase to include other attributes, with GPU acceleration. The real-time interpolation is implemented in a vertex shader program of a programmable GPU. By utilizing the multi-texturing capabilities of current graphics hardware, there is assigned to each vertex the information of two corresponding vertices of two static 3D key-models”) and a playback time moment while preserving the texture as retrieved from the frame N (See Vogt: Figs. 1-2, and [0003], “As used herein, a dynamic 3D object is defined as a three-dimensional virtual object that dynamically changes its shape through deformations or relative motions of its parts. The definition of dynamic can be extended to not only include geometrical changes but also changes in appearance, color, texture, and other attributes of the object. Examples of a dynamic 3D object are a virtual person walking or a virtual heart beating. A static 3D key-model is herein defined as a model of a dynamic 3D object in a fixed pose with a particular set of attributes. Typical representations are static wireframe or triangulated surface models, which can be stored, e.g., in form of VRML files”; [0016], “In accordance with another aspect of the invention, the step of entering first and second vertex information comprises entering attributes, including any of color, texture, and normal vector information”; and [0087], “Several 3D models of that little comic-like figure were created in different poses with Maya and were imported as static 3D key-models into the AR system in accordance with the present invention. The interpolation between any two of those static 3D key-modes, that is, implementation layer (a), is accomplished by the GPU vertex shader program as depicted in FIG. 2 and Table 1. The dynamics in between both static 3D key-models is achieved by slowly increasing the parameter keyFrameBlend from 0.0 to 1.0 at each new rendering process”), wherein the playback time moment is between N and N+1, wherein N+ I is a frame number corresponding to a frame N+ I immediately following the frame Nin the vide (See Vogt: Figs. 1-2, and [0069], “A preferred embodiment of the present invention includes the above-mentioned real-time interpolation of corresponding vertex coordinates in between the static 3D key-models, and the extension of the information for all vertices in each static 3D key-model in the modeling-phase to include other attributes, with GPU acceleration. The real-time interpolation is implemented in a vertex shader program of a programmable GPU. By utilizing the multi-texturing capabilities of current graphics hardware, there is assigned to each vertex the information of two corresponding vertices of two static 3D key-models”; and [0087], “Several 3D models of that little comic-like figure were created in different poses with Maya and were imported as static 3D key-models into the AR system in accordance with the present invention. The interpolation between any two of those static 3D key-modes, that is, implementation layer (a), is accomplished by the GPU vertex shader program as depicted in FIG. 2 and Table 1. The dynamics in between both static 3D key-models is achieved by slowly increasing the parameter keyFrameBlend from 0.0 to 1.0 at each new rendering process”. Note that blending th4 N and N+1 frame according to the same playback time fraction between 0 and 1 while only the vertex positions are adjusted, and this is mapped to the texture preserved.),
wherein the playback time moment is between N and N+1 (See Kim: Figs. 1-2, and [0038], "The animation system 110 is also illustrated to include a selection module 204. The selection module 204 is implemented to select a second frame 116 from the digital video having the object. An identification module 206 identifies features of the object from the second frame 116 as part of the neural network 122. The neural network 122 includes a mapping module 208 and a prediction module 210. When implemented, the mapping module 208 maps the identified features of the object in the second frame 116 to the vertices of the mesh, and the prediction module 210 generates vertex offsets based on this mapping. Finally, a warping module 212 of the animation system 110 warps the mesh based on the vertex offsets to reproduce the object as it appears in the second frame 116"; [0047], "In one or more implementations, regularization may help avoid overfitting as part of training the neural network 122. Regularization is a form of regression which regulates or minimizes coefficients of the lost function 218, and this helps prevent the neural network 122 from fitting noise in training. This also improves generalization in machine learning. For example, Laplacian regularization may be used to calculate distances between a Laplacian coordinate of a vertex before and after warping the mesh based on the predicted vertex offset. In another example, edge-length regularization may be used to penalize differences in edge lengths before and after warping the mesh"; Figs. 4 and 9, and [0064], "In one example, the animation system 904 may include an encoder and a decoder of a neural network to predict vertex offsets for warping the combined mesh 902 to match features of the object 406 from the second frame 116. In another example, the animation system 904 may include a Neural Renderer to receive faces, textures, initial vertex positions, and the predicted vertex offsets to warp the combined mesh 902 and render the warped mesh as an image of the object 406. For example, the neural network may be trained using a loss function based on a comparison of the rendered image of the object and the object 406. In an example, the animation system 904 may include a generator and a discriminator of a generative adversarial network. For example, the generator may generate a refined image from the rendered image and the discriminator may receive the refined image and a ground truth to determine whether the refined image is real or fake. In one or more implementations, the generator can be trained by backpropagation of a result of the discriminator's determination whether the refined image is considered real or fake. In this way, the animation system 904 may include a Neural Renderer to render the warped mesh as an image of the object 406 such that the image of the object is a reproduction of the object 406, and the animation system may include a generative adversarial network to refine the image of the object at a high frequency to correct the warping effects that make the rendered image of the object distinguishable from the object 406. In other words, the refined image can correct high frequency features of the rendered image which make it distinguishable from the object 406. Thus, the animation system 904 can transfer the motions of an object in a digital video to another object without needing any information about the definitions describing the motions of the object in the video". Note that warping adjusts vertices by offsets to match target appearance, and warps mesh based on the vertex offset to reproduce objects in the second frame N+1, enabling intermediate deformations for animation sequences),
wherein N+1 is a frame number corresponding to a frame N+1 immediately following the frame Nin the video (Se Cower: Fig. 8, and [0086], "Frame sequence 820 includes upsampled video and includes frames 811,822,823,824,815,826,827,828, and 819, which are displayed at a higher framerate. From the received video stream that includes frames 811, 815, and 819, additional frames are obtained by interpolation, as described herein. The additional frames in frame sequence 820 are 822, 823, 824, 826, 827, and 828, and are obtained by interpolation and added to the sequence according to the techniques described herein. As a result of using interpolated frames, the frame sequence 820 can be displayed at a higher framerate (since intermediate frames 822-824 and 826-828 are available) with no jumpiness while the bandwidth utilized to receive the video remains the same as that for frame sequence 810"; and [0089], "In some embodiments, the number of intermediate frames that are generated between the first frame and the second frame is based on a difference in the frame rate of the received video and the frame rate at which the video is to be played back. For example, if the video is received at 10 FPS and is to be played by at 30 FPS, 2 intermediate frames are generated for each pair of consecutive frames of the video. In this instance, if the original video has 100 frames (corresponding to 10 seconds of video), 200 intermediate frames are generated to obtain a video of 300 frames which can be played back at 30 FPS for 10 seconds of video. The frame rate for video playback may be selected based on various factors, e.g., the refresh rate of display 241, the computational capabilities of device 200, the rate of motion in the received video, etc."); and
the rendering device configured to render the adjusted 3D mesh and the texture according to one or more camera parameters associated with a display device (See (Graziosi: Fig. 1, and [0041], "To address voxelization, mesh scaling and offset information is sent in Atlas Adaptation Parameter Set (AAPS). Available camera parameters are able to be used. Alternatively, new syntax elements are introduced for voxelization (where only scaling and offset are used). The following is exemplary syntax"), wherein the display device is configured to display the video (See Kim: Figs. 4-6, and [0052], "FIG. 5 is an illustration depicting a representation 500 of a rendering of a warped mesh as an image of the object. The representation 500 is illustrated to include a rendering module 502 implemented to render an image 504 of a warped mesh. In one example, the rendering module 502 may be a Neural Renderer. The rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions.
Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 504 of the object 406"; [0055], "In one or more implementations, the animation system 110 generates the mesh 404 of the object as it appears in the first frame 114 from the first mesh 604 and the second mesh 606 using the layering dimension which is an output of the layering module 602. In one example, the output of the layering module 602 is an input to the rendering module 502. Thus, the rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 608 of the object 406"; and Figs. 1-2, and [0033], "The illustrated environment 100 also includes a display device 106 that is communicatively coupled to the computing device 102 via a wired or a wireless connection. A variety of device configurations may be used to implement the computing device 102 and/or the display device 106. The computing device 102 includes a storage device 108 and an animation system 110. The storage device 108 is illustrated to include digital content 112. An example of the digital content 112 is a digital video such as a digital video having an animated object interacting with an environment. The digital video is illustrated as frames 114-120 shown on the display device 106. The frames 114-120 are snapshots of the object as it moves in the digital video. Thus, the digital video may be observed as a series of frames 114-120 on a display of the display device 106. For example, in frame 114, the object is standing with its arms at its side, but in frame 116, the object's right arm is raised as if to direct the viewer's attention to something of interest. The object appears relaxed with its left hand partially in its pants pocket in frame 118, and the object appears to be giving a thumb's up with its left hand in frame 120. Thus, when frames 114-120 are displayed in succession, the object's various poses in each of the frames 114-120 may appear as the animation of the object in the digital video". Note that the neural 3D renderer renders the warped mesh, and outputs the rendered images to the display device).
Regarding claim 8, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 7 as outlined above. Further, Kim teaches that the system of claim 7, wherein the offset vector are determined by adjusting the offset vectors to reduce a measurement representing a mismatch between the frame N and the frame N+I (See Kim: Figs. 1-2, and [0039], " The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections ").
Regarding claim 9, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 7 as outlined above. Further, Kim teaches that the system of claim 7, wherein the offset vectors are predicted by a neural network trained on a plurality of pairs of frames of the video, wherein a training objective of the neural network is a measurement representing a mismatch between an offset-adjusted first frame of a pair of frames from the plurality of frames and a second frame of the pair of frames (See Kim: Figs. 1-4, and [0004], " The animation system identifies features of the object in the second frame using an encoder of a neural network and maps the identified features to the vertices of the mesh using a decoder of the neural network. The mapping generates a vertex offset for the mesh which predicts how the mesh is to be deformed to reproduce the animated object as it appears in the second frame. The computing device implements the warping module to warp the mesh based on the vertex offset and then a Neural Renderer renders the warped mesh as an image. The training module compares the rendered image to the second frame and trains the neural network with a loss function based on the comparison"; [0039], “The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections”; and [0049], “FIG. 4 is an illustration depicting a representation 400 of a mesh from a first frame of a digital video and an object from a second frame of the digital video. As shown in FIG. 4, the representation 400 includes a digital video 402 having frames 114-120 that include an animated object. The representation 400 also includes a mesh 404 of the object as it appears in the first frame 114 and the object 406 as it appears in the second frame 116. The mesh 404 includes a plurality of vertices which correspond to features of the object in the first frame 114, and some features of the object 406 are different than features of the object in the first frame 114. For example, the object in the first frame 114 may be standing with its hands at its sides and the object 406 may have its right arm raised as if to direct the viewer's attention to something of interest”).
Regarding claim 10, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 7 as outlined above. Further, Kim teaches that the system of claim 7, wherein the offset vectors are determined such that rasterized projections of the offset vectors on a set of two-dimensional views of a scene in the video match one or more optical flow fields between the frame N and the frame N+1 of the scene for the two-dimensional views (See Kim: Fig. 9, and [0064], "In one example, the animation system 904 may include an encoder and a decoder of a neural network to predict vertex offsets for warping the combined mesh 902 to match features of the object 406 from the second frame 116. In another example, the animation system 904 may include a Neural Renderer to receive faces, textures, initial vertex positions, and the predicted vertex offsets to warp the combined mesh 902 and render the warped mesh as an image of the object 406. For example, the neural network may be trained using a loss function based on a comparison of the rendered image of the object and the object 406. In an example, the animation system 904 may include a generator and a discriminator of a generative adversarial network. For example, the generator may generate a refined image from the rendered image and the discriminator may receive the refined image and a ground truth to determine whether the refined image is real or fake. In one or more implementations, the generator can be trained by backpropagation of a result of the discriminator's determination whether the refined image is considered real or fake. In this way, the animation system 904 may include a Neural Renderer to render the warped mesh as an image of the object 406 such that the image of the object is a reproduction of the object 406, and the animation system may include a generative adversarial network to refine the image of the object at a high frequency to correct the warping effects that make the rendered image of the object distinguishable from the object 406. In other words, the refined image can correct high frequency features of the rendered image which make it distinguishable from the object 406. Thus, the animation system 904 can transfer the motions of an object in a digital video to another object without needing any information about the definitions describing the motions of the object in the video").
Regarding claim 11, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 7 as outlined above. Further, Kim teaches that the system of claim 7, further comprising:
comparing a rendered image of offset-adjusted frame N, defined as the frame N after the adjusting, with a rendered image of the frame N+1 (See Kim: Figs. 1-2, and [0046], "In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video");
computing a loss function based on the comparison (See Kim: Figs. 1-2, and [0039], "The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network "learns" how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections"); and
adjusting the offset vectors based on the loss function (See Kim: Figs. 1-2, and [0039], "The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network "learns" how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections"; and Fig. 7, and [0059], "The animation system 702 includes a generative adversarial network which is illustrated as a generator module 710 and a discriminator module 712. Once trained, the generative adversarial network is configured to generate refined images that are indistinguishable from the object 406 in the second frame 116. The generator module 710 receives the rendered image 608 from the rendering module 708 as an input, and the generator module 710 outputs a refined image of the object 406 in the second frame 116. In this way, the generator module 710 is implemented to generate refined image candidates which are then evaluated by the discriminator module 712, e.g., to determine whether the candidates are real or fake 714. A goal of the generator module 710 is therefore to generate a candidate that is considered real by the discriminator module 712, e.g., through comparison to a ground truth. Accordingly, the generator module 710 is trained as part of adversarial back-and-forth communication between the generative and discriminator modules in order to generate "real" candidates". Note that the training continues until the refined images sufficiently matching the target object images).
Regarding claim 12, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 11 as outlined above. Further, Kim teaches that the system of claim 11, wherein the offset vectors are predicted by a neural network trained using the computed loss function (See Kim: Figs. 1-3, and [0045], " The warping module 212 warps the mesh based on the vertex offset (block 314) and the training module 214 trains the neural network based on a comparison of the warped mesh with the object from the second frame (block 316). For example, the training module 214 may use the loss function 218 to train the neural network to improve the quality of the reproduction of the object in the second frame 116").
Regarding claim 13, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 1 as outlined above. Further, Kim teaches that a non-transitory machine-readable storage medium storing a set of instructions that when executed by one or more processors, cause the one or more processors to perform operations (See Kim: Fig. 10, and [0075], ""Computer-readable signal media" may refer to a signal-bearing medium that is configured to transmit instructions to the hardware of the computing device 1002, such as via a network. Signal media typically may embody computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as carrier waves, data signals, or other transport mechanism. Signal media also include any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media") comprising:
. receiving, at a video reader and decoder, a frame N corresponding to a frame number N in a three-dimensional video comprising a plurality of frames; (See Kim: Figs. 1-2, and [0024], "The selection module selects a second frame from the digital video which also has the animated object but in a different orientation than in the first frame. In the first frame, the object might be standing with its arms at its sides, but the object may have its right arm raised with its right elbow bent in the second frame. The identification module identifies features of the animated object in the second frame using an encoder of a neural network"; and [0029], "The described systems and techniques improve digital animation by allowing digital animators to transfer an animation of an object to another object. This improvement also significantly increases the efficiency of digital animation. These systems and techniques improve a user experience for digital animators by eliminating the tedious task of animating an object to have features of another animated object. Additionally, the systems and techniques can improve computational efficiency by using neural networks to transfer animation from an object to another object instead of requiring computations to render animation definitions for every object requiring animation". Note that the system receives and processes frames from the digital video sequentially with frame N (such as the first or second frame) in the meshing module using neural networks)
retrieving, at the video reader and decoder, a 3D mesh including a plurality of vertices, a texture, and one or more offset vectors associated with each vertex of the 3D mesh in the frame N (See Kim: Figs. 1-5, and [0025], "The animation system also includes a mapping module and a prediction module. The mapping module maps the identified features of the object in the second frame to the vertices of the combined mesh. Next, the prediction module generates a vertex offset for the mesh using a decoder of the neural network. This vertex offset is a prediction of how the combined mesh is to be deformed to reproduce the animated object as it appears in the second frame"; [0046], "In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video"; and [0052], "FIG. 5 is an illustration depicting a representation 500 of a rendering of a warped mesh as an image of the object. The representation 500 is illustrated to include a rendering module 502 implemented to render an image 504 of a warped mesh. In one example, the rendering module 502 may be a Neural Renderer. The rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 504 of the object 406". Note that the meshing module generate mesh with plurality of vertices corresponding to object features in the first frame (N));
transmitting, to a rendering device, the 3D mesh, the texture, and the offset vectors associated with the frame N (See Kim: Figs. 1-3, and [0035], "According to various implementations, such a machine-learning model uses supervised learning, semi-supervised learning, unsupervised learning, or reinforcement learning. For example, the machine-learning model can include, but is not limited to, clustering, decision trees, support vector machines, linear regression, logistic regression, Bayesian networks, random forest learning, dimensionality reduction algorithms, boosting algorithms, artificial neural networks (e.g., fully-connected neural networks, deep convolutional neural networks, or recurrent neural networks), deep learning, etc. Thus, a machine-learning model makes high-level abstractions in data by generating data-driven predictions or decisions from the known input data. In this manner, the animation system 110 receives a digital video having an animated object as an input, and the animation system 110 employs the neural network 122 to output a reproduction of the animated object. In one or more implementations, the animation system 110 may output the reproduction of the animated object without having any information about the animated object in the digital video"; [0045], "The warping module 212 warps the mesh based on the vertex offset (block 314) and the training module 214 trains the neural network based on a comparison of the warped mesh with the object from the second frame (block 316). For example, the training module 214 may use the loss function 218 to train the neural network to improve the quality of the reproduction of the object in the second frame 116"; and [0046], "In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video". Note that the mesh and vertex offset are passed to the warping and rendering module (neural renderer) to generate the output images);
adjusting, at the rendering device, a three-dimensional position of at least one vertex of the 3D mesh by interpolating the three-dimensional position of the at least one vertex according to the corresponding offset vectors (See Vogt: Figs. 1-2, and [0070], “The vertex information of the first static 3D key-model is put into the position, normal, color, and texture attributes of the OpenGL vertex. The information of the vertex of the second static 3D key-model is put into four of the remaining texture coordinates of the same OpenGL vertex. The actual interpolation of the two vertices is achieved in the vertex shader program on the GPU. See FIG. 2, which shows a schematic representation of the usage, in accordance with principles of the present invention, of a GPU vertex shader for the generation of a dynamic 3D object from the input of two different static 3D key-models of the object. The "keyFrameBlend" parameter controls the linear interpolation between both static 3D key-models. See Table 1 below, which provides an example of Cg code for the linear interpolation of static 3D key-models on programmable vertex shaders. TABLE-US-00001 TABLE 1 struct vert { float4 position : POSITION; float3 normal : NORMAL; float3 color : COLOR; float3 texCoord : TEXCOORDO; }; vert main( float4 position1 : POSITION, float4 normal1 : NORMAL float4 color1 : COLOR float4 texCoord1 : TEXCOORD0, float4 position2 : TEXCOORD1, float4 normal2 : TEXCOORD2, float4 color2 : TEXCOORD3, float4 texCoord2 : TEXCOORD4, uniform float keyFrameBlend, uniform float4.times.4 ModelViewProj ) { vert OUT; OUT.position = mul { ModelViewProj, ( lerp(position1, position2, keyFrameBlend ) ) ); OUT.normal = normalize( lerp(normal1, normal2, keyFrameBlend) ) ); OUT.color = lerp(color1, color2, keyFrameBlend ); OUT.texCoord = lerp( texCoord1, texCoord2, keyFrameBlend ); return OUT”; and [0069], “A preferred embodiment of the present invention includes the above-mentioned real-time interpolation of corresponding vertex coordinates in between the static 3D key-models, and the extension of the information for all vertices in each static 3D key-model in the modeling-phase to include other attributes, with GPU acceleration. The real-time interpolation is implemented in a vertex shader program of a programmable GPU. By utilizing the multi-texturing capabilities of current graphics hardware, there is assigned to each vertex the information of two corresponding vertices of two static 3D key-models”) and a playback time moment (See Kim: Figs. 1-2, and [0038], "The animation system 110 is also illustrated to include a selection module 204. The selection module 204 is implemented to select a second frame 116 from the digital video having the object. An identification module 206 identifies features of the object from the second frame 116 as part of the neural network 122. The neural network 122 includes a mapping module 208 and a prediction module 210. When implemented, the mapping module 208 maps the identified features of the object in the second frame 116 to the vertices of the mesh, and the prediction module 210 generates vertex offsets based on this mapping. Finally, a warping module 212 of the animation system 110 warps the mesh based on the vertex offsets to reproduce the object as it appears in the second frame 116"; [0047], "In one or more implementations, regularization may help avoid overfitting as part of training the neural network 122. Regularization is a form of regression which regulates or minimizes coefficients of the lost function 218, and this helps prevent the neural network 122 from fitting noise in training. This also improves generalization in machine learning. For example, Laplacian regularization may be used to calculate distances between a Laplacian coordinate of a vertex before and after warping the mesh based on the predicted vertex offset. In another example, edge-length regularization may be used to penalize differences in edge lengths before and after warping the mesh"; Figs. 4 and 9, and [0064], "In one example, the animation system 904 may include an encoder and a decoder of a neural network to predict vertex offsets for warping the combined mesh 902 to match features of the object 406 from the second frame 116. In another example, the animation system 904 may include a Neural Renderer to receive faces, textures, initial vertex positions, and the predicted vertex offsets to warp the combined mesh 902 and render the warped mesh as an image of the object 406. For example, the neural network may be trained using a loss function based on a comparison of the rendered image of the object and the object 406. In an example, the animation system 904 may include a generator and a discriminator of a generative adversarial network. For example, the generator may generate a refined image from the rendered image and the discriminator may receive the refined image and a ground truth to determine whether the refined image is real or fake. In one or more implementations, the generator can be trained by backpropagation of a result of the discriminator's determination whether the refined image is considered real or fake. In this way, the animation system 904 may include a Neural Renderer to render the warped mesh as an image of the object 406 such that the image of the object is a reproduction of the object 406, and the animation system may include a generative adversarial network to refine the image of the object at a high frequency to correct the warping effects that make the rendered image of the object distinguishable from the object 406. In other words, the refined image can correct high frequency features of the rendered image which make it distinguishable from the object 406. Thus, the animation system 904 can transfer the motions of an object in a digital video to another object without needing any information about the definitions describing the motions of the object in the video". Note that warping adjusts vertices by offsets to match target appearance, and warps mesh based on the vertex offset to reproduce objects in the second frame N+1, enabling intermediate deformations for animation sequences)
while preserving the texture as retrieved from the frame N (See Vogt: Figs. 1-2, and [0003], “As used herein, a dynamic 3D object is defined as a three-dimensional virtual object that dynamically changes its shape through deformations or relative motions of its parts. The definition of dynamic can be extended to not only include geometrical changes but also changes in appearance, color, texture, and other attributes of the object. Examples of a dynamic 3D object are a virtual person walking or a virtual heart beating. A static 3D key-model is herein defined as a model of a dynamic 3D object in a fixed pose with a particular set of attributes. Typical representations are static wireframe or triangulated surface models, which can be stored, e.g., in form of VRML files”; [0016], “In accordance with another aspect of the invention, the step of entering first and second vertex information comprises entering attributes, including any of color, texture, and normal vector information”; and [0087], “Several 3D models of that little comic-like figure were created in different poses with Maya and were imported as static 3D key-models into the AR system in accordance with the present invention. The interpolation between any two of those static 3D key-modes, that is, implementation layer (a), is accomplished by the GPU vertex shader program as depicted in FIG. 2 and Table 1. The dynamics in between both static 3D key-models is achieved by slowly increasing the parameter keyFrameBlend from 0.0 to 1.0 at each new rendering process”), wherein the playback time moment is between N and N+1, wherein N+ I is a frame number corresponding to a frame N+ I immediately following the frame Nin the vide (See Vogt: Figs. 1-2, and [0069], “A preferred embodiment of the present invention includes the above-mentioned real-time interpolation of corresponding vertex coordinates in between the static 3D key-models, and the extension of the information for all vertices in each static 3D key-model in the modeling-phase to include other attributes, with GPU acceleration. The real-time interpolation is implemented in a vertex shader program of a programmable GPU. By utilizing the multi-texturing capabilities of current graphics hardware, there is assigned to each vertex the information of two corresponding vertices of two static 3D key-models”; and [0087], “Several 3D models of that little comic-like figure were created in different poses with Maya and were imported as static 3D key-models into the AR system in accordance with the present invention. The interpolation between any two of those static 3D key-modes, that is, implementation layer (a), is accomplished by the GPU vertex shader program as depicted in FIG. 2 and Table 1. The dynamics in between both static 3D key-models is achieved by slowly increasing the parameter keyFrameBlend from 0.0 to 1.0 at each new rendering process”. Note that blending th4 N and N+1 frame according to the same playback time fraction between 0 and 1 while only the vertex positions are adjusted, and this is mapped to the texture preserved.),
wherein the playback time moment is between N and N+1, wherein N+I is a frame number corresponding to a frame N+I immediately following the frame Nin the video (Se Cower: Fig. 8, and [0086], "Frame sequence 820 includes upsampled video and includes frames 811,822,823,824,815,826,827,828, and 819, which are displayed at a higher framerate. From the received video stream that includes frames 811,815, and 819, additional frames are obtained by interpolation, as described herein. The additional frames in frame sequence 820 are 822, 823, 824, 826, 827, and 828, and are obtained by interpolation and added to the sequence according to the techniques described herein. As a result of using interpolated frames, the frame sequence 820 can be displayed at a higher framerate (since intermediate frames 822-824 and 826-828 are available) with no jumpiness while the bandwidth utilized to receive the video remains the same as that for frame sequence 810"; and [0089], "In some embodiments, the number of intermediate frames that are generated between the first frame and the second frame is based on a difference in the frame rate of the received video and the frame rate at which the video is to be played back. For example, if the video is received at 10 FPS and is to be played by at 30 FPS, 2 intermediate frames are generated for each pair of consecutive frames of the video. In this instance, if the original video has 100 frames (corresponding to 10 seconds of video), 200 intermediate frames are generated to obtain a video of 300 frames which can be played back at 30 FPS for 10 seconds of video. The frame rate for video playback may be selected based on various factors, e.g., the refresh rate of display 241, the computational capabilities of device 200, the rate of motion in the received video, etc."); and
rendering the adjusted 3D mesh and the texture according to one or more camera parameters associated with a display device, wherein the display device is configured to display the video (See Kim: Figs. 4-6, and [0052], "FIG. 5 is an illustration depicting a representation 500 of a rendering of a warped mesh as an image of the object. The representation 500 is illustrated to include a rendering module 502 implemented to render an image 504 of a warped mesh. In one example, the rendering module 502 may be a Neural Renderer. The rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 504 of the object 406"; [0055], "In one or more implementations, the animation system 110 generates the mesh 404 of the object as it appears in the first frame 114 from the first mesh 604 and the second mesh 606 using the layering dimension which is an output of the layering module 602. In one example, the output of the layering module 602 is an input to the rendering module 502. Thus, the rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 608 of the object 406"; and Figs. 1-2, and [0033], "The illustrated environment 100 also includes a display device 106 that is communicatively coupled to the computing device 102 via a wired or a wireless connection. A variety of device configurations may be used to implement the computing device 102 and/or the display device 106. The computing device 102 includes a storage device 108 and an animation system 110. The storage device 108 is illustrated to include digital content 112. An example of the digital content 112 is a digital video such as a digital video having an animated object interacting with an environment. The digital video is illustrated as frames 114-120 shown on the display device 106. The frames 114-120 are snapshots of the object as it moves in the digital video. Thus, the digital video may be observed as a series of frames 114-120 on a display of the display device 106. For example, in frame 114, the object is standing with its arms at its side, but in frame 116, the object's right arm is raised as if to direct the viewer's attention to something of interest. The object appears relaxed with its left hand partially in its pants pocket in frame 118, and the object appears to be giving a thumb's up with its left hand in frame 120. Thus, when frames 114-120 are displayed in succession, the object's various poses in each of the frames 114-120 may appear as the animation of the object in the digital video". Note that the neural 3D renderer renders the warped mesh, and outputs the rendered images to the display device).
Regarding claim 14, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 1 as outlined above. Further, Kim, Cower, Graziosi and Vogt teach that a method for training a neural network for offset vector determination, the method (See Kim: Figs. 1-2, and [0039], "The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network "learns" how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections") comprising:
comparing a rendered image of an offset-adjusted mesh of a frame N corresponding to a frame number N in a video further comprised of a plurality of frames, with a rendered image of a 3D mesh of a frame N+ I corresponding to a frame number N+ I in the video (See Kim: Figs. 1-2, and [0046], "In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video"),
wherein the offset-adjusted mesh comprises a 3D mesh of the frame N including a plurality of vertices (See Kim: Fig. 3, and [0043], “FIG. 3 is a flow diagram depicting a procedure 300 in an example implementation in which a neural network is trained based on a comparison of a warped mesh and an object from a frame of a digital video. The animation system 110 receives a digital video having an object as an input (block 302). For example, the object may be the animated object shown in frames 114-120. The meshing module 202 obtains a mesh of the object from a first frame 114 of the digital video (block 304). This mesh includes a plurality of vertices which correspond to features of the object from the first frame 114, and these vertices can be offset to manipulate the object.”),
wherein one or more offset vectors associated with each vertex of the 3D mesh have been added to the plurality of vertices (See Kim: Fig. 1-3, and [0044], “The selection module 204 selects a second frame 116 from the digital video that includes the object (block 306) which is the same object in the first frame 114 but in another configuration. For example, in the first frame 114, the object could be the animated object standing with its arms at its sides, and in the second frame 116, the object could be the animated object with its right arm raised and bent at the elbow. The identification module 206 identifies features of the object from the second frame (block 308), and the mapping module 208 maps the identified features of the object to vertices of the mesh (block 310). In this manner, the animation system 110 determines how to manipulate the vertices of the mesh from the first frame 114 to reproduce the object as it appears in the second frame 116. The prediction module 210 then generates a vertex offset of the vertices of the mesh based on the mapping (block 312)”; and [0045], “The warping module 212 warps the mesh based on the vertex offset (block 314) and the training module 214 trains the neural network based on a comparison of the warped mesh with the object from the second frame (block 316). For example, the training module 214 may use the loss function 218 to train the neural network to improve the quality of the reproduction of the object in the second frame 116”), and
wherein the rendered image of the offset-adjusted mesh is generated using a texture of the frame N that remains unmodified (See Kim: Figs. 4-5, and [0049], “FIG. 4 is an illustration depicting a representation 400 of a mesh from a first frame of a digital video and an object from a second frame of the digital video. As shown in FIG. 4, the representation 400 includes a digital video 402 having frames 114-120 that include an animated object. The representation 400 also includes a mesh 404 of the object as it appears in the first frame 114 and the object 406 as it appears in the second frame 116. The mesh 404 includes a plurality of vertices which correspond to features of the object in the first frame 114, and some features of the object 406 are different than features of the object in the first frame 114. For example, the object in the first frame 114 may be standing with its hands at its sides and the object 406 may have its right arm raised as if to direct the viewer's attention to something of interest”; and [0052], “FIG. 5 is an illustration depicting a representation 500 of a rendering of a warped mesh as an image of the object. The representation 500 is illustrated to include a rendering module 502 implemented to render an image 504 of a warped mesh. In one example, the rendering module 502 may be a Neural Renderer. The rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 504 of the object 406”);
computing a loss function based on the comparing (See Kim: Figs. 1-2, and [0039], "The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network "learns" how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections"); and
training a neural network used to predict the offset vectors with the loss function (See Kim: Figs. 1-2, and [0039], "The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network "learns" how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections"; and Fig. 7, and [0059], "The animation system 702 includes a generative adversarial network which is illustrated as a generator module 710 and a discriminator module 712. Once trained, the generative adversarial network is configured to generate refined images that are indistinguishable from the object 406 in the second frame 116. The generator module 710 receives the rendered image 608 from the rendering module 708 as an input, and the generator module 710 outputs a refined image of the object 406 in the second frame 116. In this way, the generator module 710 is implemented to generate refined image candidates which are then evaluated by the discriminator module 712, e.g., to determine whether the candidates are real or fake 714. A goal of the generator module 710 is therefore to generate a candidate that is considered real by the discriminator module 712, e.g., through comparison to a ground truth. Accordingly, the generator module 710 is trained as part of adversarial back-and-forth communication between the generative and discriminator modules in order to generate "real" candidates". Note that the training continues until the refined images sufficiently matching the target object images).
Regarding claim 16, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 1 as outlined above. Further, Kim teaches that the method of claim 1, wherein the playback time moment between N and N+1 is within a semi-interval starting at the frame N and excluding the frame N+1 (See Kim: Figs. 1-2, and [0044], "The selection module 204 selects a second frame 116 from the digital video that includes the object (block 306) which is the same object in the first frame 114 but in another configuration. For example, in the first frame 114, the object could be the animated object standing with its arms at its sides, and in the second frame 116, the object could be the animated object with its right arm raised and bent at the elbow. The identification module 206 identifies features of the object from the second frame (block 308), and the mapping module 208 maps the identified features of the object to vertices of the mesh (block 310). In this manner, the animation system 110 determines how to manipulate the vertices of the mesh from the first frame 114 to reproduce the object as it appears in the second frame 116. The prediction module 210 then generates a vertex offset of the vertices of the mesh based on the mapping (block 312)". Note that offset adjustments and rendering for smooth animation between frame N and N+1 generates intermediate representation for time between these two frames, naturally operating over the time interval starting at one frame and progressively toward another frame without overlapping, i.e. [N, N+1) (forward animation) or (N, N+1] (backward animation), ad the most common one is [N, N+1) because N frame is normally received before N+1 frame).
Regarding claim 17, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 1 as outlined above. Further, Graziosi teaches that the method of claim 1, wherein the display device is a virtual reality headset (See Graziosi: Fig. 24, and [0096], "Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, a smart phone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer/player (e.g., DVD writer/player, high definition disc writer/player, ultra high definition disc writer/player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., smart watch), a vehicle (e.g., a self-driving vehicle) or any other suitable computing device").
Regarding claim 18, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 7 as outlined above. Further, Kim teaches that the system of claim 7, wherein the display device is a virtual reality headset (See Graziosi: Fig. 24, and [0096], "Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, a smart phone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer/player (e.g., DVD writer/player, high definition disc writer/player, ultra high definition disc writer/player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., smart watch), a vehicle (e.g., a self-driving vehicle) or any other suitable computing device").
Regarding claim 19, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 7 as outlined above. Further, Kim teaches that the system of claim 7, wherein the playback time moment between N and N+1 is within a semi-interval starting at the frame N and excluding the frame N+1 (See Kim: Figs. 1-2, and [0044], "The selection module 204 selects a second frame 116 from the digital video that includes the object (block 306) which is the same object in the first frame 114 but in another configuration. For example, in the first frame 114, the object could be the animated object standing with its arms at its sides, and in the second frame 116, the object could be the animated object with its right arm raised and bent at the elbow. The identification module 206 identifies features of the object from the second frame (block 308), and the mapping module 208 maps the identified features of the object to vertices of the mesh (block 310). In this manner, the animation system 110 determines how to manipulate the vertices of the mesh from the first frame 114 to reproduce the object as it appears in the second frame 116. The prediction module 210 then generates a vertex offset of the vertices of the mesh based on the mapping (block 312)". Note that offset adjustments and rendering for smooth animation between frame N and N+1 generates intermediate representation for time between these two frames, naturally operating over the time interval starting at one frame and progressively toward another frame without overlapping, i.e. [N, N+1) (forward animation) or (N, N+1] (backward animation), and the most common one is [N, N+1) because N frame is normally received before N+1 frame).
Regarding claim 20, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 1 as outlined above. Further, Graziosi teaches that the method of claim 1, wherein the three-dimensional video is constructed from any of a random distribution of cameras pointing towards one or more three-dimensional scenes, and multiview-stereo data (See Graziosi: Fig. 1, and [0032], "A method of compression of 3D mesh data using projections of mesh surface data and video representation of connectivity data is described herein. The method utilizes 3D surface patches to represent a set of connected triangles on a mesh surface. The projected surface data is stored in patches (a mesh patch) that is encoded in atlas data. The connectivity of the mesh, that is, the vertices and the triangles of the surface patch, are encoded using video-based compression techniques. The data is encapsulated in a new video component named vertex video data, and the disclosed structure allows for progressive mesh coding by separating sets of vertices in layers, and creating levels of detail for the mesh connectivity. This approach extends the functionality of the V3C (volumetric video-based) standard, currently being used for coding of point cloud and multiview plus depth content").
Regarding claim 21, Kim, Cower, Graziosi and Vogt teach all the features with respect to claim 1 as outlined above. Further, Kim teaches that the method of claim 14, wherein the rendered image of the offset-adjusted mesh is generated by a differentiable renderer (See Kim: Fig. 7, and [0058], “The animation system 702 also includes a rendering module 708 which can be implemented to warp the mesh 404 based on the predicted vertex offsets and render the warped mesh as an image 608. The vertex offset 704 and the layering module 706 are illustrated as inputs to a rendering module 708. Thus, the rendering module 708 is implemented to receive the vertex offset 704 and the layering module 706 as inputs and the rendering module 708 renders an image 608 as an output. In one example, the rendering module 708 may include a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, to render the mesh as an image based on the vertex offsets predicted by the neural network 122. This rendered image can then be compared to an input image from the digital video to train the neural network 122. For example, the input image may include the object 406. Once trained, the neural network 112 implements the rendering module 708 to render an image 608 that is the same as the object 406 in the second frame”. Note that the Kato neural renderer is well known to be a differentiable renderer whose
Entire purpose is to allow gradients of an image-space loss to flow back though the rasterization process to the mesh vertices.), and wherein the loss function is backpropagated through the differentiable renderer to estimate a gradient of the loss function with respect to the one or more offset vectors (See Kim: Fig. 7, and [0060], “Thus, the generator module 710 is trained to refine rendered images in a manner that is considered “real” by a discriminator module 712 of the generative adversarial network. As part of this training, the generator module 710 receives the rendered image 608 as an input, and from this, generates a refined image of the animated object 406 in the second frame 116. The refined image is then communicated to the discriminator module 712 to determine whether this refined image is real or fake 714. Thus, the discriminator module 712 receives as inputs the refined image as well as the second frame 116, and the discriminator module 712 is implemented to determine whether the refined image is real or fake 714. The generator module 710 is trained by backpropagation of a result of the comparison, e.g., whether the refined image is considered real or fake 714. In this way, the generator module 710 is trained to refine images automatically and without user intervention”).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
