DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are pending under this Office action.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
Claim Limitation Interpreted under 35 USC 112(f)
Use of the word "means" (or "step for") in a claim with functional language creates a rebuttable presumption that the claim element is to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is invoked is rebutted when the function is recited with sufficient structure, material, or acts within the claim itself to entirely perform the recited function.
Absence of the word "means" (or "step for") in a claim creates a rebuttable presumption that the claim element is not to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is not invoked is rebutted when the claim element recites function but fails to recite sufficiently definite structure, material or acts to perform that function.
Claim elements in this application that use the word "means" (or "step for") are presumed to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action. Similarly, claim elements that do not use the word "means" (or "step for") are presumed not to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action.
Claim limitations "an input configured to", "the formatter being configured to", and "the displayer being configured to" have been interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because it uses/they use a generic placeholder " an input/the formatter/the displayer" coupled with functional language "configured to obtain/generate/display" without reciting sufficient structure to achieve the function. Furthermore, the generic placeholder is not preceded by a structural modifier.
Since the claim limitations invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, claim 7, and its related dependent claims 8-12 and 18-19, has been interpreted to cover the corresponding structure described in the specification that achieves the claimed function, and equivalents thereof.
A review of the specification shows that the following appears to be the corresponding structure described in the specification for the 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph limitation:
" a video reader and decoder configured to receive" is interpreted as to " a corresponding frame number N 105 is sent to video reader and decoder 103 " (See Specification [0034]: "In an aspect, display device 101 (e.g., a virtual reality headset) sends one or more requests to rendering device 102 and video reader and decoder 103. In particular a desired time moment 104 is sent to rendering device 102, and a corresponding frame number N 105 is sent to video reader and decoder 103. In one aspect, the time moment 104 lies in a semi-interval [N, N+1). The discrete-time index N may be associated with a video frame N, while the discrete-time index N+1 may be associated with a video frame N+1").
" the video reader and decoder further configured to retrieve" is interpreted as to " A 3D mesh including a plurality of vertices, a texture, and one or more offset vectors associated with each vertex of the 3D mesh in the frame N are retrieved and transmitted to a rendering device" (See Specification (Abstract): "A 3D mesh including a plurality of vertices, a texture, and one or more offset vectors associated with each vertex of the 3D mesh in the frame N are retrieved and transmitted to a rendering device ").
"the rendering device configured to adjust" is interpreted as to "The three-dimensional positions of the vertices V.sub.N are then adjusted by vertex adjustment 107 based on the received desired time moment 104 and the offsets O.sub.N. Details about the operation of vertex adjustment 107 are described subsequently" (See Specification [0035]: "In one aspect, rendering device 102 then receives the three-dimensional video frame F.sub.N (with offsets O.sub.N) as data 106, from video reader and decoder 103. In one aspect, the three-dimensional video frame F.sub.N includes a mesh and a texture associated with three-dimensional video frame F.sub.N. The three-dimensional positions of the vertices V.sub.N are then adjusted by vertex adjustment 107 based on the received desired time moment 104 and the offsets O.sub.N. Details about the operation of vertex adjustment 107 are described subsequently ").
" the rendering device configured to rendering" is interpreted as to "After vertex adjustment 107, the mesh and the texture (which remains unmodified) 108 are sent to the mesh rendering 110 (a rendering process) " (See Specification [0036]: "After vertex adjustment 107, the mesh and the texture (which remains unmodified) 108 are sent to the mesh rendering 110 (a rendering process), which also takes into account the desired camera parameters 109 sent by display device 101. For example, for a virtual reality display, the desired camera parameters are based on the position and orientation of the virtual reality headset in the physical space. The rendering process 110 then generates a single two-dimensional view or a stereopair of two-dimensional views 111 of the textured mesh 108 based on the received camera parameters 109. This view or a pair of views 111 are then sent to the display device 101 and output to one or more screens of display device 101").
If applicant wishes to provide further explanation or dispute the examiner's interpretation of the corresponding structure, applicant must identify the corresponding structure with reference to the specification by page and line number, and to the drawing, if any, by reference characters in response to this Office action.
If applicant does not intend to have the claim limitations treated under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may amend the claim(s) so that it/they will clearly not invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, or present a sufficient showing that the claim recites/recite sufficient structure, material, or acts for performing the claimed function to preclude application of 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
For more information, see MPEP § 2173 et seq. and Supplementary Examination Guidelines for Determining Compliance With 35 U.S.C. 112 and for Treatment of Related Issues in Patent Applications, 76 FR 7162, 7167 (Feb. 9, 2011).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 13 and 15 are rejected under §35 U.S.C. 101 as not falling within one of the four statutory categories of invention because the claimed invention is directed to computer program per se. See MPEP 2106(1). A claim directed toward a non-transitory computer readable medium having the program encoded thereon establishes a sufficient functional relationship between the program and a computer so as to remove it from the realm of “program per se”. MPEP 2111.05(111). Hence, adding the limitation of “non-transitory” before “machine-readable storage medium” for claims 13 and 15 would resolve this issue.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kim, etc. (US 20200265294 A1) in view of Cower (US 20210182540 A1), further in view of Graziosi, etc. (US 20220108483 A1).
Regarding claim 1, Kim teaches that a method for displaying a three-dimensional video comprising a plurality of frames, at a different frame rate from which the video was stored or acquired, the method (See Kim: Figs. 1-2, and [0032], “FIG. 1 is an illustration of an environment 100 in an example implementation that is operable to employ digital systems and techniques as described herein. The illustrated environment 100 includes a computing device 102 connected to a network 104. The computing device 102 may be configured as a desktop computer, a laptop computer, a mobile device (e.g., assuming a handheld configuration such as a tablet or mobile phone), and so forth. Thus, the computing device 102 may range from a full resource device with substantial memory and processor resources (e.g., personal computers, game consoles) to a low-resource device with limited memory and/or processing resources (e.g., mobile devices). Additionally, the computing device 102 may be representative of a plurality of different devices, such as multiple servers utilized by a business to perform operations “over the cloud.””; [0003], “Systems and techniques are described for object animation using generative neural networks. A computing device implements an animation system which receives a digital video having an animated object as an input. The animation system includes a meshing module, a warping module, and a training module. The meshing module obtains a mesh of the animated object from a first frame of the digital video. This mesh has a plurality of vertices which correspond to features of the object in the first frame. The animation system selects a second frame from the digital video which also has the animated object but in a different orientation than in the first frame. For example, the object might be standing with its arms at its sides in the first frame and the object may have its right arm raised with its right elbow bent in the second frame”; and [0036], “FIG. 2 depicts a system 200 in an example implementation showing operation of an animation system 110. As described above, the animation system 110 receives as an input a digital video having frames of an animated object, and the animation system 110 employs the neural network 122 to output a reproduction of the animated object. In more general terms, the animated object in the digital video is an object. In one or more implementations, the neural network 122 can include an encoder to analyze an image of an object and a decoder to predict vertex offsets for vertices of a mesh of the object”) comprising:
receiving, at a video reader and decoder, a frame N corresponding to a frame number N in the video (See Kim: Figs. 1-2, and [0024], “The selection module selects a second frame from the digital video which also has the animated object but in a different orientation than in the first frame. In the first frame, the object might be standing with its arms at its sides, but the object may have its right arm raised with its right elbow bent in the second frame. The identification module identifies features of the animated object in the second frame using an encoder of a neural network”; and [0029], “The described systems and techniques improve digital animation by allowing digital animators to transfer an animation of an object to another object. This improvement also significantly increases the efficiency of digital animation. These systems and techniques improve a user experience for digital animators by eliminating the tedious task of animating an object to have features of another animated object. Additionally, the systems and techniques can improve computational efficiency by using neural networks to transfer animation from an object to another object instead of requiring computations to render animation definitions for every object requiring animation”. Note that the system receives and processes frames from the digital video sequentially with frame N (such as the first or second frame) in the meshing module using neural networks);
retrieving, at the video reader and decoder, a 3D mesh including a plurality of vertices, a texture, and one or more offset vectors associated with each vertex of the 3D mesh in the frame N (See Kim: Figs. 1-5, and [0025], “The animation system also includes a mapping module and a prediction module. The mapping module maps the identified features of the object in the second frame to the vertices of the combined mesh. Next, the prediction module generates a vertex offset for the mesh using a decoder of the neural network. This vertex offset is a prediction of how the combined mesh is to be deformed to reproduce the animated object as it appears in the second frame”; [0046], “In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video”; and [0052], “FIG. 5 is an illustration depicting a representation 500 of a rendering of a warped mesh as an image of the object. The representation 500 is illustrated to include a rendering module 502 implemented to render an image 504 of a warped mesh. In one example, the rendering module 502 may be a Neural Renderer. The rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 504 of the object 406”. Note that the meshing module generate mesh with plurality of vertices corresponding to object features in the first frame (N));
transmitting, to a rendering device, the 3D mesh, the texture, and the offset vectors associated with the frame N (See Kim: Figs. 1-3, and [0035], “According to various implementations, such a machine-learning model uses supervised learning, semi-supervised learning, unsupervised learning, or reinforcement learning. For example, the machine-learning model can include, but is not limited to, clustering, decision trees, support vector machines, linear regression, logistic regression, Bayesian networks, random forest learning, dimensionality reduction algorithms, boosting algorithms, artificial neural networks (e.g., fully-connected neural networks, deep convolutional neural networks, or recurrent neural networks), deep learning, etc. Thus, a machine-learning model makes high-level abstractions in data by generating data-driven predictions or decisions from the known input data. In this manner, the animation system 110 receives a digital video having an animated object as an input, and the animation system 110 employs the neural network 122 to output a reproduction of the animated object. In one or more implementations, the animation system 110 may output the reproduction of the animated object without having any information about the animated object in the digital video”; [0045], “The warping module 212 warps the mesh based on the vertex offset (block 314) and the training module 214 trains the neural network based on a comparison of the warped mesh with the object from the second frame (block 316). For example, the training module 214 may use the loss function 218 to train the neural network to improve the quality of the reproduction of the object in the second frame 116”; and [0046], “In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video”. Note that the mesh and vertex offset are passed to the warping and rendering module (neural renderer) to generate the output images);
adjusting, at the rendering device, at least one vertex of the 3D mesh according to the corresponding offset vectors and a playback time moment (See Kim: Figs. 1-2, and [0038], “The animation system 110 is also illustrated to include a selection module 204. The selection module 204 is implemented to select a second frame 116 from the digital video having the object. An identification module 206 identifies features of the object from the second frame 116 as part of the neural network 122. The neural network 122 includes a mapping module 208 and a prediction module 210. When implemented, the mapping module 208 maps the identified features of the object in the second frame 116 to the vertices of the mesh, and the prediction module 210 generates vertex offsets based on this mapping. Finally, a warping module 212 of the animation system 110 warps the mesh based on the vertex offsets to reproduce the object as it appears in the second frame 116”; [0047], “In one or more implementations, regularization may help avoid overfitting as part of training the neural network 122. Regularization is a form of regression which regulates or minimizes coefficients of the lost function 218, and this helps prevent the neural network 122 from fitting noise in training. This also improves generalization in machine learning. For example, Laplacian regularization may be used to calculate distances between a Laplacian coordinate of a vertex before and after warping the mesh based on the predicted vertex offset. In another example, edge-length regularization may be used to penalize differences in edge lengths before and after warping the mesh”; Figs. 4 and 9, and [0064], “In one example, the animation system 904 may include an encoder and a decoder of a neural network to predict vertex offsets for warping the combined mesh 902 to match features of the object 406 from the second frame 116. In another example, the animation system 904 may include a Neural Renderer to receive faces, textures, initial vertex positions, and the predicted vertex offsets to warp the combined mesh 902 and render the warped mesh as an image of the object 406. For example, the neural network may be trained using a loss function based on a comparison of the rendered image of the object and the object 406. In an example, the animation system 904 may include a generator and a discriminator of a generative adversarial network. For example, the generator may generate a refined image from the rendered image and the discriminator may receive the refined image and a ground truth to determine whether the refined image is real or fake. In one or more implementations, the generator can be trained by backpropagation of a result of the discriminator's determination whether the refined image is considered real or fake. In this way, the animation system 904 may include a Neural Renderer to render the warped mesh as an image of the object 406 such that the image of the object is a reproduction of the object 406, and the animation system may include a generative adversarial network to refine the image of the object at a high frequency to correct the warping effects that make the rendered image of the object distinguishable from the object 406. In other words, the refined image can correct high frequency features of the rendered image which make it distinguishable from the object 406. Thus, the animation system 904 can transfer the motions of an object in a digital video to another object without needing any information about the definitions describing the motions of the object in the video”. Note that warping adjusts vertices by offsets to match target appearance, and warps mesh based on the vertex offset to reproduce objects in the second frame N+1, enabling intermediate deformations for animation sequences), wherein the playback time moment is between N and N+1, wherein N+1 is a frame number corresponding to a frame N+1 immediately following the frame N in the video; and
rendering the adjusted 3D mesh and the texture according to one or more camera parameters associated with a display device, wherein the display device is configured to display the video (See Kim: Figs. 4-6, and [0052], “FIG. 5 is an illustration depicting a representation 500 of a rendering of a warped mesh as an image of the object. The representation 500 is illustrated to include a rendering module 502 implemented to render an image 504 of a warped mesh. In one example, the rendering module 502 may be a Neural Renderer. The rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 504 of the object 406”; [0055], “In one or more implementations, the animation system 110 generates the mesh 404 of the object as it appears in the first frame 114 from the first mesh 604 and the second mesh 606 using the layering dimension which is an output of the layering module 602. In one example, the output of the layering module 602 is an input to the rendering module 502. Thus, the rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 608 of the object 406”; and Figs. 1-2, and [0033], “The illustrated environment 100 also includes a display device 106 that is communicatively coupled to the computing device 102 via a wired or a wireless connection. A variety of device configurations may be used to implement the computing device 102 and/or the display device 106. The computing device 102 includes a storage device 108 and an animation system 110. The storage device 108 is illustrated to include digital content 112. An example of the digital content 112 is a digital video such as a digital video having an animated object interacting with an environment. The digital video is illustrated as frames 114-120 shown on the display device 106. The frames 114-120 are snapshots of the object as it moves in the digital video. Thus, the digital video may be observed as a series of frames 114-120 on a display of the display device 106. For example, in frame 114, the object is standing with its arms at its side, but in frame 116, the object's right arm is raised as if to direct the viewer's attention to something of interest. The object appears relaxed with its left hand partially in its pants pocket in frame 118, and the object appears to be giving a thumb's up with its left hand in frame 120. Thus, when frames 114-120 are displayed in succession, the object's various poses in each of the frames 114-120 may appear as the animation of the object in the digital video”. Note that the neural 3D renderer renders the warped mesh, and outputs the rendered images to the display device).
However, Kim fails to explicitly disclose that at a different frame rate from which the video was stored or acquired; wherein the playback time moment is between N and N+1, wherein N+1 is a frame number corresponding to a frame N+1 immediately following the frame N in the video; and according to one or more camera parameters associated with a display device.
However, Cower teaches that at a different frame rate from which the video was stored or acquired (See Cower: Fig. 2, and [0050], “In some embodiments, the decoder 202 receives a set of encoded video frames via the communication unit 239. The decoder 202 decodes the video frames, for example, by decrypting (e.g., when the video is encrypted) and decompressing the encoded video frames. Once the decoder 202 decodes the video frames, in conventional video processing, the video frames are typically rendered for display. However, per techniques described herein, the decoder 202 skips scheduling of the video frames for presentation, e.g., when the frame rate of the received video is lower than a frame rate for the video application 103. As a result, instead of scheduling the video frames for presentation, the decoder 202 provides each of the decoded video frames to the video analyzer 204. For example, the set of video frames includes a first video frame and a second video frame, where the second video frame is subsequent to the first video frame, e.g., is associated with a timestamp that is later than a timestamp associated with the first video frame, such that the second video frame occupies a later position in a chronological sequence of frames than the first video frame. In some embodiments, the second video frame directly follows the first video frame, for example, the two frames are identified for sequential display with no other frames in between”); and
wherein the playback time moment is between N and N+1, wherein N+1 is a frame number corresponding to a frame N+1 immediately following the frame N in the video (Se Cower: Fig. 8, and [0086], “Frame sequence 820 includes upsampled video and includes frames 811, 822, 823, 824, 815, 826, 827, 828, and 819, which are displayed at a higher framerate. From the received video stream that includes frames 811, 815, and 819, additional frames are obtained by interpolation, as described herein. The additional frames in frame sequence 820 are 822, 823, 824, 826, 827, and 828, and are obtained by interpolation and added to the sequence according to the techniques described herein. As a result of using interpolated frames, the frame sequence 820 can be displayed at a higher framerate (since intermediate frames 822-824 and 826-828 are available) with no jumpiness while the bandwidth utilized to receive the video remains the same as that for frame sequence 810”; and [0089], “In some embodiments, the number of intermediate frames that are generated between the first frame and the second frame is based on a difference in the frame rate of the received video and the frame rate at which the video is to be played back. For example, if the video is received at 10 FPS and is to be played by at 30 FPS, 2 intermediate frames are generated for each pair of consecutive frames of the video. In this instance, if the original video has 100 frames (corresponding to 10 seconds of video), 200 intermediate frames are generated to obtain a video of 300 frames which can be played back at 30 FPS for 10 seconds of video. The frame rate for video playback may be selected based on various factors, e.g., the refresh rate of display 241, the computational capabilities of device 200, the rate of motion in the received video, etc.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Kim to have at a different frame rate from which the video was stored or acquired; wherein the playback time moment is between N and N+1, wherein N+1 is a frame number corresponding to a frame N+1 immediately following the frame N in the video as taught by Cower in order to use reduced computational power to display video with a perceived higher frame rate (See Cower: Fig. 1, and [0029], “The various embodiments described below have several advantages. First, the processing is performed by the user device that displays the video. As a result, the video application uses reduced computational power to display video with a perceived higher frame rate. Second, the embodiments also provide higher frame rates than received video frame rate, even when the video is received with the use of end-to-end encryption between a sender device and a receiver device that displays the video. Third, the interpolation is computationally efficient because the structure of the video frames is interpolated and not the texture”). Kim teaches a method and system that may animate the objects in the 3D video playback with dynamic temporal vertex interpolation using deformation and or offset vector via neural networks for frame rate conversion; while Cower teaches a system and method that explicitly adjust the number of frames and the time amount (time stamps) during playback when the frame rate and display refresh rate are different. Therefore, it is obvious to one of ordinary skill in the art to modify Kim by Cower to adjust the 3D mesh based on the frame rate and playback refresh rate. The motivation to modify Kim by Cower is “Use of known technique to improve similar devices (methods, or products) in the same way”.
However, Kim, modified by Cower, fails to explicitly disclose that according to one or more camera parameters associated with a display device.
However, Graziosi teaches that according to one or more camera parameters associated with a display device (See (Graziosi: Fig. 1, and [0041], “To address voxelization, mesh scaling and offset information is sent in Atlas Adaptation Parameter Set (AAPS). Available camera parameters are able to be used. Alternatively, new syntax elements are introduced for voxelization (where only scaling and offset are used). The following is exemplary syntax”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Kim to have according to one or more camera parameters associated with a display device as taught by Graziosi in order to enable progressive mesh coding by separating sets of vertices in layers and generating levels of detail for mesh connectivity (See Graziosi: Fig. 1, and [0099], “In operation, the video based mesh compression method enables more efficient and more accurate 3D content encoding compared to previous implementations”). Kim teaches a method and system that may animate the objects in the 3D video playback with dynamic temporal vertex interpolation using deformation and or offset vector via neural networks for frame rate conversion; while Graziosi teaches a video based mesh compression method and system that use the camera parameters together with the voxelization, mesh scaling and offset information in video mesh coding and decoding in order to enable progressive encoding more efficient. Therefore, it is obvious to one of ordinary skill in the art to modify Kim by Graziosi to use the camera parameters in 3D video animation rendering (actually camera/view parameters are inherent in 3D rendering). The motivation to modify Kim by Graziosi is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 2, Kim, Cower, and Graziosi teach all the features with respect to claim 1 as outlined above. Further, Kim teaches that the method of claim 1, wherein the retrieving the offset vector further comprises:
substantially optimizing a closeness measure between the frame N and the frame N+1 (See Kim: Fig. 2, and [0039], “The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections”); and
determining the offset vectors based on the substantially optimized closeness measure (See Kim: Figs. 1-7, and [0038], “The animation system 110 is also illustrated to include a selection module 204. The selection module 204 is implemented to select a second frame 116 from the digital video having the object. An identification module 206 identifies features of the object from the second frame 116 as part of the neural network 122. The neural network 122 includes a mapping module 208 and a prediction module 210. When implemented, the mapping module 208 maps the identified features of the object in the second frame 116 to the vertices of the mesh, and the prediction module 210 generates vertex offsets based on this mapping. Finally, a warping module 212 of the animation system 110 warps the mesh based on the vertex offsets to reproduce the object as it appears in the second frame 116”; [0050], “Thus, a technological problem addressed in the following is how to manipulate the mesh 404 to appear as the object 406. In one or more implementations, a solution to this technological problem includes generating offsets to the vertices of the mesh 404 and training the neural network 122 to predict offsets to the vertices of the mesh 404 such that the predicted offsets to the vertices warp the mesh 404 to appear as the object 406”; and [0059], “The animation system 702 includes a generative adversarial network which is illustrated as a generator module 710 and a discriminator module 712. Once trained, the generative adversarial network is configured to generate refined images that are indistinguishable from the object 406 in the second frame 116. The generator module 710 receives the rendered image 608 from the rendering module 708 as an input, and the generator module 710 outputs a refined image of the object 406 in the second frame 116. In this way, the generator module 710 is implemented to generate refined image candidates which are then evaluated by the discriminator module 712, e.g., to determine whether the candidates are real or fake 714. A goal of the generator module 710 is therefore to generate a candidate that is considered real by the discriminator module 712, e.g., through comparison to a ground truth. Accordingly, the generator module 710 is trained as part of adversarial back-and-forth communication between the generative and discriminator modules in order to generate “real” candidates”. Note that the offsets are generated and predicted by the neural networks, and refined and optimized through training to achieve closeness or minimizing the cost/loss function).
Regarding claim 3, Kim, Cower, and Graziosi teach all the features with respect to claim 1 as outlined above. Further, Kim teaches that the method of claim 1, wherein the retrieving the offset vectors further comprises:
determining a closeness measure between frame N and frame N+1 via a neural network (See Kim: Figs. 1-2, and [0039], “The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections”); and
determining the offset vectors based on the determined closeness measure (See Kim: Figs. 1-3, and [0044], “The selection module 204 selects a second frame 116 from the digital video that includes the object (block 306) which is the same object in the first frame 114 but in another configuration. For example, in the first frame 114, the object could be the animated object standing with its arms at its sides, and in the second frame 116, the object could be the animated object with its right arm raised and bent at the elbow. The identification module 206 identifies features of the object from the second frame (block 308), and the mapping module 208 maps the identified features of the object to vertices of the mesh (block 310). In this manner, the animation system 110 determines how to manipulate the vertices of the mesh from the first frame 114 to reproduce the object as it appears in the second frame 116. The prediction module 210 then generates a vertex offset of the vertices of the mesh based on the mapping (block 312)”).
Regarding claim 4, Kim, Cower, and Graziosi teach all the features with respect to claim 1 as outlined above. Further, Kim teaches that the method of claim 1, wherein the retrieving the offset vectors further comprises: determining one or more offset vectors such that rasterized projections on a set of two-dimensional views of a scene in the video match one or more optical flow fields between the frame N and the frame N+1 of the scene for the two-dimensional views (See Kim: Fig. 9, and [0064], “In one example, the animation system 904 may include an encoder and a decoder of a neural network to predict vertex offsets for warping the combined mesh 902 to match features of the object 406 from the second frame 116. In another example, the animation system 904 may include a Neural Renderer to receive faces, textures, initial vertex positions, and the predicted vertex offsets to warp the combined mesh 902 and render the warped mesh as an image of the object 406. For example, the neural network may be trained using a loss function based on a comparison of the rendered image of the object and the object 406. In an example, the animation system 904 may include a generator and a discriminator of a generative adversarial network. For example, the generator may generate a refined image from the rendered image and the discriminator may receive the refined image and a ground truth to determine whether the refined image is real or fake. In one or more implementations, the generator can be trained by backpropagation of a result of the discriminator's determination whether the refined image is considered real or fake. In this way, the animation system 904 may include a Neural Renderer to render the warped mesh as an image of the object 406 such that the image of the object is a reproduction of the object 406, and the animation system may include a generative adversarial network to refine the image of the object at a high frequency to correct the warping effects that make the rendered image of the object distinguishable from the object 406. In other words, the refined image can correct high frequency features of the rendered image which make it distinguishable from the object 406. Thus, the animation system 904 can transfer the motions of an object in a digital video to another object without needing any information about the definitions describing the motions of the object in the video”).
Regarding claim 5, Kim, Cower, and Graziosi teach all the features with respect to claim 1 as outlined above. Further, Kim teaches that the method of claim 1, further comprising:
comparing a rendered image of offset-adjusted frame N, defined as the frame N after the adjusting, with a rendered image of the frame N+1 (See Kim: Figs. 1-2, and [0046], “In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video”);
computing a loss function based on the comparison (See Kim: Figs. 1-2, and [0039], “The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections”); and
adjusting the offset vectors based on the loss function (See Kim: Figs. 1-2, and [0039], “The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections”; and Fig. 7, and [0059], “The animation system 702 includes a generative adversarial network which is illustrated as a generator module 710 and a discriminator module 712. Once trained, the generative adversarial network is configured to generate refined images that are indistinguishable from the object 406 in the second frame 116. The generator module 710 receives the rendered image 608 from the rendering module 708 as an input, and the generator module 710 outputs a refined image of the object 406 in the second frame 116. In this way, the generator module 710 is implemented to generate refined image candidates which are then evaluated by the discriminator module 712, e.g., to determine whether the candidates are real or fake 714. A goal of the generator module 710 is therefore to generate a candidate that is considered real by the discriminator module 712, e.g., through comparison to a ground truth. Accordingly, the generator module 710 is trained as part of adversarial back-and-forth communication between the generative and discriminator modules in order to generate “real” candidates”. Note that the training continues until the refined images sufficiently matching the target object images).
Regarding claim 6, Kim, Cower, and Graziosi teach all the features with respect to claim 5 as outlined above. Further, Kim teaches that the method of claim 5, wherein the method further comprises using the computed loss function for training a neural network that is used to predict the offset vectors (See Kim: Figs. 5-8, and [0061], “FIG. 8 is a flow diagram depicting a procedure 800 in an example implementation in which a generator is trained by a discriminator. Meshes are combined using a layering dimension (block 802). For example, the layering module 602 can be implemented to combine the meshes using the layering dimension to distinguish relative depth of features of an object, and the rendering module 502 can be implemented to warp the combined meshes based on initial vertex positions of the combined meshes and predicted vertex offsets. The warped mesh is rendered as an image of the object (block 804). For example, the rendering module 708 may be implemented to render the image of the object. As described above, the rendered image can be compared to the object from the second frame, and based on the comparison a loss function 218 is used to train the neural network 122”).
Regarding claim 7, Kim, Cower, and Graziosi teach all the features with respect to claim 1 as outlined above. Further, Kim, Cower, and Graziosi teach that a system for displaying a three-dimensional video comprising a plurality of frames, at a different frame rate from which the video was stored or acquired (See Cower: Fig. 2, and [0050], “In some embodiments, the decoder 202 receives a set of encoded video frames via the communication unit 239. The decoder 202 decodes the video frames, for example, by decrypting (e.g., when the video is encrypted) and decompressing the encoded video frames. Once the decoder 202 decodes the video frames, in conventional video processing, the video frames are typically rendered for display. However, per techniques described herein, the decoder 202 skips scheduling of the video frames for presentation, e.g., when the frame rate of the received video is lower than a frame rate for the video application 103. As a result, instead of scheduling the video frames for presentation, the decoder 202 provides each of the decoded video frames to the video analyzer 204. For example, the set of video frames includes a first video frame and a second video frame, where the second video frame is subsequent to the first video frame, e.g., is associated with a timestamp that is later than a timestamp associated with the first video frame, such that the second video frame occupies a later position in a chronological sequence of frames than the first video frame. In some embodiments, the second video frame directly follows the first video frame, for example, the two frames are identified for sequential display with no other frames in between”), the system (See Kim: Figs. 1-2, and [0032], “FIG. 1 is an illustration of an environment 100 in an example implementation that is operable to employ digital systems and techniques as described herein. The illustrated environment 100 includes a computing device 102 connected to a network 104. The computing device 102 may be configured as a desktop computer, a laptop computer, a mobile device (e.g., assuming a handheld configuration such as a tablet or mobile phone), and so forth. Thus, the computing device 102 may range from a full resource device with substantial memory and processor resources (e.g., personal computers, game consoles) to a low-resource device with limited memory and/or processing resources (e.g., mobile devices). Additionally, the computing device 102 may be representative of a plurality of different devices, such as multiple servers utilized by a business to perform operations “over the cloud.””; [0003], “Systems and techniques are described for object animation using generative neural networks. A computing device implements an animation system which receives a digital video having an animated object as an input. The animation system includes a meshing module, a warping module, and a training module. The meshing module obtains a mesh of the animated object from a first frame of the digital video. This mesh has a plurality of vertices which correspond to features of the object in the first frame. The animation system selects a second frame from the digital video which also has the animated object but in a different orientation than in the first frame. For example, the object might be standing with its arms at its sides in the first frame and the object may have its right arm raised with its right elbow bent in the second frame”; and [0036], “FIG. 2 depicts a system 200 in an example implementation showing operation of an animation system 110. As described above, the animation system 110 receives as an input a digital video having frames of an animated object, and the animation system 110 employs the neural network 122 to output a reproduction of the animated object. In more general terms, the animated object in the digital video is an object. In one or more implementations, the neural network 122 can include an encoder to analyze an image of an object and a decoder to predict vertex offsets for vertices of a mesh of the object”) comprising:
a video reader and decoder configured to receive a frame N corresponding to a frame number N in the video (See Kim: Figs. 1-2, and [0024], “The selection module selects a second frame from the digital video which also has the animated object but in a different orientation than in the first frame. In the first frame, the object might be standing with its arms at its sides, but the object may have its right arm raised with its right elbow bent in the second frame. The identification module identifies features of the animated object in the second frame using an encoder of a neural network”; and [0029], “The described systems and techniques improve digital animation by allowing digital animators to transfer an animation of an object to another object. This improvement also significantly increases the efficiency of digital animation. These systems and techniques improve a user experience for digital animators by eliminating the tedious task of animating an object to have features of another animated object. Additionally, the systems and techniques can improve computational efficiency by using neural networks to transfer animation from an object to another object instead of requiring computations to render animation definitions for every object requiring animation”. Note that the system receives and processes frames from the digital video sequentially with frame N (such as the first or second frame) in the meshing module using neural networks);
the video reader and decoder further configured to retrieve a 3D mesh including a plurality of vertices, a texture and one or more offset vectors associated with each vertex of the 3D mesh in the frame N (See Kim: Figs. 1-5, and [0025], “The animation system also includes a mapping module and a prediction module. The mapping module maps the identified features of the object in the second frame to the vertices of the combined mesh. Next, the prediction module generates a vertex offset for the mesh using a decoder of the neural network. This vertex offset is a prediction of how the combined mesh is to be deformed to reproduce the animated object as it appears in the second frame”; [0046], “In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video”; and [0052], “FIG. 5 is an illustration depicting a representation 500 of a rendering of a warped mesh as an image of the object. The representation 500 is illustrated to include a rendering module 502 implemented to render an image 504 of a warped mesh. In one example, the rendering module 502 may be a Neural Renderer. The rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 504 of the object 406”. Note that the meshing module generate mesh with plurality of vertices corresponding to object features in the first frame (N)), and
transmit, to a rendering device, the 3D mesh, the texture, the and offset vectors associated with the frame N (See Kim: Figs. 1-3, and [0035], “According to various implementations, such a machine-learning model uses supervised learning, semi-supervised learning, unsupervised learning, or reinforcement learning. For example, the machine-learning model can include, but is not limited to, clustering, decision trees, support vector machines, linear regression, logistic regression, Bayesian networks, random forest learning, dimensionality reduction algorithms, boosting algorithms, artificial neural networks (e.g., fully-connected neural networks, deep convolutional neural networks, or recurrent neural networks), deep learning, etc. Thus, a machine-learning model makes high-level abstractions in data by generating data-driven predictions or decisions from the known input data. In this manner, the animation system 110 receives a digital video having an animated object as an input, and the animation system 110 employs the neural network 122 to output a reproduction of the animated object. In one or more implementations, the animation system 110 may output the reproduction of the animated object without having any information about the animated object in the digital video”; [0045], “The warping module 212 warps the mesh based on the vertex offset (block 314) and the training module 214 trains the neural network based on a comparison of the warped mesh with the object from the second frame (block 316). For example, the training module 214 may use the loss function 218 to train the neural network to improve the quality of the reproduction of the object in the second frame 116”; and [0046], “In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video”. Note that the mesh and vertex offset are passed to the warping and rendering module (neural renderer) to generate the output images);
the rendering device configured to adjust all vertices of the 3D mesh according to the corresponding offset vectors and a playback time moment, wherein the playback time moment is between N and N+1 (See Kim: Figs. 1-2, and [0038], “The animation system 110 is also illustrated to include a selection module 204. The selection module 204 is implemented to select a second frame 116 from the digital video having the object. An identification module 206 identifies features of the object from the second frame 116 as part of the neural network 122. The neural network 122 includes a mapping module 208 and a prediction module 210. When implemented, the mapping module 208 maps the identified features of the object in the second frame 116 to the vertices of the mesh, and the prediction module 210 generates vertex offsets based on this mapping. Finally, a warping module 212 of the animation system 110 warps the mesh based on the vertex offsets to reproduce the object as it appears in the second frame 116”; [0047], “In one or more implementations, regularization may help avoid overfitting as part of training the neural network 122. Regularization is a form of regression which regulates or minimizes coefficients of the lost function 218, and this helps prevent the neural network 122 from fitting noise in training. This also improves generalization in machine learning. For example, Laplacian regularization may be used to calculate distances between a Laplacian coordinate of a vertex before and after warping the mesh based on the predicted vertex offset. In another example, edge-length regularization may be used to penalize differences in edge lengths before and after warping the mesh”; Figs. 4 and 9, and [0064], “In one example, the animation system 904 may include an encoder and a decoder of a neural network to predict vertex offsets for warping the combined mesh 902 to match features of the object 406 from the second frame 116. In another example, the animation system 904 may include a Neural Renderer to receive faces, textures, initial vertex positions, and the predicted vertex offsets to warp the combined mesh 902 and render the warped mesh as an image of the object 406. For example, the neural network may be trained using a loss function based on a comparison of the rendered image of the object and the object 406. In an example, the animation system 904 may include a generator and a discriminator of a generative adversarial network. For example, the generator may generate a refined image from the rendered image and the discriminator may receive the refined image and a ground truth to determine whether the refined image is real or fake. In one or more implementations, the generator can be trained by backpropagation of a result of the discriminator's determination whether the refined image is considered real or fake. In this way, the animation system 904 may include a Neural Renderer to render the warped mesh as an image of the object 406 such that the image of the object is a reproduction of the object 406, and the animation system may include a generative adversarial network to refine the image of the object at a high frequency to correct the warping effects that make the rendered image of the object distinguishable from the object 406. In other words, the refined image can correct high frequency features of the rendered image which make it distinguishable from the object 406. Thus, the animation system 904 can transfer the motions of an object in a digital video to another object without needing any information about the definitions describing the motions of the object in the video”. Note that warping adjusts vertices by offsets to match target appearance, and warps mesh based on the vertex offset to reproduce objects in the second frame N+1, enabling intermediate deformations for animation sequences), wherein N+1 is a frame number corresponding to a frame N+1 immediately following the frame Nin the video (Se Cower: Fig. 8, and [0086], “Frame sequence 820 includes upsampled video and includes frames 811, 822, 823, 824, 815, 826, 827, 828, and 819, which are displayed at a higher framerate. From the received video stream that includes frames 811, 815, and 819, additional frames are obtained by interpolation, as described herein. The additional frames in frame sequence 820 are 822, 823, 824, 826, 827, and 828, and are obtained by interpolation and added to the sequence according to the techniques described herein. As a result of using interpolated frames, the frame sequence 820 can be displayed at a higher framerate (since intermediate frames 822-824 and 826-828 are available) with no jumpiness while the bandwidth utilized to receive the video remains the same as that for frame sequence 810”; and [0089], “In some embodiments, the number of intermediate frames that are generated between the first frame and the second frame is based on a difference in the frame rate of the received video and the frame rate at which the video is to be played back. For example, if the video is received at 10 FPS and is to be played by at 30 FPS, 2 intermediate frames are generated for each pair of consecutive frames of the video. In this instance, if the original video has 100 frames (corresponding to 10 seconds of video), 200 intermediate frames are generated to obtain a video of 300 frames which can be played back at 30 FPS for 10 seconds of video. The frame rate for video playback may be selected based on various factors, e.g., the refresh rate of display 241, the computational capabilities of device 200, the rate of motion in the received video, etc.”); and
the rendering device configured to rendering the adjusted 3D mesh and the texture according to camera parameters associated with a display device (See (Graziosi: Fig. 1, and [0041], “To address voxelization, mesh scaling and offset information is sent in Atlas Adaptation Parameter Set (AAPS). Available camera parameters are able to be used. Alternatively, new syntax elements are introduced for voxelization (where only scaling and offset are used). The following is exemplary syntax”), wherein the display device is configured to display the video (See Kim: Figs. 4-6, and [0052], “FIG. 5 is an illustration depicting a representation 500 of a rendering of a warped mesh as an image of the object. The representation 500 is illustrated to include a rendering module 502 implemented to render an image 504 of a warped mesh. In one example, the rendering module 502 may be a Neural Renderer. The rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 504 of the object 406”; [0055], “In one or more implementations, the animation system 110 generates the mesh 404 of the object as it appears in the first frame 114 from the first mesh 604 and the second mesh 606 using the layering dimension which is an output of the layering module 602. In one example, the output of the layering module 602 is an input to the rendering module 502. Thus, the rendering module 502 receives as an input an initial mesh 404 and predicted vertex offsets. The initial mesh 404 may include faces, textures, and initial vertex positions. Thus, the rendering module 502 can receive faces, textures, initial vertex positions, and predicted vertex offsets as inputs. In this way, the rendering module 502 is implemented to warp the initial mesh 404 based on the initial vertex positions and the predicted vertex offsets to appear as the object 406, and then render the warped mesh as an image 608 of the object 406”; and Figs. 1-2, and [0033], “The illustrated environment 100 also includes a display device 106 that is communicatively coupled to the computing device 102 via a wired or a wireless connection. A variety of device configurations may be used to implement the computing device 102 and/or the display device 106. The computing device 102 includes a storage device 108 and an animation system 110. The storage device 108 is illustrated to include digital content 112. An example of the digital content 112 is a digital video such as a digital video having an animated object interacting with an environment. The digital video is illustrated as frames 114-120 shown on the display device 106. The frames 114-120 are snapshots of the object as it moves in the digital video. Thus, the digital video may be observed as a series of frames 114-120 on a display of the display device 106. For example, in frame 114, the object is standing with its arms at its side, but in frame 116, the object's right arm is raised as if to direct the viewer's attention to something of interest. The object appears relaxed with its left hand partially in its pants pocket in frame 118, and the object appears to be giving a thumb's up with its left hand in frame 120. Thus, when frames 114-120 are displayed in succession, the object's various poses in each of the frames 114-120 may appear as the animation of the object in the digital video”. Note that the neural 3D renderer renders the warped mesh, and outputs the rendered images to the display device).
Regarding claim 8, Kim, Cower, and Graziosi teach all the features with respect to claim 7 as outlined above. Further, Kim teaches that the system of claim 7, wherein the retrieving the offset vector further comprises:
substantially optimizing a closeness measure between the frame N and the frame N+1 (See Kim: Fig. 2, and [0039], “The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections”); and
determining the offset vectors based on the substantially optimized closeness measure (See Kim: Figs. 1-7, and [0038], “The animation system 110 is also illustrated to include a selection module 204. The selection module 204 is implemented to select a second frame 116 from the digital video having the object. An identification module 206 identifies features of the object from the second frame 116 as part of the neural network 122. The neural network 122 includes a mapping module 208 and a prediction module 210. When implemented, the mapping module 208 maps the identified features of the object in the second frame 116 to the vertices of the mesh, and the prediction module 210 generates vertex offsets based on this mapping. Finally, a warping module 212 of the animation system 110 warps the mesh based on the vertex offsets to reproduce the object as it appears in the second frame 116”; [0050], “Thus, a technological problem addressed in the following is how to manipulate the mesh 404 to appear as the object 406. In one or more implementations, a solution to this technological problem includes generating offsets to the vertices of the mesh 404 and training the neural network 122 to predict offsets to the vertices of the mesh 404 such that the predicted offsets to the vertices warp the mesh 404 to appear as the object 406”; and [0059], “The animation system 702 includes a generative adversarial network which is illustrated as a generator module 710 and a discriminator module 712. Once trained, the generative adversarial network is configured to generate refined images that are indistinguishable from the object 406 in the second frame 116. The generator module 710 receives the rendered image 608 from the rendering module 708 as an input, and the generator module 710 outputs a refined image of the object 406 in the second frame 116. In this way, the generator module 710 is implemented to generate refined image candidates which are then evaluated by the discriminator module 712, e.g., to determine whether the candidates are real or fake 714. A goal of the generator module 710 is therefore to generate a candidate that is considered real by the discriminator module 712, e.g., through comparison to a ground truth. Accordingly, the generator module 710 is trained as part of adversarial back-and-forth communication between the generative and discriminator modules in order to generate “real” candidates”. Note that the offsets are generated and predicted by the neural networks, and refined and optimized through training to achieve closeness or minimizing the cost/loss function).
Regarding claim 9, Kim, Cower, and Graziosi teach all the features with respect to claim 7 as outlined above. Further, Kim teaches that the system of claim 7, wherein the retrieving the offset vectors further comprises:
determining a closeness measure between frame N and frame N+1 via a neural network (See Kim: Figs. 1-2, and [0039], “The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections”); and
determining the offset vectors based on the determined closeness measure (See Kim: Figs. 1-3, and [0044], “The selection module 204 selects a second frame 116 from the digital video that includes the object (block 306) which is the same object in the first frame 114 but in another configuration. For example, in the first frame 114, the object could be the animated object standing with its arms at its sides, and in the second frame 116, the object could be the animated object with its right arm raised and bent at the elbow. The identification module 206 identifies features of the object from the second frame (block 308), and the mapping module 208 maps the identified features of the object to vertices of the mesh (block 310). In this manner, the animation system 110 determines how to manipulate the vertices of the mesh from the first frame 114 to reproduce the object as it appears in the second frame 116. The prediction module 210 then generates a vertex offset of the vertices of the mesh based on the mapping (block 312)”).
Regarding claim 10, Kim, Cower, and Graziosi teach all the features with respect to claim 7 as outlined above. Further, Kim teaches that the system of claim 7, wherein the retrieving the offset vectors further comprises: determining one or more offset vectors such that rasterized projections on a set of two-dimensional views of a scene in the video match one or more optical flow fields between the frame N and the frame N+1 of the scene for the two-dimensional views (See Kim: Fig. 9, and [0064], “In one example, the animation system 904 may include an encoder and a decoder of a neural network to predict vertex offsets for warping the combined mesh 902 to match features of the object 406 from the second frame 116. In another example, the animation system 904 may include a Neural Renderer to receive faces, textures, initial vertex positions, and the predicted vertex offsets to warp the combined mesh 902 and render the warped mesh as an image of the object 406. For example, the neural network may be trained using a loss function based on a comparison of the rendered image of the object and the object 406. In an example, the animation system 904 may include a generator and a discriminator of a generative adversarial network. For example, the generator may generate a refined image from the rendered image and the discriminator may receive the refined image and a ground truth to determine whether the refined image is real or fake. In one or more implementations, the generator can be trained by backpropagation of a result of the discriminator's determination whether the refined image is considered real or fake. In this way, the animation system 904 may include a Neural Renderer to render the warped mesh as an image of the object 406 such that the image of the object is a reproduction of the object 406, and the animation system may include a generative adversarial network to refine the image of the object at a high frequency to correct the warping effects that make the rendered image of the object distinguishable from the object 406. In other words, the refined image can correct high frequency features of the rendered image which make it distinguishable from the object 406. Thus, the animation system 904 can transfer the motions of an object in a digital video to another object without needing any information about the definitions describing the motions of the object in the video”).
Regarding claim 11, Kim, Cower, and Graziosi teach all the features with respect to claim 7 as outlined above. Further, Kim teaches that the system of claim 7, further comprising:
comparing a rendered image of offset-adjusted frame N, defined as the frame N after the adjusting, with a rendered image of the frame N+1 (See Kim: Figs. 1-2, and [0046], “In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video”);
computing a loss function based on the comparison (See Kim: Figs. 1-2, and [0039], “The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections”); and
adjusting the offset vectors based on the loss function (See Kim: Figs. 1-2, and [0039], “The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections”; and Fig. 7, and [0059], “The animation system 702 includes a generative adversarial network which is illustrated as a generator module 710 and a discriminator module 712. Once trained, the generative adversarial network is configured to generate refined images that are indistinguishable from the object 406 in the second frame 116. The generator module 710 receives the rendered image 608 from the rendering module 708 as an input, and the generator module 710 outputs a refined image of the object 406 in the second frame 116. In this way, the generator module 710 is implemented to generate refined image candidates which are then evaluated by the discriminator module 712, e.g., to determine whether the candidates are real or fake 714. A goal of the generator module 710 is therefore to generate a candidate that is considered real by the discriminator module 712, e.g., through comparison to a ground truth. Accordingly, the generator module 710 is trained as part of adversarial back-and-forth communication between the generative and discriminator modules in order to generate “real” candidates”. Note that the training continues until the refined images sufficiently matching the target object images).
Regarding claim 12, Kim, Cower, and Graziosi teach all the features with respect to claim 11 as outlined above. Further, Kim teaches that the system of claim 11, wherein the method further comprises using the computed loss function for training a neural network that is used to predict the offset vectors (See Kim: Figs. 5-8, and [0061], “FIG. 8 is a flow diagram depicting a procedure 800 in an example implementation in which a generator is trained by a discriminator. Meshes are combined using a layering dimension (block 802). For example, the layering module 602 can be implemented to combine the meshes using the layering dimension to distinguish relative depth of features of an object, and the rendering module 502 can be implemented to warp the combined meshes based on initial vertex positions of the combined meshes and predicted vertex offsets. The warped mesh is rendered as an image of the object (block 804). For example, the rendering module 708 may be implemented to render the image of the object. As described above, the rendered image can be compared to the object from the second frame, and based on the comparison a loss function 218 is used to train the neural network 122”).
Regarding claim 13, Kim, Cower, and Graziosi teach all the features with respect to claim 1 as outlined above. Further, Kim teaches that a machine-readable storage medium storing a set of instructions that are executable by one or more processors of a system, video reader and decoder, and a rendering device for displaying a three-dimensional video at a different frame rate from which the video was stored or acquired, wherein the set of instructions is configured to perform the method of claim 1 (See Kim: Fig. 10, and [0075], ““Computer-readable signal media” may refer to a signal-bearing medium that is configured to transmit instructions to the hardware of the computing device 1002, such as via a network. Signal media typically may embody computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as carrier waves, data signals, or other transport mechanism. Signal media also include any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media”).
Regarding claim 14, Kim, Cower, and Graziosi teach all the features with respect to claim 1 as outlined above. Further, Kim, Cower, and Graziosi teach that a method for training a neural network for offset vectors determination, the method (See Kim: Figs. 1-2, and [0039], “The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections”) comprising:
comparing a rendered image of an offset-adjusted mesh of a frame N corresponding to a frame number N in a video further comprised of a plurality of frames, with a rendered image of the mesh of a frame N+1 corresponding to a frame number N+1 in the video (See Kim: Figs. 1-2, and [0046], “In one example, the neural network 122 may include an encoder for analyzing the object in the second frame 116, a decoder for predicting the vertex offsets to the mesh from the first frame 114, and a Neural Renderer as generally described by Kato et al., Neural 3D Mesh Renderer, arXiv:1711.07566v1 [cs.CV] 20 Nov. 2017, for rendering the mesh with the vertex offsets as an image. The comparison module 216 may be implemented to compare the rendered image to the object in the second frame 116, and a loss function 218 is then used to train the neural network 122. In this manner, the neural network reproduces the animation from the digital video without having or needing any information about the animation from the digital video”);
computing a loss function based on the comparing (See Kim: Figs. 1-2, and [0039], “The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections”); and
training a neural network used to predict the offset vectors with the loss function (See Kim: Figs. 1-2, and [0039], “The animation system 110 includes a training module 214. The training module 214 includes a comparison module 216 and a loss function 218. The comparison module 216 compares the warped mesh to the object from the second frame 116, and the loss function 218 is used to train the neural network 122. In one example, the loss function 218 may describe reconstruction loss. In this manner, the neural network “learns” how to reproduce the animated object using, solely in this example, the mesh from the first frame 114 of the digital video and offsets to the vertices of the mesh. Further discussion of these and other examples is included in the following sections”; and Fig. 7, and [0059], “The animation system 702 includes a generative adversarial network which is illustrated as a generator module 710 and a discriminator module 712. Once trained, the generative adversarial network is configured to generate refined images that are indistinguishable from the object 406 in the second frame 116. The generator module 710 receives the rendered image 608 from the rendering module 708 as an input, and the generator module 710 outputs a refined image of the object 406 in the second frame 116. In this way, the generator module 710 is implemented to generate refined image candidates which are then evaluated by the discriminator module 712, e.g., to determine whether the candidates are real or fake 714. A goal of the generator module 710 is therefore to generate a candidate that is considered real by the discriminator module 712, e.g., through comparison to a ground truth. Accordingly, the generator module 710 is trained as part of adversarial back-and-forth communication between the generative and discriminator modules in order to generate “real” candidates”. Note that the training continues until the refined images sufficiently matching the target object images).
Regarding claim 15, Kim, Cower, and Graziosi teach all the features with respect to claim 14 as outlined above. Further, Kim teaches that a machine-readable storage medium storing a set of instructions that are executable by one or more processors of a system for training a neural network for offset determination, wherein the set of instructions are configured to perform the method of claim 14 (See Kim: Fig. 10, and [0075], ““Computer-readable signal media” may refer to a signal-bearing medium that is configured to transmit instructions to the hardware of the computing device 1002, such as via a network. Signal media typically may embody computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as carrier waves, data signals, or other transport mechanism. Signal media also include any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media”).
Regarding claim 16, Kim, Cower, and Graziosi teach all the features with respect to claim 1 as outlined above. Further, Kim teaches that the method of claim 1, wherein the playback time interval between N and N+1 is a semi-interval [N, N+1) (See Kim: Figs. 1-2, and [0044], “The selection module 204 selects a second frame 116 from the digital video that includes the object (block 306) which is the same object in the first frame 114 but in another configuration. For example, in the first frame 114, the object could be the animated object standing with its arms at its sides, and in the second frame 116, the object could be the animated object with its right arm raised and bent at the elbow. The identification module 206 identifies features of the object from the second frame (block 308), and the mapping module 208 maps the identified features of the object to vertices of the mesh (block 310). In this manner, the animation system 110 determines how to manipulate the vertices of the mesh from the first frame 114 to reproduce the object as it appears in the second frame 116. The prediction module 210 then generates a vertex offset of the vertices of the mesh based on the mapping (block 312)”. Note that offset adjustments and rendering for smooth animation between frame N and N+1 generates intermediate representation for time between these two frames, naturally operating over the time interval starting at one frame and progressively toward another frame without overlapping, i.e. [N, N+1) (forward animation) or (N, N+1] (backward animation), ad the most common one is [N, N+1) because N frame is normally received before N+1 frame).
Regarding claim 17, Kim, Cower, and Graziosi teach all the features with respect to claim 1 as outlined above. Further, Graziosi teaches that the method of claim 1, wherein the display device is a virtual reality headset (See Graziosi: Fig. 24, and [0096], “Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, a smart phone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer/player (e.g., DVD writer/player, high definition disc writer/player, ultra high definition disc writer/player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., smart watch), a vehicle (e.g., a self-driving vehicle) or any other suitable computing device”).
Regarding claim 18, Kim, Cower, and Graziosi teach all the features with respect to claim 7 as outlined above. Further, Kim teaches that the system of claim 7, wherein the display device is a virtual reality headset (See Graziosi: Fig. 24, and [0096], “Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile telephone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, a smart phone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer/player (e.g., DVD writer/player, high definition disc writer/player, ultra high definition disc writer/player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., smart watch), a vehicle (e.g., a self-driving vehicle) or any other suitable computing device”).
Regarding claim 19, Kim, Cower, and Graziosi teach all the features with respect to claim 7 as outlined above. Further, Kim teaches that the system of claim 7, wherein the playback time interval between N and N+1 is a semi-interval [N, N+1) (See Kim: Figs. 1-2, and [0044], “The selection module 204 selects a second frame 116 from the digital video that includes the object (block 306) which is the same object in the first frame 114 but in another configuration. For example, in the first frame 114, the object could be the animated object standing with its arms at its sides, and in the second frame 116, the object could be the animated object with its right arm raised and bent at the elbow. The identification module 206 identifies features of the object from the second frame (block 308), and the mapping module 208 maps the identified features of the object to vertices of the mesh (block 310). In this manner, the animation system 110 determines how to manipulate the vertices of the mesh from the first frame 114 to reproduce the object as it appears in the second frame 116. The prediction module 210 then generates a vertex offset of the vertices of the mesh based on the mapping (block 312)”. Note that offset adjustments and rendering for smooth animation between frame N and N+1 generates intermediate representation for time between these two frames, naturally operating over the time interval starting at one frame and progressively toward another frame without overlapping, i.e. [N, N+1) (forward animation) or (N, N+1] (backward animation), and the most common one is [N, N+1) because N frame is normally received before N+1 frame).
Regarding claim 20, Kim, Cower, and Graziosi teach all the features with respect to claim 1 as outlined above. Further, Graziosi teaches that the method of claim 1, wherein the three-dimensional video is constructed from any of a random distribution of cameras pointing towards one or more three-dimensional scenes, and multiview-stereo data (See Graziosi: Fig. 1, and [0032], “A method of compression of 3D mesh data using projections of mesh surface data and video representation of connectivity data is described herein. The method utilizes 3D surface patches to represent a set of connected triangles on a mesh surface. The projected surface data is stored in patches (a mesh patch) that is encoded in atlas data. The connectivity of the mesh, that is, the vertices and the triangles of the surface patch, are encoded using video-based compression techniques. The data is encapsulated in a new video component named vertex video data, and the disclosed structure allows for progressive mesh coding by separating sets of vertices in layers, and creating levels of detail for the mesh connectivity. This approach extends the functionality of the V3C (volumetric video-based) standard, currently being used for coding of point cloud and multiview plus depth content”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights