DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-8 and 10-21 are pending under this Office action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5, 10-12, 15, 17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Bouaziz, etc. (US 20140362091 A1) in view of Rosenfeld (US 20020101422 A1).
Regarding claim 1, Bouaziz teaches that a facial expression simulation method (See Bouaziz: Fig. 5, and [0017], “Embodiments of the subject facial animation method according to the present disclosure provide for real-time face tracking and animation and require no user-specific training or calibration or any other form of manual assistance, thus enabling a broad range of applications of performance-based facial animation and virtual interaction, for example, at consumer level. Embodiments can raise tracking quality while keeping the acquisition system simple enough for consumer-level applications and avoiding any manual system calibration or training. In particular, embodiments of the method require neither user-specific pre-processing, nor any calibration or user-assisted training, thereby making the tracking system directly operational for any new user”; and [0089], “FIG. 5 is a flowchart of a method according to one embodiment of the present disclosure. The method 500, which may be a computer-implemented method, may start in step 502 by providing a dynamic expression model including a plurality of blendshapes. In step 504, tracking data or input data corresponding to facial expressions of a user may be received, such as the input data 302 discussed with reference to FIG. 3”), comprising:
collecting a local facial image to be processed of a target object (See Bouaziz: Fig. 5, and [0010], “A first aspect of the present disclosure provides a method for real-time facial animation, comprising providing a dynamic expression model and receiving tracking data corresponding to a facial expression of a user. Tracking parameters are estimated based on the dynamic expression model and the tracking data. Furthermore, the dynamic expression model is refined based on the tracking data and the estimated tracking parameters”; [0018], “According to yet another preferred embodiment, said estimating of tracking parameters, such as weights for the blendshapes of the dynamic expression model, is performed in a first stage, and said refining of the dynamic expression model is performed in a second stage, wherein the first stage and the second stage are iteratively repeated. Accordingly, in the first stage a rigid alignment of the tracking data and tracking parameters, such as the blendshape weights, may be estimated keeping the dynamic expression model fixed. In the second stage, the user-specific dynamic expression model may be refined keeping the tracking parameters fixed. Hence, while the facial tracking is accomplished in real-time, the dynamic expression model may be continuously refined to the currently tracked user following an online modeling approach. For example, a fixed number of blendshapes of the dynamic expression model can be refined to the facial performance and geometry of the tracked user. The refinement approach is advantageous, since it needs not to extend the dynamic expression model, for example by adding further blendshapes. Using a fixed number of blendshapes optimizes memory consumption and computational performance”; and [0089], “FIG. 5 is a flowchart of a method according to one embodiment of the present disclosure. The method 500, which may be a computer-implemented method, may start in step 502 by providing a dynamic expression model including a plurality of blendshapes. In step 504, tracking data or input data corresponding to facial expressions of a user may be received, such as the input data 302 discussed with reference to FIG. 3”), and
generating an expression coefficient corresponding to the local facial image to be processed (See Bouaziz: Figs. 3 and 5, and [0013], “In a preferred embodiment, the dynamic expression model includes a plurality of blendshapes and the tracking parameters include weights for the blendshapes”; and [0014], “The blendshapes of the dynamic expression model may be organized as a set of blendshapes, wherein each blendshape may correspond to a polygon mesh or point cloud or any other representation of a geometrical 3D surface suitable for representing a facial expression. Each blendshape may preferably corresponds to a pre-defined facial expression, for example, matching pre-defined semantics of common face animation controllers such as smile, frown, mouth-open, etc. Preferably, the plurality of blendshapes may include 3D meshes having the same static mesh combinatorics, which may be represented by stacked coordinate vectors offering a compact representation”. Note that the blendshapes and tracking parameters including the weights is mapped to the facial expression coefficient);
wherein the local facial image to be processed belongs to an expression image sequence (See Bouaziz: Figs. 1-5, and [0011], “The tracking data may be organized in frames, wherein each frame of tracking data corresponds to a particular facial expression of the user captured in this frame. Accordingly, the tracking data may include one or more frames and each frame of tracking data may correspond to a current facial expression of the user according to the facial performance of the user. For example, the tracking data may be provided as optical 3D and/or 2D data, such as a series of video frames including depth information, which may be provided by commodity, RGB-D sensing devices. Yet, the present disclosure is not limited to a particular sensing device or optical data only and may further include electro-magnetic or acoustic tracking data. Each frame of tracking data may be used to estimate the tracking parameters, which may be further used to generate a graphical representation for the current facial expression corresponding to the current frame. The frame of tracking data in combination with the estimated tracking parameters may be further used for refinement of the dynamic expression model. Accordingly, an initial dynamic expression model may be directly used for tracking and is continuously refined to better match the facial characteristics of the user according to an online modeling approach. In this description the term online modeling is used in the sense of an online algorithm or approach, which processes input piece-by-piece in a serial fashion, for example, in the order that the input is provided to the algorithm, without having the entire input, such as all frames of tracking data, available from the start. Hence, an online algorithm could be understood in contrast to an offline algorithm which directly requires the entire input data. Therefore, the dynamic expression model may be continuously refined using a current piece of tracking data in each frame in a serial fashion”; and [0021], “In yet another embodiment, the method further comprises receiving tracking data corresponding to a neutral facial expression of the user and initializing the dynamic expression model using the tracking data corresponding to the neutral facial expression of the user. The user may, for example, enter a field of view of a tracking sensor in a neutral facial expression. The corresponding tracking data may be used to initialize at least one of the plurality of blendshapes of the dynamic expression model in order to reflect the neutral facial expression of the user. This initial approximation of the neutral facial expression may be further refined in subsequent alterations, such as alterations of the first and second stages”. Note that the first stage estimation and second stage refinement is explicitly mapped to coefficients (weights) generation and refinement to sequence position or order), and the expression coefficient is determined on the basis of the position of the local facial image to be processed in the expression image sequence; and
simulating a facial expression of the target object according to the expression coefficient (See Bouaziz: Fig. 3-7, and [0069], “The term "real-time" used throughout this disclosure refers to a performance of a computing system or processing device subject to timing constraints, which specify operational deadlines from input or a processing event to an output or a corresponding response. Accordingly, computing or processing systems operating in real-time must guarantee a response according to strict timing conditions, for example, within a range of milliseconds. Preferably, in media systems a real-time response should be delivered without a perceivable delay for the user. For example, a graphical output should be kept at constant frame rates of at least 15 Hz with a latency to the user input of at least 150 milliseconds. Preferably, the frame rates are within a range of 20 Hz to 150 Hz, such as within two of 20, 25, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140 and 150 Hz and, most preferably, at 25 Hz. The latency may be preferably at least 160 milliseconds, preferably within a range of 10 milliseconds to 160 milliseconds, such as within two of 150, 140, 130, 120, 110, 100, 90, 80, 70, 60, 50, 40, 30, 20, and 10 milliseconds, and most preferably of 150 milliseconds. The real-time performance of embodiments of the present disclosure can be achieved by separation of the tracking refinement 308 and model refinement 310. The interactive generation of the virtual avatar 314 can be accomplished using blendshapes and the computed blendshape weights. Concurrently, the user-specific dynamic expression model may be selectively refined in order to meet the timing constraints”; [0085], “FIG. 4 shows four example images including the acquired image data for one frame and the resulting facial expression of the virtual avatar generated according to one embodiment of the present disclosure. The resulting facial expression of the virtual avatar may correspond to the virtual avatar 314 as shown in FIG. 3. Furthermore, the acquired image data may correspond to the image data 304 of the input data 302 as shown in FIG. 3”; and [0097], “FIG. 7 shows a comparison of different blendshape weights used to generate a resulting facial expression, including a comparison between l.sub.1 and l.sub.2 regularization for the blendshape weight optimization according to Equation (1) discussed above. The l.sub.1 regularization leads to a lower average fitting error of 2.27 mm compared to 2.72 mm for the l.sub.2 regularization. The l.sub.1 regularization also significantly reduces the number of non-zero blendshape weights. Accordingly, the l.sub.1 regularization leads to a significant speed-up of the subsequent model refinement stage, since blendshape refinement is only performed on blendshapes with non-zero blendshape weights”. Note that facial animation is generated based on blendshapes and weights, and this is mapped to the current limitation).
However, Bouaziz fails to explicitly disclose that and the expression coefficient is determined on the basis of the position of the local facial image to be processed in the expression image sequence.
However, Rosenfeld teaches that and the expression coefficient is determined on the basis of the position of the local facial image to be processed in the expression image sequence (See Rosenfeld: Figs. 1-3, and [0024], “In accordance with the present invention, there is provided as illustrated in FIGS. 1-3, a method for controlling and automatically animating lip synchronization and facial expressions of three dimensional animated characters using weighted morph targets and time aligned phonetic transcriptions of recorded text. The method utilizes a set of rules that determine the systems output comprising a stream of morph weight sets when a sequence of timed phonemes is encountered. Other timed data, such as timed emotional state data or emotemes such as "surprise, "disgust, "embarrassment", "timid smile", pitch, amplitued, noise amounts or the like, may be inputted to affect the output stream of morph weight sets”; [0026], “There is also provided, according to the invention a method for automatically animating lip synchronization and facial expression of three dimensional characters for use with a computer animation system, comprising the steps of: determining means for producing a stream of morph weight sets when a sequence of phonemes is encountered; evaluating a plurality of time aligned phonetic transcriptions or other timed data such as pitch, amplitude, noise amounts and the like, against the determining means for producing a stream of morph weight sets; applying said determining means for producing a stream of morph weight sets to generate an output morph weight set stream, allowing for an appropriate morph weight set correspondence with each of a plurality of time aligned phonetic transcription sub-sequences and correct time parameters applied to a plurality of morph weight set transitions between a representation of a prior time aligned phonetic transcription subsequence and a current one, whereby lip synchronization and facial expressions of animated characters is automatically controlled and produced”; [0031], “If for example, specific TAPT subsequence does not fit the criteria for any secondary rules, then the default rules take effect. If, on the other hand, the TAPT sub-sequence does fit the criteria for a secondary rule(s) they take precedence over the default rules. A TAPT sub-sequence take into account the current phoneme and duration, and a number of the preceding and following phonemes and duration's as well may be specified”; and [0036], “With reference now to FIG. 3, method 10 for automatically animating lip synchronization and facial expression of three dimensional characters for use with a computer animation system is shown including box 56 showing the step of determining means for producing a stream of morph weight sets when a sequence of phonemes is encountered. Box 53, showing the step of evaluating a plurality of time aligned phonetic transcriptions or other timed ata such as pitch, amplitude, noise amounts, and the like, against said determining means for producing a stream of morph weight sets. In box 60 the steps of applying said determining means for producing a stream of morph weight sets to generate an output morph weight set stream, allowing for an appropriate morph weight set correspondence with each of a plurality of time aligned phonetic transcription sub-sequences and correct time parameters applied to a plurality of morph weight set transitions between a representation of a prior time aligned phonetic transcription sub-sequence and a current one, whereby lip synchronization and facial expressions of animated characters is automatically controlled and produced are shown according to the invention”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Bouaziz to have and the expression coefficient is determined on the basis of the position of the local facial image to be processed in the expression image sequence as taught by Rosenfeld in order to automatically create lip synchronization and facial expression in three dimensional animated characters in extremely rapid and cost effective manner by specifying morph weight set transition rules (See Rosenfeld: Figs.1-3, and [0014], “Accordingly, it is the primary object of this invention to provide a method for automatically animating lip synchronization and facial expression of three dimensional characters, which is integrated with computer means for producing accurate and realistic lip synchronization and facial expressions in animated characters. The method of the present invention further provides an extremely rapid and cost effective means to automatically create lip synchronization and facial expression in three dimensional animated characters”). Bouaziz teaches a method and system that may generate real-time facial animation by refining the dynamic expression model based on the facial tracking parameters; while Rosenfeld teaches a system and method that may control and automatically animate lip synchronization and facial expressions of three dimensional animated characters using weighted morph targets and time aligned phonetic transcriptions of recorded text based on the exact timing position in the sequences. Therefore, it is obvious to one of ordinary skill in the art to modify Bouaziz by Rosenfeld to determine the facial expression coefficients (weights) based on the timing position in animation sequences. The motivation to modify Bouaziz by Rosenfeld is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 2, Bouaziz and Rosenfeld teach all the features with respect to claim 1 as outlined above. Further, Rosenfeld teaches that the method according to claim 1, wherein the expression image sequence comprises a plurality of key images of a target expression action, the time sequence relationship of the plurality of key images in the expression image sequence characterizes an action process of the target expression action, and the plurality of key images divide the expression image sequence into at least two expression action intervals (See Rosenfeld: Figs. 1-3, and [0025], “The method comprises, in one embodiment, configuring a set of default correspondence rules between a plurality of visual phoneme groups and a plurality of morph weight sets; and specifying a plurality of morph weight set transition rules for specifying durational data for the generation of transitionary curses between the plurality of morph weight sets, allowing for the production of a stream of specified morph weight sets to be processed by a computer animation system for integration with other animation, whereby animated lip synchronization and facial expression of animated characters may be automatically produced”; and [0030], “The rules of the present method may be categorized in three main groupings; default rules, auxiliary rules and post processing rules. The default rules must be complete enough to create valid output for any TAPT encountered at any point in the TAPT. The secondary rules are used in special cases; for example, to substitute alternative morph weight set correspondences and/or transition rules if the identified criteria are met. The post processing rules are used to further manipulate the morph weight set stream after the default or secondary rules are applied, and can further modify the members of the morph weight sets determined by the default and secondary rules and interpolation”. Note that the stream of weights with transitionary points is mapped to the key frames); and
wherein the expression coefficient is determined based on the expression action interval in which the local facial image to be processed is located (See Rosenfeld: Figs. 1-3, and [0034], “In FIG. 1, a flow chart illustrates the preferred steps of the methodology 10 or automatically animating lip synchronization and facial expression of three dimensional animated characters of the present invention. A specific sub-sequence 20 is selected from the TAPT file 12 and is evaluated 22 to determine if any secondary rule criteria for morph weight set target apply. Time aligned emotional transcription file 14 data may be inputted or data from an optional time aligned data file 16 may be used. Also shown is a parallel method 18 which may be configured identical to the primary method described, however, using different timed data rules and different delta sets. Sub-sequence 20 is evaluated 22 to determine if any secondary rule criteria apply. If yes, then a morph weight set is assigned 24 according to the secondary rules, if no, then a morph weight set is assigned 26 according to the default rules. If the sub-string meets any secondary rule criteria for transition specification 28 then a transition start and end time are assigned according to the secondary rules 32, if no, then assign transition start and end times 30 according to default rules. Then an intermediate file of transition keyframes using target weights and transition rules as generated are created 34, and if any keyframe sequences fit post process before interpolation rules they are applied here 36. This data may be output 38 here if desired. If not, then interpolate using any method post processed keyframes to a desired frequency or frame rate 40 and if any morph weight sequences generated fit post processing after interpolation criteria, they are applied 42 at this point. If parallel methods or systems are used to process other timed aligned data, they may be concatenated here 44, and the data output 46”; and [0041], “The post processing rules may be applied either before or after the above described interpolation step, or both. Some rules may apply only to keyframes before interpolation, some to interpolated data. If applied before the interpolation step, this affects the keyframes. if applied after, it effects the interpolated data. Post processing can use the morph weight sets calculated by the default and secondary rules. Post processing rules can use the morph weigh sets or sequences as in box 44 of FIG. 1, calculated by the default and secondary rules. Post processing rules can modify the individual members of the morph weight sets previously generated. Post processing rules may be applied in addition to other rules, including other post processing rules. Once the rule set up is completed as described, the method of the present invention can take any number and length TAPT's as input, and automatically output the corresponding morph weight set stream as seen in FIGS. 1-3”. Note that rules evaluate current and adjacent sub-sequences and timing to determine weights for positions within those intervals, ad this is mapped to the current limitation).
Regarding claim 5, Bouaziz and Rosenfeld teach all the features with respect to claim 1 as outlined above. Further, Bouaziz teaches that the method according to claims 1, wherein simulating a facial expression of the target object according to the expression coefficient comprises:
recognizing an expression type characterized by the express10n image sequence, and determining a target feature dimension corresponding to the expression type among a plurality of facial feature dimensions of the target object (See Bouaziz: Figs. 1-3, and [0060], “The blendshape b.sub.0 representing the neutral facial expression may be approximated to the face geometry of a current user by applying an identity model 104 of the dynamic expression model 100. The identity model 104 may include a mean face m, which may be derived from a large set of meshes of different human faces with one-to-one vertex correspondences in neutral expressions. Furthermore, the identity model 104 may include a plurality of eigenvectors. In an embodiment, the identity model 104 can be an identity PCA model 104, which may be generated using principle component analysis (PCA) on stacked vertex coordinate vectors of respective meshes of the large set of meshes of different human faces. For example, the identity PCA model 104 may include the first l PCA eigenvectors P=[p.sub.1, . . . , p.sub.l] and the blendshape b.sub.0 for the neutral facial expression may be approximated as b.sub.0=m+Py with suitable linear coefficients y=[y.sub.1, . . . , y.sub.l].sup.T.”; and [0072], “During model refinement 310, the identity PCA parameters y=[y.sub.1, . . . , y.sub.l].sup.T for the neutral face expression b.sub.0 of the user and the deformation coefficients Z={z.sub.1, . . . , z.sub.n}, with z.sub.i=[z.sub.i,1, . . . , z.sub.i,k].sup.T for each blendshape b.sub.i are determined”. Note that PCA analysis is mapped to identify the expression type);
wherein respective facial feature dimensions of the target object have their respective original expression coefficients (See Bouaziz: Figs. 1-3, and [0020], “According to one embodiment, the plurality of blendshapes at least includes a blendshape b.sub.0 representing a neutral facial expression and the dynamic expression model further includes an identity principal component analysis (PCA) model, the method further including matching the blendshape b.sub.0 representing the neutral facial expression to the neutral expression of the user based on the tracking data and the identity PCA model. The identity PCA model may represent variations of face geometries across different users and may be used to initialize the plurality of blendshapes including the blendshape b.sub.0 to the face geometry of the user. The variations of face geometries may be, for example, captured with a morphable model as, for example, proposed by V. Blanz V. and T. Vetter in "A morphable model for the syntheses of 3D faces", SIGGRAPH 1999, which is incorporated herein in its entirety. Given a large set of meshes of different human faces with a one-to-one vertex correspondence in neutral expression, a reduced representation may be built using PCA on stacked vertex coordinate vectors of the meshes. The identity PCA model may include a resulting mean face and one or more eigenvectors forming an orthonormal basis. Accordingly the blendshape b.sub.0 representing the neutral facial expression of a specific user can be estimated as a linear combination of the mean face and at least some of the eigenvectors with suitable linear coefficients, such that the blendshape b.sub.0 approximates the facial expression represented by the tracking data”; and [0060], “The blendshape b.sub.0 representing the neutral facial expression may be approximated to the face geometry of a current user by applying an identity model 104 of the dynamic expression model 100. The identity model 104 may include a mean face m, which may be derived from a large set of meshes of different human faces with one-to-one vertex correspondences in neutral expressions. Furthermore, the identity model 104 may include a plurality of eigenvectors. In an embodiment, the identity model 104 can be an identity PCA model 104, which may be generated using principle component analysis (PCA) on stacked vertex coordinate vectors of respective meshes of the large set of meshes of different human faces. For example, the identity PCA model 104 may include the first l PCA eigenvectors P=[p.sub.1, . . . , p.sub.l] and the blendshape b.sub.0 for the neutral facial expression may be approximated as b.sub.0=m+Py with suitable linear coefficients y=[y.sub.1, . . . , y.sub.l].sup.T.”; and [0072], “During model refinement 310, the identity PCA parameters y=[y.sub.1, . . . , y.sub.l].sup.T for the neutral face expression b.sub.0 of the user and the deformation coefficients Z={z.sub.1, . . . , z.sub.n}, with z.sub.i=[z.sub.i,1, . . . , z.sub.i,k].sup.T for each blendshape b.sub.i are determined”. Note that blendshape parameterization with the original neural facial expression is mapped to the original expression); and
correcting an original expression coefficient of the target feature dimension to the expression coefficient corresponding to the local facial image to be processed, and simulating the facial expression of the target object according to the corrected expression coefficients of the respective facial feature dimensions (See Bouaziz: Figs. 1-3, and [0060], “The blendshape b.sub.0 representing the neutral facial expression may be approximated to the face geometry of a current user by applying an identity model 104 of the dynamic expression model 100. The identity model 104 may include a mean face m, which may be derived from a large set of meshes of different human faces with one-to-one vertex correspondences in neutral expressions. Furthermore, the identity model 104 may include a plurality of eigenvectors. In an embodiment, the identity model 104 can be an identity PCA model 104, which may be generated using principle component analysis (PCA) on stacked vertex coordinate vectors of respective meshes of the large set of meshes of different human faces. For example, the identity PCA model 104 may include the first l PCA eigenvectors P=[p.sub.1, . . . , p.sub.l] and the blendshape b.sub.0 for the neutral facial expression may be approximated as b.sub.0=m+Py with suitable linear coefficients y=[y.sub.1, . . . , y.sub.l].sup.T.”; and [0076], “For example, the alignment may be achieved by minimizing the common iterative closest point (ICP) energy with point-plane constraints and solving for the PCA coefficients y, the deformation coefficient z.sub.0, and the rigid head pose (R, t). The optimization problem can be defined as: arg min R , t , y , z 0 A 0 ( Rb 0 + t ) - c 0 2 2 + .beta. 1 D P y 2 2 + .beta. 2 D E z 0 2 2 + .beta. 3 z 0 2 2 . ##EQU00003## . In this formulation, (A.sub.0, c.sub.0) summarizes the ICP constraint equations in the first term of the objective function. The remaining summands are regularization terms with corresponding positive scalar weights .beta..sub.1, .beta..sub.2, and .beta..sub.3. The term D.sub.py regularizes the PCA weights, where D.sub.P is a diagonal matrix containing the inverse of the standard deviation of the PCA basis. The term D.sub.EZ.sub.0 regularizes the deformation coefficients by measuring the bending of the deformation. D.sub.E is the diagonal matrix of eigenvalues corresponding to the eigenvectors in E of the Laplacian matrix L. The last summand penalizes the magnitude of the deformation vectors”. Note that PCA analysis with optimization under the ICP constraints is mapped to correct the expression coefficient).
Regarding claim 10, Bouaziz and Rosenfeld teach all the features with respect to claim 1 as outlined above. Further, Bouaziz and Rosenfeld teach that an electronic device, comprising a memory and a processor, wherein the memory is configured to store a computer program which, when executed by the processor, implements a facial expression simulation method (See Bouaziz: Fig. 5, and [0017], “Embodiments of the subject facial animation method according to the present disclosure provide for real-time face tracking and animation and require no user-specific training or calibration or any other form of manual assistance, thus enabling a broad range of applications of performance-based facial animation and virtual interaction, for example, at consumer level. Embodiments can raise tracking quality while keeping the acquisition system simple enough for consumer-level applications and avoiding any manual system calibration or training. In particular, embodiments of the method require neither user-specific pre-processing, nor any calibration or user-assisted training, thereby making the tracking system directly operational for any new user”; and [0089], “FIG. 5 is a flowchart of a method according to one embodiment of the present disclosure. The method 500, which may be a computer-implemented method, may start in step 502 by providing a dynamic expression model including a plurality of blendshapes. In step 504, tracking data or input data corresponding to facial expressions of a user may be received, such as the input data 302 discussed with reference to FIG. 3”), comprising:
collecting a local facial image to be processed of a target object (See Bouaziz: Fig. 5, and [0010], “A first aspect of the present disclosure provides a method for real-time facial animation, comprising providing a dynamic expression model and receiving tracking data corresponding to a facial expression of a user. Tracking parameters are estimated based on the dynamic expression model and the tracking data. Furthermore, the dynamic expression model is refined based on the tracking data and the estimated tracking parameters”; [0018], “According to yet another preferred embodiment, said estimating of tracking parameters, such as weights for the blendshapes of the dynamic expression model, is performed in a first stage, and said refining of the dynamic expression model is performed in a second stage, wherein the first stage and the second stage are iteratively repeated. Accordingly, in the first stage a rigid alignment of the tracking data and tracking parameters, such as the blendshape weights, may be estimated keeping the dynamic expression model fixed. In the second stage, the user-specific dynamic expression model may be refined keeping the tracking parameters fixed. Hence, while the facial tracking is accomplished in real-time, the dynamic expression model may be continuously refined to the currently tracked user following an online modeling approach. For example, a fixed number of blendshapes of the dynamic expression model can be refined to the facial performance and geometry of the tracked user. The refinement approach is advantageous, since it needs not to extend the dynamic expression model, for example by adding further blendshapes. Using a fixed number of blendshapes optimizes memory consumption and computational performance”; and [0089], “FIG. 5 is a flowchart of a method according to one embodiment of the present disclosure. The method 500, which may be a computer-implemented method, may start in step 502 by providing a dynamic expression model including a plurality of blendshapes. In step 504, tracking data or input data corresponding to facial expressions of a user may be received, such as the input data 302 discussed with reference to FIG. 3”), and generating an expression coefficient corresponding to the local facial image to be processed (See Bouaziz: Figs. 3 and 5, and [0013], “In a preferred embodiment, the dynamic expression model includes a plurality of blendshapes and the tracking parameters include weights for the blendshapes”; and [0014], “The blendshapes of the dynamic expression model may be organized as a set of blendshapes, wherein each blendshape may correspond to a polygon mesh or point cloud or any other representation of a geometrical 3D surface suitable for representing a facial expression. Each blendshape may preferably corresponds to a pre-defined facial expression, for example, matching pre-defined semantics of common face animation controllers such as smile, frown, mouth-open, etc. Preferably, the plurality of blendshapes may include 3D meshes having the same static mesh combinatorics, which may be represented by stacked coordinate vectors offering a compact representation”. Note that the blendshapes and tracking parameters including the weights is mapped to the facial expression coefficient);
wherein the local facial image to be processed belongs to an expression image sequence (See Bouaziz: Figs. 1-5, and [0011], “The tracking data may be organized in frames, wherein each frame of tracking data corresponds to a particular facial expression of the user captured in this frame. Accordingly, the tracking data may include one or more frames and each frame of tracking data may correspond to a current facial expression of the user according to the facial performance of the user. For example, the tracking data may be provided as optical 3D and/or 2D data, such as a series of video frames including depth information, which may be provided by commodity, RGB-D sensing devices. Yet, the present disclosure is not limited to a particular sensing device or optical data only and may further include electro-magnetic or acoustic tracking data. Each frame of tracking data may be used to estimate the tracking parameters, which may be further used to generate a graphical representation for the current facial expression corresponding to the current frame. The frame of tracking data in combination with the estimated tracking parameters may be further used for refinement of the dynamic expression model. Accordingly, an initial dynamic expression model may be directly used for tracking and is continuously refined to better match the facial characteristics of the user according to an online modeling approach. In this description the term online modeling is used in the sense of an online algorithm or approach, which processes input piece-by-piece in a serial fashion, for example, in the order that the input is provided to the algorithm, without having the entire input, such as all frames of tracking data, available from the start. Hence, an online algorithm could be understood in contrast to an offline algorithm which directly requires the entire input data. Therefore, the dynamic expression model may be continuously refined using a current piece of tracking data in each frame in a serial fashion”; and [0021], “In yet another embodiment, the method further comprises receiving tracking data corresponding to a neutral facial expression of the user and initializing the dynamic expression model using the tracking data corresponding to the neutral facial expression of the user. The user may, for example, enter a field of view of a tracking sensor in a neutral facial expression. The corresponding tracking data may be used to initialize at least one of the plurality of blendshapes of the dynamic expression model in order to reflect the neutral facial expression of the user. This initial approximation of the neutral facial expression may be further refined in subsequent alterations, such as alterations of the first and second stages”. Note that the first stage estimation and second stage refinement is explicitly mapped to coefficients (weights) generation and refinement to sequence position or order), and the expression coefficient is determined on the basis of the position of the local facial image to be processed in the expression image sequence (See Rosenfeld: Figs. 1-3, and [0024], “In accordance with the present invention, there is provided as illustrated in FIGS. 1-3, a method for controlling and automatically animating lip synchronization and facial expressions of three dimensional animated characters using weighted morph targets and time aligned phonetic transcriptions of recorded text. The method utilizes a set of rules that determine the systems output comprising a stream of morph weight sets when a sequence of timed phonemes is encountered. Other timed data, such as timed emotional state data or emotemes such as "surprise, "disgust, "embarrassment", "timid smile", pitch, amplitued, noise amounts or the like, may be inputted to affect the output stream of morph weight sets”; [0026], “There is also provided, according to the invention a method for automatically animating lip synchronization and facial expression of three dimensional characters for use with a computer animation system, comprising the steps of: determining means for producing a stream of morph weight sets when a sequence of phonemes is encountered; evaluating a plurality of time aligned phonetic transcriptions or other timed data such as pitch, amplitude, noise amounts and the like, against the determining means for producing a stream of morph weight sets; applying said determining means for producing a stream of morph weight sets to generate an output morph weight set stream, allowing for an appropriate morph weight set correspondence with each of a plurality of time aligned phonetic transcription sub-sequences and correct time parameters applied to a plurality of morph weight set transitions between a representation of a prior time aligned phonetic transcription subsequence and a current one, whereby lip synchronization and facial expressions of animated characters is automatically controlled and produced”; [0031], “If for example, specific TAPT subsequence does not fit the criteria for any secondary rules, then the default rules take effect. If, on the other hand, the TAPT sub-sequence does fit the criteria for a secondary rule(s) they take precedence over the default rules. A TAPT sub-sequence take into account the current phoneme and duration, and a number of the preceding and following phonemes and duration's as well may be specified”; and [0036], “With reference now to FIG. 3, method 10 for automatically animating lip synchronization and facial expression of three dimensional characters for use with a computer animation system is shown including box 56 showing the step of determining means for producing a stream of morph weight sets when a sequence of phonemes is encountered. Box 53, showing the step of evaluating a plurality of time aligned phonetic transcriptions or other timed ata such as pitch, amplitude, noise amounts, and the like, against said determining means for producing a stream of morph weight sets. In box 60 the steps of applying said determining means for producing a stream of morph weight sets to generate an output morph weight set stream, allowing for an appropriate morph weight set correspondence with each of a plurality of time aligned phonetic transcription sub-sequences and correct time parameters applied to a plurality of morph weight set transitions between a representation of a prior time aligned phonetic transcription sub-sequence and a current one, whereby lip synchronization and facial expressions of animated characters is automatically controlled and produced are shown according to the invention”); and
simulating a facial expression of the target object according to the expression coefficient (See Bouaziz: Fig. 3-7, and [0069], “The term "real-time" used throughout this disclosure refers to a performance of a computing system or processing device subject to timing constraints, which specify operational deadlines from input or a processing event to an output or a corresponding response. Accordingly, computing or processing systems operating in real-time must guarantee a response according to strict timing conditions, for example, within a range of milliseconds. Preferably, in media systems a real-time response should be delivered without a perceivable delay for the user. For example, a graphical output should be kept at constant frame rates of at least 15 Hz with a latency to the user input of at least 150 milliseconds. Preferably, the frame rates are within a range of 20 Hz to 150 Hz, such as within two of 20, 25, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140 and 150 Hz and, most preferably, at 25 Hz. The latency may be preferably at least 160 milliseconds, preferably within a range of 10 milliseconds to 160 milliseconds, such as within two of 150, 140, 130, 120, 110, 100, 90, 80, 70, 60, 50, 40, 30, 20, and 10 milliseconds, and most preferably of 150 milliseconds. The real-time performance of embodiments of the present disclosure can be achieved by separation of the tracking refinement 308 and model refinement 310. The interactive generation of the virtual avatar 314 can be accomplished using blendshapes and the computed blendshape weights. Concurrently, the user-specific dynamic expression model may be selectively refined in order to meet the timing constraints”; [0085], “FIG. 4 shows four example images including the acquired image data for one frame and the resulting facial expression of the virtual avatar generated according to one embodiment of the present disclosure. The resulting facial expression of the virtual avatar may correspond to the virtual avatar 314 as shown in FIG. 3. Furthermore, the acquired image data may correspond to the image data 304 of the input data 302 as shown in FIG. 3”; and [0097], “FIG. 7 shows a comparison of different blendshape weights used to generate a resulting facial expression, including a comparison between l.sub.1 and l.sub.2 regularization for the blendshape weight optimization according to Equation (1) discussed above. The l.sub.1 regularization leads to a lower average fitting error of 2.27 mm compared to 2.72 mm for the l.sub.2 regularization. The l.sub.1 regularization also significantly reduces the number of non-zero blendshape weights. Accordingly, the l.sub.1 regularization leads to a significant speed-up of the subsequent model refinement stage, since blendshape refinement is only performed on blendshapes with non-zero blendshape weights”. Note that facial animation is generated based on blendshapes and weights, and this is mapped to the current limitation).
Regarding claim 11, Bouaziz and Rosenfeld teach all the features with respect to claim 1 as outlined above. Further, Bouaziz and Rosenfeld teach that a non-transitory computer-readable storage medium, configured to store a computer program which, when executed by a processor, implements a facial expression simulation method (See Bouaziz: Fig. 5, and [0017], “Embodiments of the subject facial animation method according to the present disclosure provide for real-time face tracking and animation and require no user-specific training or calibration or any other form of manual assistance, thus enabling a broad range of applications of performance-based facial animation and virtual interaction, for example, at consumer level. Embodiments can raise tracking quality while keeping the acquisition system simple enough for consumer-level applications and avoiding any manual system calibration or training. In particular, embodiments of the method require neither user-specific pre-processing, nor any calibration or user-assisted training, thereby making the tracking system directly operational for any new user”; and [0089], “FIG. 5 is a flowchart of a method according to one embodiment of the present disclosure. The method 500, which may be a computer-implemented method, may start in step 502 by providing a dynamic expression model including a plurality of blendshapes. In step 504, tracking data or input data corresponding to facial expressions of a user may be received, such as the input data 302 discussed with reference to FIG. 3”), comprising:
collecting a local facial image to be processed of a target object (See Bouaziz: Fig. 5, and [0010], “A first aspect of the present disclosure provides a method for real-time facial animation, comprising providing a dynamic expression model and receiving tracking data corresponding to a facial expression of a user. Tracking parameters are estimated based on the dynamic expression model and the tracking data. Furthermore, the dynamic expression model is refined based on the tracking data and the estimated tracking parameters”; [0018], “According to yet another preferred embodiment, said estimating of tracking parameters, such as weights for the blendshapes of the dynamic expression model, is performed in a first stage, and said refining of the dynamic expression model is performed in a second stage, wherein the first stage and the second stage are iteratively repeated. Accordingly, in the first stage a rigid alignment of the tracking data and tracking parameters, such as the blendshape weights, may be estimated keeping the dynamic expression model fixed. In the second stage, the user-specific dynamic expression model may be refined keeping the tracking parameters fixed. Hence, while the facial tracking is accomplished in real-time, the dynamic expression model may be continuously refined to the currently tracked user following an online modeling approach. For example, a fixed number of blendshapes of the dynamic expression model can be refined to the facial performance and geometry of the tracked user. The refinement approach is advantageous, since it needs not to extend the dynamic expression model, for example by adding further blendshapes. Using a fixed number of blendshapes optimizes memory consumption and computational performance”; and [0089], “FIG. 5 is a flowchart of a method according to one embodiment of the present disclosure. The method 500, which may be a computer-implemented method, may start in step 502 by providing a dynamic expression model including a plurality of blendshapes. In step 504, tracking data or input data corresponding to facial expressions of a user may be received, such as the input data 302 discussed with reference to FIG. 3”), and generating an expression coefficient corresponding to the local facial image to be processed (See Bouaziz: Figs. 3 and 5, and [0013], “In a preferred embodiment, the dynamic expression model includes a plurality of blendshapes and the tracking parameters include weights for the blendshapes”; and [0014], “The blendshapes of the dynamic expression model may be organized as a set of blendshapes, wherein each blendshape may correspond to a polygon mesh or point cloud or any other representation of a geometrical 3D surface suitable for representing a facial expression. Each blendshape may preferably corresponds to a pre-defined facial expression, for example, matching pre-defined semantics of common face animation controllers such as smile, frown, mouth-open, etc. Preferably, the plurality of blendshapes may include 3D meshes having the same static mesh combinatorics, which may be represented by stacked coordinate vectors offering a compact representation”. Note that the blendshapes and tracking parameters including the weights is mapped to the facial expression coefficient);
wherein the local facial image to be processed belongs to an expression image sequence (See Bouaziz: Figs. 1-5, and [0011], “The tracking data may be organized in frames, wherein each frame of tracking data corresponds to a particular facial expression of the user captured in this frame. Accordingly, the tracking data may include one or more frames and each frame of tracking data may correspond to a current facial expression of the user according to the facial performance of the user. For example, the tracking data may be provided as optical 3D and/or 2D data, such as a series of video frames including depth information, which may be provided by commodity, RGB-D sensing devices. Yet, the present disclosure is not limited to a particular sensing device or optical data only and may further include electro-magnetic or acoustic tracking data. Each frame of tracking data may be used to estimate the tracking parameters, which may be further used to generate a graphical representation for the current facial expression corresponding to the current frame. The frame of tracking data in combination with the estimated tracking parameters may be further used for refinement of the dynamic expression model. Accordingly, an initial dynamic expression model may be directly used for tracking and is continuously refined to better match the facial characteristics of the user according to an online modeling approach. In this description the term online modeling is used in the sense of an online algorithm or approach, which processes input piece-by-piece in a serial fashion, for example, in the order that the input is provided to the algorithm, without having the entire input, such as all frames of tracking data, available from the start. Hence, an online algorithm could be understood in contrast to an offline algorithm which directly requires the entire input data. Therefore, the dynamic expression model may be continuously refined using a current piece of tracking data in each frame in a serial fashion”; and [0021], “In yet another embodiment, the method further comprises receiving tracking data corresponding to a neutral facial expression of the user and initializing the dynamic expression model using the tracking data corresponding to the neutral facial expression of the user. The user may, for example, enter a field of view of a tracking sensor in a neutral facial expression. The corresponding tracking data may be used to initialize at least one of the plurality of blendshapes of the dynamic expression model in order to reflect the neutral facial expression of the user. This initial approximation of the neutral facial expression may be further refined in subsequent alterations, such as alterations of the first and second stages”. Note that the first stage estimation and second stage refinement is explicitly mapped to coefficients (weights) generation and refinement to sequence position or order), and the expression coefficient is determined on the basis of the position of the local facial image to be processed in the expression image sequence (See Rosenfeld: Figs. 1-3, and [0024], “In accordance with the present invention, there is provided as illustrated in FIGS. 1-3, a method for controlling and automatically animating lip synchronization and facial expressions of three dimensional animated characters using weighted morph targets and time aligned phonetic transcriptions of recorded text. The method utilizes a set of rules that determine the systems output comprising a stream of morph weight sets when a sequence of timed phonemes is encountered. Other timed data, such as timed emotional state data or emotemes such as "surprise, "disgust, "embarrassment", "timid smile", pitch, amplitued, noise amounts or the like, may be inputted to affect the output stream of morph weight sets”; [0026], “There is also provided, according to the invention a method for automatically animating lip synchronization and facial expression of three dimensional characters for use with a computer animation system, comprising the steps of: determining means for producing a stream of morph weight sets when a sequence of phonemes is encountered; evaluating a plurality of time aligned phonetic transcriptions or other timed data such as pitch, amplitude, noise amounts and the like, against the determining means for producing a stream of morph weight sets; applying said determining means for producing a stream of morph weight sets to generate an output morph weight set stream, allowing for an appropriate morph weight set correspondence with each of a plurality of time aligned phonetic transcription sub-sequences and correct time parameters applied to a plurality of morph weight set transitions between a representation of a prior time aligned phonetic transcription subsequence and a current one, whereby lip synchronization and facial expressions of animated characters is automatically controlled and produced”; [0031], “If for example, specific TAPT subsequence does not fit the criteria for any secondary rules, then the default rules take effect. If, on the other hand, the TAPT sub-sequence does fit the criteria for a secondary rule(s) they take precedence over the default rules. A TAPT sub-sequence take into account the current phoneme and duration, and a number of the preceding and following phonemes and duration's as well may be specified”; and [0036], “With reference now to FIG. 3, method 10 for automatically animating lip synchronization and facial expression of three dimensional characters for use with a computer animation system is shown including box 56 showing the step of determining means for producing a stream of morph weight sets when a sequence of phonemes is encountered. Box 53, showing the step of evaluating a plurality of time aligned phonetic transcriptions or other timed ata such as pitch, amplitude, noise amounts, and the like, against said determining means for producing a stream of morph weight sets. In box 60 the steps of applying said determining means for producing a stream of morph weight sets to generate an output morph weight set stream, allowing for an appropriate morph weight set correspondence with each of a plurality of time aligned phonetic transcription sub-sequences and correct time parameters applied to a plurality of morph weight set transitions between a representation of a prior time aligned phonetic transcription sub-sequence and a current one, whereby lip synchronization and facial expressions of animated characters is automatically controlled and produced are shown according to the invention”); and
simulating a facial expression of the target object according to the expression coefficient (See Bouaziz: Fig. 3-7, and [0069], “The term "real-time" used throughout this disclosure refers to a performance of a computing system or processing device subject to timing constraints, which specify operational deadlines from input or a processing event to an output or a corresponding response. Accordingly, computing or processing systems operating in real-time must guarantee a response according to strict timing conditions, for example, within a range of milliseconds. Preferably, in media systems a real-time response should be delivered without a perceivable delay for the user. For example, a graphical output should be kept at constant frame rates of at least 15 Hz with a latency to the user input of at least 150 milliseconds. Preferably, the frame rates are within a range of 20 Hz to 150 Hz, such as within two of 20, 25, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140 and 150 Hz and, most preferably, at 25 Hz. The latency may be preferably at least 160 milliseconds, preferably within a range of 10 milliseconds to 160 milliseconds, such as within two of 150, 140, 130, 120, 110, 100, 90, 80, 70, 60, 50, 40, 30, 20, and 10 milliseconds, and most preferably of 150 milliseconds. The real-time performance of embodiments of the present disclosure can be achieved by separation of the tracking refinement 308 and model refinement 310. The interactive generation of the virtual avatar 314 can be accomplished using blendshapes and the computed blendshape weights. Concurrently, the user-specific dynamic expression model may be selectively refined in order to meet the timing constraints”; [0085], “FIG. 4 shows four example images including the acquired image data for one frame and the resulting facial expression of the virtual avatar generated according to one embodiment of the present disclosure. The resulting facial expression of the virtual avatar may correspond to the virtual avatar 314 as shown in FIG. 3. Furthermore, the acquired image data may correspond to the image data 304 of the input data 302 as shown in FIG. 3”; and [0097], “FIG. 7 shows a comparison of different blendshape weights used to generate a resulting facial expression, including a comparison between l.sub.1 and l.sub.2 regularization for the blendshape weight optimization according to Equation (1) discussed above. The l.sub.1 regularization leads to a lower average fitting error of 2.27 mm compared to 2.72 mm for the l.sub.2 regularization. The l.sub.1 regularization also significantly reduces the number of non-zero blendshape weights. Accordingly, the l.sub.1 regularization leads to a significant speed-up of the subsequent model refinement stage, since blendshape refinement is only performed on blendshapes with non-zero blendshape weights”. Note that facial animation is generated based on blendshapes and weights, and this is mapped to the current limitation).
Regarding claim 12, Bouaziz and Rosenfeld teach all the features with respect to claim 10 as outlined above. Further, Rosenfeld teaches that the electronic device according to claim 10, wherein the expression image sequence comprises a plurality of key images of a target expression action, the time sequence relationship of the plurality of key images in the expression image sequence characterizes an action process of the target expression action, and the plurality of key images divide the expression image sequence into at least two expression action intervals (See Rosenfeld: Figs. 1-3, and [0025], “The method comprises, in one embodiment, configuring a set of default correspondence rules between a plurality of visual phoneme groups and a plurality of morph weight sets; and specifying a plurality of morph weight set transition rules for specifying durational data for the generation of transitionary curses between the plurality of morph weight sets, allowing for the production of a stream of specified morph weight sets to be processed by a computer animation system for integration with other animation, whereby animated lip synchronization and facial expression of animated characters may be automatically produced”; and [0030], “The rules of the present method may be categorized in three main groupings; default rules, auxiliary rules and post processing rules. The default rules must be complete enough to create valid output for any TAPT encountered at any point in the TAPT. The secondary rules are used in special cases; for example, to substitute alternative morph weight set correspondences and/or transition rules if the identified criteria are met. The post processing rules are used to further manipulate the morph weight set stream after the default or secondary rules are applied, and can further modify the members of the morph weight sets determined by the default and secondary rules and interpolation”. Note that the stream of weights with transitionary points is mapped to the key frames); and
wherein the expression coefficient is determined based on the expression action interval in which the local facial image to be processed is located (See Rosenfeld: Figs. 1-3, and [0034], “In FIG. 1, a flow chart illustrates the preferred steps of the methodology 10 or automatically animating lip synchronization and facial expression of three dimensional animated characters of the present invention. A specific sub-sequence 20 is selected from the TAPT file 12 and is evaluated 22 to determine if any secondary rule criteria for morph weight set target apply. Time aligned emotional transcription file 14 data may be inputted or data from an optional time aligned data file 16 may be used. Also shown is a parallel method 18 which may be configured identical to the primary method described, however, using different timed data rules and different delta sets. Sub-sequence 20 is evaluated 22 to determine if any secondary rule criteria apply. If yes, then a morph weight set is assigned 24 according to the secondary rules, if no, then a morph weight set is assigned 26 according to the default rules. If the sub-string meets any secondary rule criteria for transition specification 28 then a transition start and end time are assigned according to the secondary rules 32, if no, then assign transition start and end times 30 according to default rules. Then an intermediate file of transition keyframes using target weights and transition rules as generated are created 34, and if any keyframe sequences fit post process before interpolation rules they are applied here 36. This data may be output 38 here if desired. If not, then interpolate using any method post processed keyframes to a desired frequency or frame rate 40 and if any morph weight sequences generated fit post processing after interpolation criteria, they are applied 42 at this point. If parallel methods or systems are used to process other timed aligned data, they may be concatenated here 44, and the data output 46”; and [0041], “The post processing rules may be applied either before or after the above described interpolation step, or both. Some rules may apply only to keyframes before interpolation, some to interpolated data. If applied before the interpolation step, this affects the keyframes. if applied after, it effects the interpolated data. Post processing can use the morph weight sets calculated by the default and secondary rules. Post processing rules can use the morph weigh sets or sequences as in box 44 of FIG. 1, calculated by the default and secondary rules. Post processing rules can modify the individual members of the morph weight sets previously generated. Post processing rules may be applied in addition to other rules, including other post processing rules. Once the rule set up is completed as described, the method of the present invention can take any number and length TAPT's as input, and automatically output the corresponding morph weight set stream as seen in FIGS. 1-3”. Note that rules evaluate current and adjacent sub-sequences and timing to determine weights for positions within those intervals, ad this is mapped to the current limitation).
Regarding claim 15, Bouaziz and Rosenfeld teach all the features with respect to claim 10 as outlined above. Further, Bouaziz teaches that the electronic device according to claim 10, wherein simulating a facial expression of the target object according to the expression coefficient comprises:
recognizing an expression type characterized by the expression image sequence, and determining a target feature dimension corresponding to the expression type among a plurality of facial feature dimensions of the target object (See Bouaziz: Figs. 1-3, and [0060], “The blendshape b.sub.0 representing the neutral facial expression may be approximated to the face geometry of a current user by applying an identity model 104 of the dynamic expression model 100. The identity model 104 may include a mean face m, which may be derived from a large set of meshes of different human faces with one-to-one vertex correspondences in neutral expressions. Furthermore, the identity model 104 may include a plurality of eigenvectors. In an embodiment, the identity model 104 can be an identity PCA model 104, which may be generated using principle component analysis (PCA) on stacked vertex coordinate vectors of respective meshes of the large set of meshes of different human faces. For example, the identity PCA model 104 may include the first l PCA eigenvectors P=[p.sub.1, . . . , p.sub.l] and the blendshape b.sub.0 for the neutral facial expression may be approximated as b.sub.0=m+Py with suitable linear coefficients y=[y.sub.1, . . . , y.sub.l].sup.T.”; and [0072], “During model refinement 310, the identity PCA parameters y=[y.sub.1, . . . , y.sub.l].sup.T for the neutral face expression b.sub.0 of the user and the deformation coefficients Z={z.sub.1, . . . , z.sub.n}, with z.sub.i=[z.sub.i,1, . . . , z.sub.i,k].sup.T for each blendshape b.sub.i are determined”. Note that PCA analysis is mapped to identify the expression type);
wherein respective facial feature dimensions of the target object have their respective original expression coefficients (See Bouaziz: Figs. 1-3, and [0020], “According to one embodiment, the plurality of blendshapes at least includes a blendshape b.sub.0 representing a neutral facial expression and the dynamic expression model further includes an identity principal component analysis (PCA) model, the method further including matching the blendshape b.sub.0 representing the neutral facial expression to the neutral expression of the user based on the tracking data and the identity PCA model. The identity PCA model may represent variations of face geometries across different users and may be used to initialize the plurality of blendshapes including the blendshape b.sub.0 to the face geometry of the user. The variations of face geometries may be, for example, captured with a morphable model as, for example, proposed by V. Blanz V. and T. Vetter in "A morphable model for the syntheses of 3D faces", SIGGRAPH 1999, which is incorporated herein in its entirety. Given a large set of meshes of different human faces with a one-to-one vertex correspondence in neutral expression, a reduced representation may be built using PCA on stacked vertex coordinate vectors of the meshes. The identity PCA model may include a resulting mean face and one or more eigenvectors forming an orthonormal basis. Accordingly the blendshape b.sub.0 representing the neutral facial expression of a specific user can be estimated as a linear combination of the mean face and at least some of the eigenvectors with suitable linear coefficients, such that the blendshape b.sub.0 approximates the facial expression represented by the tracking data”; and [0060], “The blendshape b.sub.0 representing the neutral facial expression may be approximated to the face geometry of a current user by applying an identity model 104 of the dynamic expression model 100. The identity model 104 may include a mean face m, which may be derived from a large set of meshes of different human faces with one-to-one vertex correspondences in neutral expressions. Furthermore, the identity model 104 may include a plurality of eigenvectors. In an embodiment, the identity model 104 can be an identity PCA model 104, which may be generated using principle component analysis (PCA) on stacked vertex coordinate vectors of respective meshes of the large set of meshes of different human faces. For example, the identity PCA model 104 may include the first l PCA eigenvectors P=[p.sub.1, . . . , p.sub.l] and the blendshape b.sub.0 for the neutral facial expression may be approximated as b.sub.0=m+Py with suitable linear coefficients y=[y.sub.1, . . . , y.sub.l].sup.T.”; and [0072], “During model refinement 310, the identity PCA parameters y=[y.sub.1, . . . , y.sub.l].sup.T for the neutral face expression b.sub.0 of the user and the deformation coefficients Z={z.sub.1, . . . , z.sub.n}, with z.sub.i=[z.sub.i,1, . . . , z.sub.i,k].sup.T for each blendshape b.sub.i are determined”. Note that blendshape parameterization with the original neural facial expression is mapped to the original expression); and
correcting an original expression coefficient of the target feature dimension to the expression coefficient corresponding to the local facial image to be processed, and simulating the facial expression of the target object according to the corrected expression coefficients of the respective facial feature dimensions (See Bouaziz: Figs. 1-3, and [0060], “The blendshape b.sub.0 representing the neutral facial expression may be approximated to the face geometry of a current user by applying an identity model 104 of the dynamic expression model 100. The identity model 104 may include a mean face m, which may be derived from a large set of meshes of different human faces with one-to-one vertex correspondences in neutral expressions. Furthermore, the identity model 104 may include a plurality of eigenvectors. In an embodiment, the identity model 104 can be an identity PCA model 104, which may be generated using principle component analysis (PCA) on stacked vertex coordinate vectors of respective meshes of the large set of meshes of different human faces. For example, the identity PCA model 104 may include the first l PCA eigenvectors P=[p.sub.1, . . . , p.sub.l] and the blendshape b.sub.0 for the neutral facial expression may be approximated as b.sub.0=m+Py with suitable linear coefficients y=[y.sub.1, . . . , y.sub.l].sup.T.”; and [0076], “For example, the alignment may be achieved by minimizing the common iterative closest point (ICP) energy with point-plane constraints and solving for the PCA coefficients y, the deformation coefficient z.sub.0, and the rigid head pose (R, t). The optimization problem can be defined as: arg min R , t , y , z 0 A 0 ( Rb 0 + t ) - c 0 2 2 + .beta. 1 D P y 2 2 + .beta. 2 D E z 0 2 2 + .beta. 3 z 0 2 2 . ##EQU00003## . In this formulation, (A.sub.0, c.sub.0) summarizes the ICP constraint equations in the first term of the objective function. The remaining summands are regularization terms with corresponding positive scalar weights .beta..sub.1, .beta..sub.2, and .beta..sub.3. The term D.sub.py regularizes the PCA weights, where D.sub.P is a diagonal matrix containing the inverse of the standard deviation of the PCA basis. The term D.sub.EZ.sub.0 regularizes the deformation coefficients by measuring the bending of the deformation. D.sub.E is the diagonal matrix of eigenvalues corresponding to the eigenvectors in E of the Laplacian matrix L. The last summand penalizes the magnitude of the deformation vectors”. Note that PCA analysis with optimization under the ICP constraints is mapped to correct the expression coefficient).
Regarding claim 17, Bouaziz and Rosenfeld teach all the features with respect to claim 11 as outlined above. Further, Rosenfeld teaches that the non-transitory computer-readable storage medium according to claim 11, wherein the expression image sequence comprises a plurality of key images of a target expression action, the time sequence relationship of the plurality of key images in the expression image sequence characterizes an action process of the target expression action, and the plurality of key images divide the expression image sequence into at least two expression action intervals (See Rosenfeld: Figs. 1-3, and [0025], “The method comprises, in one embodiment, configuring a set of default correspondence rules between a plurality of visual phoneme groups and a plurality of morph weight sets; and specifying a plurality of morph weight set transition rules for specifying durational data for the generation of transitionary curses between the plurality of morph weight sets, allowing for the production of a stream of specified morph weight sets to be processed by a computer animation system for integration with other animation, whereby animated lip synchronization and facial expression of animated characters may be automatically produced”; and [0030], “The rules of the present method may be categorized in three main groupings; default rules, auxiliary rules and post processing rules. The default rules must be complete enough to create valid output for any TAPT encountered at any point in the TAPT. The secondary rules are used in special cases; for example, to substitute alternative morph weight set correspondences and/or transition rules if the identified criteria are met. The post processing rules are used to further manipulate the morph weight set stream after the default or secondary rules are applied, and can further modify the members of the morph weight sets determined by the default and secondary rules and interpolation”. Note that the stream of weights with transitionary points is mapped to the key frames); and
wherein the expression coefficient is determined based on the expression action interval in which the local facial image to be processed is located (See Rosenfeld: Figs. 1-3, and [0034], “In FIG. 1, a flow chart illustrates the preferred steps of the methodology 10 or automatically animating lip synchronization and facial expression of three dimensional animated characters of the present invention. A specific sub-sequence 20 is selected from the TAPT file 12 and is evaluated 22 to determine if any secondary rule criteria for morph weight set target apply. Time aligned emotional transcription file 14 data may be inputted or data from an optional time aligned data file 16 may be used. Also shown is a parallel method 18 which may be configured identical to the primary method described, however, using different timed data rules and different delta sets. Sub-sequence 20 is evaluated 22 to determine if any secondary rule criteria apply. If yes, then a morph weight set is assigned 24 according to the secondary rules, if no, then a morph weight set is assigned 26 according to the default rules. If the sub-string meets any secondary rule criteria for transition specification 28 then a transition start and end time are assigned according to the secondary rules 32, if no, then assign transition start and end times 30 according to default rules. Then an intermediate file of transition keyframes using target weights and transition rules as generated are created 34, and if any keyframe sequences fit post process before interpolation rules they are applied here 36. This data may be output 38 here if desired. If not, then interpolate using any method post processed keyframes to a desired frequency or frame rate 40 and if any morph weight sequences generated fit post processing after interpolation criteria, they are applied 42 at this point. If parallel methods or systems are used to process other timed aligned data, they may be concatenated here 44, and the data output 46”; and [0041], “The post processing rules may be applied either before or after the above described interpolation step, or both. Some rules may apply only to keyframes before interpolation, some to interpolated data. If applied before the interpolation step, this affects the keyframes. if applied after, it effects the interpolated data. Post processing can use the morph weight sets calculated by the default and secondary rules. Post processing rules can use the morph weigh sets or sequences as in box 44 of FIG. 1, calculated by the default and secondary rules. Post processing rules can modify the individual members of the morph weight sets previously generated. Post processing rules may be applied in addition to other rules, including other post processing rules. Once the rule set up is completed as described, the method of the present invention can take any number and length TAPT's as input, and automatically output the corresponding morph weight set stream as seen in FIGS. 1-3”. Note that rules evaluate current and adjacent sub-sequences and timing to determine weights for positions within those intervals, ad this is mapped to the current limitation).
Regarding claim 20, Bouaziz and Rosenfeld teach all the features with respect to claim 17 as outlined above. Further, Bouaziz teaches that the non-transitory computer-readable storage medium according to claim 17, wherein simulating a facial expression of the target object according to the expression coefficient comprises:
recognizing an expression type characterized by the express10n image sequence, and determining a target feature dimension corresponding to the expression type among a plurality of facial feature dimensions of the target object (See Bouaziz: Figs. 1-3, and [0060], “The blendshape b.sub.0 representing the neutral facial expression may be approximated to the face geometry of a current user by applying an identity model 104 of the dynamic expression model 100. The identity model 104 may include a mean face m, which may be derived from a large set of meshes of different human faces with one-to-one vertex correspondences in neutral expressions. Furthermore, the identity model 104 may include a plurality of eigenvectors. In an embodiment, the identity model 104 can be an identity PCA model 104, which may be generated using principle component analysis (PCA) on stacked vertex coordinate vectors of respective meshes of the large set of meshes of different human faces. For example, the identity PCA model 104 may include the first l PCA eigenvectors P=[p.sub.1, . . . , p.sub.l] and the blendshape b.sub.0 for the neutral facial expression may be approximated as b.sub.0=m+Py with suitable linear coefficients y=[y.sub.1, . . . , y.sub.l].sup.T.”; and [0072], “During model refinement 310, the identity PCA parameters y=[y.sub.1, . . . , y.sub.l].sup.T for the neutral face expression b.sub.0 of the user and the deformation coefficients Z={z.sub.1, . . . , z.sub.n}, with z.sub.i=[z.sub.i,1, . . . , z.sub.i,k].sup.T for each blendshape b.sub.i are determined”. Note that PCA analysis is mapped to identify the expression type);
wherein respective facial feature dimensions of the target object have their respective original expression coefficients (See Bouaziz: Figs. 1-3, and [0020], “According to one embodiment, the plurality of blendshapes at least includes a blendshape b.sub.0 representing a neutral facial expression and the dynamic expression model further includes an identity principal component analysis (PCA) model, the method further including matching the blendshape b.sub.0 representing the neutral facial expression to the neutral expression of the user based on the tracking data and the identity PCA model. The identity PCA model may represent variations of face geometries across different users and may be used to initialize the plurality of blendshapes including the blendshape b.sub.0 to the face geometry of the user. The variations of face geometries may be, for example, captured with a morphable model as, for example, proposed by V. Blanz V. and T. Vetter in "A morphable model for the syntheses of 3D faces", SIGGRAPH 1999, which is incorporated herein in its entirety. Given a large set of meshes of different human faces with a one-to-one vertex correspondence in neutral expression, a reduced representation may be built using PCA on stacked vertex coordinate vectors of the meshes. The identity PCA model may include a resulting mean face and one or more eigenvectors forming an orthonormal basis. Accordingly the blendshape b.sub.0 representing the neutral facial expression of a specific user can be estimated as a linear combination of the mean face and at least some of the eigenvectors with suitable linear coefficients, such that the blendshape b.sub.0 approximates the facial expression represented by the tracking data”; and [0060], “The blendshape b.sub.0 representing the neutral facial expression may be approximated to the face geometry of a current user by applying an identity model 104 of the dynamic expression model 100. The identity model 104 may include a mean face m, which may be derived from a large set of meshes of different human faces with one-to-one vertex correspondences in neutral expressions. Furthermore, the identity model 104 may include a plurality of eigenvectors. In an embodiment, the identity model 104 can be an identity PCA model 104, which may be generated using principle component analysis (PCA) on stacked vertex coordinate vectors of respective meshes of the large set of meshes of different human faces. For example, the identity PCA model 104 may include the first l PCA eigenvectors P=[p.sub.1, . . . , p.sub.l] and the blendshape b.sub.0 for the neutral facial expression may be approximated as b.sub.0=m+Py with suitable linear coefficients y=[y.sub.1, . . . , y.sub.l].sup.T.”; and [0072], “During model refinement 310, the identity PCA parameters y=[y.sub.1, . . . , y.sub.l].sup.T for the neutral face expression b.sub.0 of the user and the deformation coefficients Z={z.sub.1, . . . , z.sub.n}, with z.sub.i=[z.sub.i,1, . . . , z.sub.i,k].sup.T for each blendshape b.sub.i are determined”. Note that blendshape parameterization with the original neural facial expression is mapped to the original expression); and
correcting an original expression coefficient of the target feature dimension to the expression coefficient corresponding to the local facial image to be processed, and simulating the facial expression of the target object according to the corrected expression coefficients of the respective facial feature dimensions (See Bouaziz: Figs. 1-3, and [0060], “The blendshape b.sub.0 representing the neutral facial expression may be approximated to the face geometry of a current user by applying an identity model 104 of the dynamic expression model 100. The identity model 104 may include a mean face m, which may be derived from a large set of meshes of different human faces with one-to-one vertex correspondences in neutral expressions. Furthermore, the identity model 104 may include a plurality of eigenvectors. In an embodiment, the identity model 104 can be an identity PCA model 104, which may be generated using principle component analysis (PCA) on stacked vertex coordinate vectors of respective meshes of the large set of meshes of different human faces. For example, the identity PCA model 104 may include the first l PCA eigenvectors P=[p.sub.1, . . . , p.sub.l] and the blendshape b.sub.0 for the neutral facial expression may be approximated as b.sub.0=m+Py with suitable linear coefficients y=[y.sub.1, . . . , y.sub.l].sup.T.”; and [0076], “For example, the alignment may be achieved by minimizing the common iterative closest point (ICP) energy with point-plane constraints and solving for the PCA coefficients y, the deformation coefficient z.sub.0, and the rigid head pose (R, t). The optimization problem can be defined as: arg min R , t , y , z 0 A 0 ( Rb 0 + t ) - c 0 2 2 + .beta. 1 D P y 2 2 + .beta. 2 D E z 0 2 2 + .beta. 3 z 0 2 2 . ##EQU00003## . In this formulation, (A.sub.0, c.sub.0) summarizes the ICP constraint equations in the first term of the objective function. The remaining summands are regularization terms with corresponding positive scalar weights .beta..sub.1, .beta..sub.2, and .beta..sub.3. The term D.sub.py regularizes the PCA weights, where D.sub.P is a diagonal matrix containing the inverse of the standard deviation of the PCA basis. The term D.sub.EZ.sub.0 regularizes the deformation coefficients by measuring the bending of the deformation. D.sub.E is the diagonal matrix of eigenvalues corresponding to the eigenvectors in E of the Laplacian matrix L. The last summand penalizes the magnitude of the deformation vectors”. Note that PCA analysis with optimization under the ICP constraints is mapped to correct the expression coefficient).
Claims 3-4, 13-14, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Bouaziz, etc. (US 20140362091 A1) in view of Rosenfeld (US 20020101422 A1), further in view of Maurer, etc. (US 6580811 B2).
Regarding claim 3, Bouaziz and Rosenfeld teach all the features with respect to claim 2 as outlined above. However, Bouaziz, modified by Rosenfeld, fails to explicitly disclose that the method according to claim 2, wherein the key images in the expression image sequence are determined by: recognizing position information of a preset facial feature in respective local facial images of the expression image sequence; determining a plurality of expression change critical nodes according to the position information, and using the local facial images corresponding to the expression change critical nodes as the key images in the expression image sequence.
However, Maurer teaches that the method according to claim 2, wherein the key images in the expression image sequence are determined by:
recognizing position information of a preset facial feature in respective local facial images of the expression image sequence (See Maurer: Figs. 1-8, Col. 3 Lines 61-67 ~ Col. 4 Lines 1-32, “The facial feature may be located using an elastic graph matching shown in FIG. 4. In the elastic graph matching technique, a captured image (block 40) is transformed into Gabor space using a wavelet transformation (block 42) which is described below in more detail with respect to FIG. 5. The transformed image (block 44) is represented by 40 complex values, representing wavelet components, per each pixel of the original image. Next, a rigid copy of a model graph, which is described in more detail below with respect to FIG. 7, is positioned over the transformed image at varying model node positions to locate a position of optimum similarity (block 46). The search for the optimum similarity may be performed by positioning the model graph in the upper left hand corner of the image, extracting the jets at the nodes, and determining the similarity between the image graph and the model graph. The search continues by sliding the model graph left to right starting from the upper-left corner of the image (block 48). When a rough position of the face is found (block 50), the nodes are individually allowed to move, introducing elastic graph distortions (block 52). A phase-insensitive similarity function is used in order to locate a good match (block 54). A phase-sensitive similarity function is then used to locate a jet with accuracy because the phase is very sensitive to small jet displacements. The phase-insensitive and the phase-sensitive similarity functions are described below with respect to FIGS. 5-8. Note that although the graphs are shown in FIG. 4 with respect to the original image, the model graph movements and matching are actually performed on the transformed image”);
determining a plurality of expression change critical nodes according to the position information, and using the local facial images corresponding to the expression change critical nodes as the key images in the expression image sequence (See Maurer: Figs. 9-11, Col. 8 Lines 64-67 ~ Col. 9 Lines 1-1, “Tracking error may be detected by determining whether a confidence or similarity value is smaller than a predetermined threshold (block 84 of FIG. 9). The similarity (or confidence) value S may be calculated to indicate how well the two image regions in the two image frames correspond to each other simultaneous with the calculation of the displacement of a node between consecutive image frames. Typically, the confidence value is close to 1, indicating good correspondence. If the confidence value is not close to 1, either the corresponding point in the image has not been found (e.g., because the frame rate was too low compared to the velocity of the moving object), or this image region has changed so drastically from one image frame to the next, that the correspondence is no longer well defined (e.g., for the node tracking the pupil of the eye the eyelid has been closed). Nodes having a confidence value below a certain threshold may be switched off”; and Col. 6 Lines 22-28, “After the facial features are located, the facial features may be tracked over consecutive frames as illustrated in FIG. 9. The tracking technique of the invention achieves robust tracking over long frame sequences by using a tracking correction scheme that detects whether tracking of a feature or node has been lost and reinitializes the tracking process for that node”. Note that change is tracked and initialized if change is significant, and this is mapped to the current limitation).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Bouaziz to have the method according to claim 2, wherein the key images in the expression image sequence are determined by: recognizing position information of a preset facial feature in respective local facial images of the expression image sequence; determining a plurality of expression change critical nodes according to the position information, and using the local facial images corresponding to the expression change critical nodes as the key images in the expression image sequence as taught by Maurer in order to enable efficient recognition of person's head and hand features (See Maurer: Figs.13-14, and Col. 11 Lines 55-60, “The facial sensing of the invention may be applied to the creation and animation of static and dynamic avatars as shown in FIG. 14. The avatar may be based on a generic facial model or based on a person specific facial model. The tracking and facial expression recognition may be used for the incarnation the avatar with the person's features”; and Col. 12 Lines 31-34, “A generic three-dimensional head model for a specific person can be generated using two facial images showing a frontal and a profile view. Facial sensing enables efficiently and robustly generation of the 3-D head model”). Bouaziz teaches a method and system that may generate real-time facial animation by refining the dynamic expression model based on the facial tracking parameters; while Maurer teaches a system and method that may generate and animate an avatar image based on facial sensing to detect the facial feature positions and the critical facial expression changes related to the feature positions. Therefore, it is obvious to one of ordinary skill in the art to modify Bouaziz by Maurer to determine the facial feature positions and the facial expression change critical nodes to more accurate generate the facial expressions for avatar animation. The motivation to modify Bouaziz by Maurer is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 4, Bouaziz and Rosenfeld teach all the features with respect to claim 2 as outlined above. Further, Rosenfeld and Maurer teach that the method according to claim 2, wherein the expression action intervals are respectively associated with a corresponding expression coefficient mapping strategy (See Rosenfeld: Figs. 1-3, and [0024], “In accordance with the present invention, there is provided as illustrated in FIGS. 1-3, a method for controlling and automatically animating lip synchronization and facial expressions of three dimensional animated characters using weighted morph targets and time aligned phonetic transcriptions of recorded text. The method utilizes a set of rules that determine the systems output comprising a stream of morph weight sets when a sequence of timed phonemes is encountered. Other timed data, such as timed emotional state data or emotemes such as "surprise, "disgust, "embarrassment", "timid smile", pitch, amplitued, noise amounts or the like, may be inputted to affect the output stream of morph weight sets”; and [0035], “In FIG. 2, the method for automatically animating lip synchronization and facial expression of three dimensional characters for films, videos, cartoons, and other animation products 10 is shown according to the invention, where box 50 show the step of configuring a set of default correspondence rules between a plurality of visual phoneme groups or other timed input data and a plurality of morph weight sets. Box 52 shows the steps of specifying a plurality of morph weight set transition rules for specifying durational data for the generation of transitionary curves between the plurality of morph weight sets, allowing for the production of a stream of specified morph weight sets to be processed by a computer animation system for integration with other animation, whereby animated lip synchronization and facial expression of animated characters may be automatically produced”. Note that different sets of rules/transition parameters of different subsequences or contexts in the time sequence define how morph weights are generated, and this is mapped to the current limitation of “mapping strategy”);
after determining the expression action interval in which the local facial image to be processed is located (See Maurer: Figs. 13-14, and Col. 12 Lines 54-67 ~ Col. 13 Lines 1-12, “An avatar image may be animated by the following common techniques (see F. I. Parke and K. Waters. Computer Facial Animation. A K Peters, Ltd. Wellesley, Mass., 1996). 1. Key framing and geometric interpolation, where a number of key poses and expressions are defined. Geometric interpolation is then used between the key frames to provide animation. Such a system is frequently referred to as a performance-based (or performance-driven) model. 2. Direct parameterization which directly maps expressions and pose to a set of parameters that are then used to drive the model. 3. Pseudo-muscle models which simulate muscle actions using geometric deformations. 4. Muscle-based models where muscles and skin are modeled using physical models. 5. 2-D and 3-D Morphing which use 2D morphing between images in a video stream to produce 2D animation. A set of landmarks are identified and used to warp between two images of a sequence. Such a technique can be extended to 3D (See, F. F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin. Synthesizing Realistic Facial Expressions from Photographs. In SIGGRAPH 98 Conference Proceedings, pages 75-84. July 1998.). 6. Other approaches such as control points and finite element models”. Note that interval identification via key frames from feature position change to segment into phases where different coefficient mappings apply, and this is mapped to the current limitation of “expression action interval”), the expression coefficient corresponding to the local facial image to be processed is generated through an expression coefficient mapping strategy associated with the expression action interval (See Rosenfeld: Figs. 1-3, and [0034], “In FIG. 1, a flow chart illustrates the preferred steps of the methodology 10 or automatically animating lip synchronization and facial expression of three dimensional animated characters of the present invention. A specific sub-sequence 20 is selected from the TAPT file 12 and is evaluated 22 to determine if any secondary rule criteria for morph weight set target apply. Time aligned emotional transcription file 14 data may be inputted or data from an optional time aligned data file 16 may be used. Also shown is a parallel method 18 which may be configured identical to the primary method described, however, using different timed data rules and different delta sets. Sub-sequence 20 is evaluated 22 to determine if any secondary rule criteria apply. If yes, then a morph weight set is assigned 24 according to the secondary rules, if no, then a morph weight set is assigned 26 according to the default rules. If the sub-string meets any secondary rule criteria for transition specification 28 then a transition start and end time are assigned according to the secondary rules 32, if no, then assign transition start and end times 30 according to default rules. Then an intermediate file of transition keyframes using target weights and transition rules as generated are created 34, and if any keyframe sequences fit post process before interpolation rules they are applied here 36. This data may be output 38 here if desired. If not, then interpolate using any method post processed keyframes to a desired frequency or frame rate 40 and if any morph weight sequences generated fit post processing after interpolation criteria, they are applied 42 at this point. If parallel methods or systems are used to process other timed aligned data, they may be concatenated here 44, and the data output 46”).
Regarding claim 13, Bouaziz and Rosenfeld teach all the features with respect to claim 12 as outlined above. Further, Maurer teaches that the electronic device according to claim 12, wherein the key images in the expression image sequence are determined by:
recognizing position information of a preset facial feature in respective local facial images of the expression image sequence (See Maurer: Figs. 1-8, Col. 3 Lines 61-67 ~ Col. 4 Lines 1-32, “The facial feature may be located using an elastic graph matching shown in FIG. 4. In the elastic graph matching technique, a captured image (block 40) is transformed into Gabor space using a wavelet transformation (block 42) which is described below in more detail with respect to FIG. 5. The transformed image (block 44) is represented by 40 complex values, representing wavelet components, per each pixel of the original image. Next, a rigid copy of a model graph, which is described in more detail below with respect to FIG. 7, is positioned over the transformed image at varying model node positions to locate a position of optimum similarity (block 46). The search for the optimum similarity may be performed by positioning the model graph in the upper left hand corner of the image, extracting the jets at the nodes, and determining the similarity between the image graph and the model graph. The search continues by sliding the model graph left to right starting from the upper-left corner of the image (block 48). When a rough position of the face is found (block 50), the nodes are individually allowed to move, introducing elastic graph distortions (block 52). A phase-insensitive similarity function is used in order to locate a good match (block 54). A phase-sensitive similarity function is then used to locate a jet with accuracy because the phase is very sensitive to small jet displacements. The phase-insensitive and the phase-sensitive similarity functions are described below with respect to FIGS. 5-8. Note that although the graphs are shown in FIG. 4 with respect to the original image, the model graph movements and matching are actually performed on the transformed image”);
determining a plurality of expression change critical nodes according to the position information, and using the local facial images corresponding to the expression change critical nodes as the key images in the expression image sequence (See Maurer: Figs. 9-11, Col. 8 Lines 64-67 ~ Col. 9 Lines 1-1, “Tracking error may be detected by determining whether a confidence or similarity value is smaller than a predetermined threshold (block 84 of FIG. 9). The similarity (or confidence) value S may be calculated to indicate how well the two image regions in the two image frames correspond to each other simultaneous with the calculation of the displacement of a node between consecutive image frames. Typically, the confidence value is close to 1, indicating good correspondence. If the confidence value is not close to 1, either the corresponding point in the image has not been found (e.g., because the frame rate was too low compared to the velocity of the moving object), or this image region has changed so drastically from one image frame to the next, that the correspondence is no longer well defined (e.g., for the node tracking the pupil of the eye the eyelid has been closed). Nodes having a confidence value below a certain threshold may be switched off”; and Col. 6 Lines 22-28, “After the facial features are located, the facial features may be tracked over consecutive frames as illustrated in FIG. 9. The tracking technique of the invention achieves robust tracking over long frame sequences by using a tracking correction scheme that detects whether tracking of a feature or node has been lost and reinitializes the tracking process for that node”. Note that change is tracked and initialized if change is significant, and this is mapped to the current limitation).
Regarding claim 14, Bouaziz and Rosenfeld teach all the features with respect to claim 12 as outlined above. Further, Rosenfeld and Maurer teach that the electronic device according to claim 12, wherein the expression action intervals are respectively associated with a corresponding expression coefficient mapping strategy (See Rosenfeld: Figs. 1-3, and [0024], “In accordance with the present invention, there is provided as illustrated in FIGS. 1-3, a method for controlling and automatically animating lip synchronization and facial expressions of three dimensional animated characters using weighted morph targets and time aligned phonetic transcriptions of recorded text. The method utilizes a set of rules that determine the systems output comprising a stream of morph weight sets when a sequence of timed phonemes is encountered. Other timed data, such as timed emotional state data or emotemes such as "surprise, "disgust, "embarrassment", "timid smile", pitch, amplitued, noise amounts or the like, may be inputted to affect the output stream of morph weight sets”; and [0035], “In FIG. 2, the method for automatically animating lip synchronization and facial expression of three dimensional characters for films, videos, cartoons, and other animation products 10 is shown according to the invention, where box 50 show the step of configuring a set of default correspondence rules between a plurality of visual phoneme groups or other timed input data and a plurality of morph weight sets. Box 52 shows the steps of specifying a plurality of morph weight set transition rules for specifying durational data for the generation of transitionary curves between the plurality of morph weight sets, allowing for the production of a stream of specified morph weight sets to be processed by a computer animation system for integration with other animation, whereby animated lip synchronization and facial expression of animated characters may be automatically produced”. Note that different sets of rules/transition parameters of different subsequences or contexts in the time sequence define how morph weights are generated, and this is mapped to the current limitation of “mapping strategy”);
after determining the expression action interval in which the local facial image to be processed is located, the expression coefficient corresponding to the local facial image to be processed is generated through an expression coefficient mapping strategy associated with the expression action interval (See Maurer: Figs. 13-14, and Col. 12 Lines 54-67 ~ Col. 13 Lines 1-12, “An avatar image may be animated by the following common techniques (see F. I. Parke and K. Waters. Computer Facial Animation. A K Peters, Ltd. Wellesley, Mass., 1996). 1. Key framing and geometric interpolation, where a number of key poses and expressions are defined. Geometric interpolation is then used between the key frames to provide animation. Such a system is frequently referred to as a performance-based (or performance-driven) model. 2. Direct parameterization which directly maps expressions and pose to a set of parameters that are then used to drive the model. 3. Pseudo-muscle models which simulate muscle actions using geometric deformations. 4. Muscle-based models where muscles and skin are modeled using physical models. 5. 2-D and 3-D Morphing which use 2D morphing between images in a video stream to produce 2D animation. A set of landmarks are identified and used to warp between two images of a sequence. Such a technique can be extended to 3D (See, F. F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin. Synthesizing Realistic Facial Expressions from Photographs. In SIGGRAPH 98 Conference Proceedings, pages 75-84. July 1998.). 6. Other approaches such as control points and finite element models”. Note that interval identification via key frames from feature position change to segment into phases where different coefficient mappings apply, and this is mapped to the current limitation of “expression action interval”), the expression coefficient corresponding to the local facial image to be processed is generated through an expression coefficient mapping strategy associated with the expression action interval (See Rosenfeld: Figs. 1-3, and [0034], “In FIG. 1, a flow chart illustrates the preferred steps of the methodology 10 or automatically animating lip synchronization and facial expression of three dimensional animated characters of the present invention. A specific sub-sequence 20 is selected from the TAPT file 12 and is evaluated 22 to determine if any secondary rule criteria for morph weight set target apply. Time aligned emotional transcription file 14 data may be inputted or data from an optional time aligned data file 16 may be used. Also shown is a parallel method 18 which may be configured identical to the primary method described, however, using different timed data rules and different delta sets. Sub-sequence 20 is evaluated 22 to determine if any secondary rule criteria apply. If yes, then a morph weight set is assigned 24 according to the secondary rules, if no, then a morph weight set is assigned 26 according to the default rules. If the sub-string meets any secondary rule criteria for transition specification 28 then a transition start and end time are assigned according to the secondary rules 32, if no, then assign transition start and end times 30 according to default rules. Then an intermediate file of transition keyframes using target weights and transition rules as generated are created 34, and if any keyframe sequences fit post process before interpolation rules they are applied here 36. This data may be output 38 here if desired. If not, then interpolate using any method post processed keyframes to a desired frequency or frame rate 40 and if any morph weight sequences generated fit post processing after interpolation criteria, they are applied 42 at this point. If parallel methods or systems are used to process other timed aligned data, they may be concatenated here 44, and the data output 46”).
Regarding claim 18, Bouaziz and Rosenfeld teach all the features with respect to claim 17 as outlined above. Further, Maurer teaches that the non-transitory computer-readable storage medium according to claim 17, wherein the key images in the expression image sequence are determined by:
recognizing position information of a preset facial feature in respective local facial images of the expression image sequence (See Maurer: Figs. 1-8, Col. 3 Lines 61-67 ~ Col. 4 Lines 1-32, “The facial feature may be located using an elastic graph matching shown in FIG. 4. In the elastic graph matching technique, a captured image (block 40) is transformed into Gabor space using a wavelet transformation (block 42) which is described below in more detail with respect to FIG. 5. The transformed image (block 44) is represented by 40 complex values, representing wavelet components, per each pixel of the original image. Next, a rigid copy of a model graph, which is described in more detail below with respect to FIG. 7, is positioned over the transformed image at varying model node positions to locate a position of optimum similarity (block 46). The search for the optimum similarity may be performed by positioning the model graph in the upper left hand corner of the image, extracting the jets at the nodes, and determining the similarity between the image graph and the model graph. The search continues by sliding the model graph left to right starting from the upper-left corner of the image (block 48). When a rough position of the face is found (block 50), the nodes are individually allowed to move, introducing elastic graph distortions (block 52). A phase-insensitive similarity function is used in order to locate a good match (block 54). A phase-sensitive similarity function is then used to locate a jet with accuracy because the phase is very sensitive to small jet displacements. The phase-insensitive and the phase-sensitive similarity functions are described below with respect to FIGS. 5-8. Note that although the graphs are shown in FIG. 4 with respect to the original image, the model graph movements and matching are actually performed on the transformed image”);
determining a plurality of expression change critical nodes according to the position information, and using the local facial images corresponding to the expression change critical nodes as the key images in the expression image sequence (See Maurer: Figs. 9-11, Col. 8 Lines 64-67 ~ Col. 9 Lines 1-1, “Tracking error may be detected by determining whether a confidence or similarity value is smaller than a predetermined threshold (block 84 of FIG. 9). The similarity (or confidence) value S may be calculated to indicate how well the two image regions in the two image frames correspond to each other simultaneous with the calculation of the displacement of a node between consecutive image frames. Typically, the confidence value is close to 1, indicating good correspondence. If the confidence value is not close to 1, either the corresponding point in the image has not been found (e.g., because the frame rate was too low compared to the velocity of the moving object), or this image region has changed so drastically from one image frame to the next, that the correspondence is no longer well defined (e.g., for the node tracking the pupil of the eye the eyelid has been closed). Nodes having a confidence value below a certain threshold may be switched off”; and Col. 6 Lines 22-28, “After the facial features are located, the facial features may be tracked over consecutive frames as illustrated in FIG. 9. The tracking technique of the invention achieves robust tracking over long frame sequences by using a tracking correction scheme that detects whether tracking of a feature or node has been lost and reinitializes the tracking process for that node”. Note that change is tracked and initialized if change is significant, and this is mapped to the current limitation).
Regarding claim 19, Bouaziz and Rosenfeld teach all the features with respect to claim 17 as outlined above. Further, Rosenfeld and Maurer teach that the non-transitory computer-readable storage medium according to claim 17, wherein the expression action intervals are respectively associated with a corresponding expression coefficient mapping strategy (See Rosenfeld: Figs. 1-3, and [0024], “In accordance with the present invention, there is provided as illustrated in FIGS. 1-3, a method for controlling and automatically animating lip synchronization and facial expressions of three dimensional animated characters using weighted morph targets and time aligned phonetic transcriptions of recorded text. The method utilizes a set of rules that determine the systems output comprising a stream of morph weight sets when a sequence of timed phonemes is encountered. Other timed data, such as timed emotional state data or emotemes such as "surprise, "disgust, "embarrassment", "timid smile", pitch, amplitued, noise amounts or the like, may be inputted to affect the output stream of morph weight sets”; and [0035], “In FIG. 2, the method for automatically animating lip synchronization and facial expression of three dimensional characters for films, videos, cartoons, and other animation products 10 is shown according to the invention, where box 50 show the step of configuring a set of default correspondence rules between a plurality of visual phoneme groups or other timed input data and a plurality of morph weight sets. Box 52 shows the steps of specifying a plurality of morph weight set transition rules for specifying durational data for the generation of transitionary curves between the plurality of morph weight sets, allowing for the production of a stream of specified morph weight sets to be processed by a computer animation system for integration with other animation, whereby animated lip synchronization and facial expression of animated characters may be automatically produced”. Note that different sets of rules/transition parameters of different subsequences or contexts in the time sequence define how morph weights are generated, and this is mapped to the current limitation of “mapping strategy”);
after determining the expression action interval in which the local facial image to be processed is located, the expression coefficient corresponding to the local facial image to be processed is generated through an expression coefficient mapping strategy associated with the expression action interval (See Maurer: Figs. 13-14, and Col. 12 Lines 54-67 ~ Col. 13 Lines 1-12, “An avatar image may be animated by the following common techniques (see F. I. Parke and K. Waters. Computer Facial Animation. A K Peters, Ltd. Wellesley, Mass., 1996). 1. Key framing and geometric interpolation, where a number of key poses and expressions are defined. Geometric interpolation is then used between the key frames to provide animation. Such a system is frequently referred to as a performance-based (or performance-driven) model. 2. Direct parameterization which directly maps expressions and pose to a set of parameters that are then used to drive the model. 3. Pseudo-muscle models which simulate muscle actions using geometric deformations. 4. Muscle-based models where muscles and skin are modeled using physical models. 5. 2-D and 3-D Morphing which use 2D morphing between images in a video stream to produce 2D animation. A set of landmarks are identified and used to warp between two images of a sequence. Such a technique can be extended to 3D (See, F. F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin. Synthesizing Realistic Facial Expressions from Photographs. In SIGGRAPH 98 Conference Proceedings, pages 75-84. July 1998.). 6. Other approaches such as control points and finite element models”. Note that interval identification via key frames from feature position change to segment into phases where different coefficient mappings apply, and this is mapped to the current limitation of “expression action interval”), the expression coefficient corresponding to the local facial image to be processed is generated through an expression coefficient mapping strategy associated with the expression action interval (See Rosenfeld: Figs. 1-3, and [0034], “In FIG. 1, a flow chart illustrates the preferred steps of the methodology 10 or automatically animating lip synchronization and facial expression of three dimensional animated characters of the present invention. A specific sub-sequence 20 is selected from the TAPT file 12 and is evaluated 22 to determine if any secondary rule criteria for morph weight set target apply. Time aligned emotional transcription file 14 data may be inputted or data from an optional time aligned data file 16 may be used. Also shown is a parallel method 18 which may be configured identical to the primary method described, however, using different timed data rules and different delta sets. Sub-sequence 20 is evaluated 22 to determine if any secondary rule criteria apply. If yes, then a morph weight set is assigned 24 according to the secondary rules, if no, then a morph weight set is assigned 26 according to the default rules. If the sub-string meets any secondary rule criteria for transition specification 28 then a transition start and end time are assigned according to the secondary rules 32, if no, then assign transition start and end times 30 according to default rules. Then an intermediate file of transition keyframes using target weights and transition rules as generated are created 34, and if any keyframe sequences fit post process before interpolation rules they are applied here 36. This data may be output 38 here if desired. If not, then interpolate using any method post processed keyframes to a desired frequency or frame rate 40 and if any morph weight sequences generated fit post processing after interpolation criteria, they are applied 42 at this point. If parallel methods or systems are used to process other timed aligned data, they may be concatenated here 44, and the data output 46”).
Claims 6-8, 16, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Bouaziz, etc. (US 20140362091 A1) in view of Rosenfeld (US 20020101422 A1), further in view of Maurer, etc. (US 6580811 B2) and Zhou, etc. (US 20150035825 A1).
Regarding claim 6, Bouaziz and Rosenfeld teach all the features with respect to claim 1 as outlined above. Further, Bouaziz and Maurer teach that the method according claims 1, wherein generating an expression coefficient corresponding to the local facial image to be processed comprises: inputting the local facial image to be processed into an expression coefficient prediction model having been trained, to output the expression coefficient corresponding to the local facial image to be processed through the expression coefficient prediction model (See Bouaziz: Figs. 1-3, and [0068], “FIG. 3 shows a flowchart of an optimization pipeline according to embodiments of the present disclosure. The optimization pipeline may receive input data 302 that may include color image 304 and depth map 306. The input data 302 may be organized in frames. Each frame of input data 302 may be processed using an interleaved optimization that sequentially refines tracking 308 and a model 310. The output of the tracking refinement 308 may comprise tracking parameters 312 including rigid alignment and blendshape weights per frame, which can be used to derive a virtual avatar 314 in real-time. Furthermore, a user-specific dynamic expression model 316 may be adapted during model refinement 310 based on facial characteristics of the observed user according to the input data 302 using an adaptive dynamic expression model 318. It is to be noted that the adaptive dynamic expression model 318 and the user-specific dynamic expression model 316 may correspond to the dynamic expression model 100 as shown in FIG. 1”; and [0069], “The term "real-time" used throughout this disclosure refers to a performance of a computing system or processing device subject to timing constraints, which specify operational deadlines from input or a processing event to an output or a corresponding response. Accordingly, computing or processing systems operating in real-time must guarantee a response according to strict timing conditions, for example, within a range of milliseconds. Preferably, in media systems a real-time response should be delivered without a perceivable delay for the user. For example, a graphical output should be kept at constant frame rates of at least 15 Hz with a latency to the user input of at least 150 milliseconds. Preferably, the frame rates are within a range of 20 Hz to 150 Hz, such as within two of 20, 25, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140 and 150 Hz and, most preferably, at 25 Hz. The latency may be preferably at least 160 milliseconds, preferably within a range of 10 milliseconds to 160 milliseconds, such as within two of 150, 140, 130, 120, 110, 100, 90, 80, 70, 60, 50, 40, 30, 20, and 10 milliseconds, and most preferably of 150 milliseconds. The real-time performance of embodiments of the present disclosure can be achieved by separation of the tracking refinement 308 and model refinement 310. The interactive generation of the virtual avatar 314 can be accomplished using blendshapes and the computed blendshape weights. Concurrently, the user-specific dynamic expression model may be selectively refined in order to meet the timing constraints”);
wherein the expression coefficient prediction model is trained (See Maurer: Fig. 15, and Col. 13 Lines 23-33, “An example of an avatar animation that uses facial feature tracking and classification may be shown with respect to FIG. 15. During the training phase the individual is prompted for a series of predetermined facial expressions (block 120), and sensing is used to track the features (block 122). At predetermined locations, jets and image patches are extracted for the various expressions (block 124). Image patches surrounding facial features are collected along with the jets 126 extracted from these features. These jets are used later to classify or tag facial features 128. This is done by using these jets to generate a personalized bunch graph and by applying the classification method described above”) by:
obtaining a sample sequence of a target expression action and dividing the sample sequence into a plurality of expression action intervals;
for any target sample image in the sample sequence, determining a target expression action interval in which the target sample image is located, and generating an expression coefficient corresponding to the target sample image according to an expression coefficient mapping strategy associated with the target expression action interval; and
training the expression coefficient prediction model using the target sample image with the generated expression coefficient.
However, Bouaziz, modified by Rosenfeld and Maurer, fails to explicitly disclose that obtaining a sample sequence of a target expression action and dividing the sample sequence into a plurality of expression action intervals; for any target sample image in the sample sequence, determining a target expression action interval in which the target sample image is located, and generating an expression coefficient corresponding to the target sample image according to an expression coefficient mapping strategy associated with the target expression action interval; and training the expression coefficient prediction model using the target sample image with the generated expression coefficient.
However, Zhou teaches that obtaining a sample sequence of a target expression action and dividing the sample sequence into a plurality of expression action intervals (See Zhou: Fig. 1-3, and [0030], “Firstly, in the present invention, a group of user images with different poses and expressions are acquired. The group of images is divided into two parts: rigid motions and non-rigid motions. The rigid motions mean that the user keeps neutral expressions and makes 15 head poses with different angles in the meantime. We use an euler angle (yaw, pitch, roll) to represent these angles: yaw is sampled from -90.degree. to 90.degree. with a sampling interval of 30.degree., keeping pitch and roll at 0.degree. in the meantime; pitch is sampled from -30.degree. to 30.degree. with a sampling interval of 15.degree. but removing 0.degree., keeping yaw and roll at 0.degree. in the meantime; roll is sampled from -30.degree. to 30.degree. with a sampling interval of 15.degree. but removing 0.degree., and keeping yaw and pitch at 0.degree. in the meantime. Noticing that we do not require that the angles of user's poses and the required angle configuration are completely exact, where probable estimation is sufficient”; and [0031], “The non-rigid motions include 15 different expressions under 3 yaw angles. These expressions are relatively large expressions, which differ greatly among different identities. These expressions are: mouth stretch, smile, brow raise, disgust, squeeze left eye, squeeze right eye, anger, jaw left, jaw right, grin, chin raise, lip pucker, lip funnel, cheek blowing and eyes closed”);
for any target sample image in the sample sequence, determining a target expression action interval in which the target sample image is located, and generating an expression coefficient corresponding to the target sample image according to an expression coefficient mapping strategy associated with the target expression action interval (See Zhou: Figs. 1-3, and [0031], “The non-rigid motions include 15 different expressions under 3 yaw angles. These expressions are relatively large expressions, which differ greatly among different identities. These expressions are: mouth stretch, smile, brow raise, disgust, squeeze left eye, squeeze right eye, anger, jaw left, jaw right, grin, chin raise, lip pucker, lip funnel, cheek blowing and eyes closed”; [0036], “The user expression blendshape model includes a neutral expression shape B.sub.0 of the user and 46 FACS expression shapes {B.sub.1, B.sub.2, . . . , B.sub.46}. These expression shapes constitute an expression linear space of the user, any expression B of the user may be obtained by a linear interpolation of basic expressions in the blendshape model”; and [0037], “Where, B.sub.0 is the neutral expression shape of the user, Bi is a basic expression shape in the user expression blendshape model, .alpha..sub.i is a coefficient of the basic expression, and B is an expression face shape obtained by interpolation”. Note that the expression coefficients are computed per image using model fitted into the specific expression group with linear interpolation, and this is mapped to “the mapping strategy”); and
training the expression coefficient prediction model using the target sample image with the generated expression coefficient (See Zhou: Figs. 1-3, and [0034], “Data preprocessing: generating a user expression blendshape model and calibrating a camera internal parameter by adopting the images with labeled 2D face feature points, and thereby obtaining 3D feature points of the images; training, by adopting the 3D feature points and the 2D images acquired in step 1, to obtain a regressor that maps 2D images to the 3D feature points”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Bouaziz to have obtaining a sample sequence of a target expression action and dividing the sample sequence into a plurality of expression action intervals; for any target sample image in the sample sequence, determining a target expression action interval in which the target sample image is located, and generating an expression coefficient corresponding to the target sample image according to an expression coefficient mapping strategy associated with the target expression action interval; and training the expression coefficient prediction model using the target sample image with the generated expression coefficient as taught by Zhou in order to enable providing the single video camera to accurately process different wide-angle facial rotations in a convenient manner (See Zhou: Figs.1-3, and [0022], “The beneficial effects of the present invention are: the present invention can be easily applied, without the need for expensive physical equipment at the facial markers or the projected structured light and etc., the user can accomplish the capture and parameterization of head poses and facial expressions, and map the parameterization result into the virtual avatar to drive face animation of the animation character simply by one-time data acquisition and preprocessing via a single video camera on a common desk computer, which facilitates the use for an ordinary user. In contrast to existing methods, the present invention may effectively process head's fast movements, large rotations and exaggerated facial expressions in videos”). Bouaziz teaches a method and system that may generate real-time facial animation by refining the dynamic expression model based on the facial tracking parameters; while Zhou teaches a system and method that may track three-dimensional (3D) positions of facial feature points in real time by utilizing a single video camera and train the facial expression coefficient model with samples obtained by dividing the sample sequence into a plurality of expression action intervals. Therefore, it is obvious to one of ordinary skill in the art to modify Bouaziz by Zhou to train the expression coefficient prediction mode with samples obtained by dividing the sample sequence into a plurality of expression action intervals. The motivation to modify Bouaziz by Zhou is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 7, Bouaziz, Rosenfeld, Maurer, and Zhou teach all the features with respect to claim 6 as outlined above. Further, Maurer and Zhou teach that the method according to claim 6, wherein dividing the sample sequence into a plurality of expression action intervals comprises:
recognizing a plurality of key images from the sample images comprised in the sample sequence, wherein the time sequence relationship of the plurality of key images in the sample sequence characterizes an action process of the target expression action (See Maurer: Figs. 4-9, and Col. 3 Lines 61-67 ~ Col. 4 Lines 1-22, “The facial feature may be located using an elastic graph matching shown in FIG. 4. In the elastic graph matching technique, a captured image (block 40) is transformed into Gabor space using a wavelet transformation (block 42) which is described below in more detail with respect to FIG. 5. The transformed image (block 44) is represented by 40 complex values, representing wavelet components, per each pixel of the original image. Next, a rigid copy of a model graph, which is described in more detail below with respect to FIG. 7, is positioned over the transformed image at varying model node positions to locate a position of optimum similarity (block 46). The search for the optimum similarity may be performed by positioning the model graph in the upper left hand corner of the image, extracting the jets at the nodes, and determining the similarity between the image graph and the model graph. The search continues by sliding the model graph left to right starting from the upper-left corner of the image (block 48). When a rough position of the face is found (block 50), the nodes are individually allowed to move, introducing elastic graph distortions (block 52). A phase-insensitive similarity function is used in order to locate a good match (block 54). A phase-sensitive similarity function is then used to locate a jet with accuracy because the phase is very sensitive to small jet displacements. The phase-insensitive and the phase-sensitive similarity functions are described below with respect to FIGS. 5-8. Note that although the graphs are shown in FIG. 4 with respect to the original image, the model graph movements and matching are actually performed on the transformed image”; and Figs. 4-9, and Col. 13 Lines 12-22, “For these techniques, facial sensing enhances the animation process by providing automatic extraction and characterization of facial features. Extracted features may be used to interpolate expressions in the case of key framing and interpolation models, or to select parameters for direct parameterized models or pseudo-muscles or muscles models. In the case of 2-D and 3-D morphing, facial sensing may be used to automatically select features on a face providing the appropriate information to perform the geometric transformation”);
dividing the sample sequence into a plurality of expression action intervals using the plurality of key images (See Zhou: Figs. 1-3, and [0030], “Firstly, in the present invention, a group of user images with different poses and expressions are acquired. The group of images is divided into two parts: rigid motions and non-rigid motions. The rigid motions mean that the user keeps neutral expressions and makes 15 head poses with different angles in the meantime. We use an euler angle (yaw, pitch, roll) to represent these angles: yaw is sampled from -90.degree. to 90.degree. with a sampling interval of 30.degree., keeping pitch and roll at 0.degree. in the meantime; pitch is sampled from -30.degree. to 30.degree. with a sampling interval of 15.degree. but removing 0.degree., keeping yaw and roll at 0.degree. in the meantime; roll is sampled from -30.degree. to 30.degree. with a sampling interval of 15.degree. but removing 0.degree., and keeping yaw and pitch at 0.degree. in the meantime. Noticing that we do not require that the angles of user's poses and the required angle configuration are completely exact, where probable estimation is sufficient”; and [0070], “After obtaining the 3D feature point locations of the current frame in the previous step, the present invention performs parameterization for the face motions in the current frame by adopting them. The face motions are mainly divided into two parts: rigid head poses represented by the transformation matrix M, and face non-rigid expressions represented by the expression blendshape coefficient a. These two parameters may be obtained by optimizing the following matching energy”).
Regarding claim 8, Bouaziz, Rosenfeld, Maurer, and Zhou teach all the features with respect to claim 7 as outlined above. Further, Maurer teaches that the method according to claim 7, wherein recognizing a plurality of key images comprises:
recognizing position information of a preset facial feature in respective sample images of the sample sequence (See Maurer: Figs. 1-8, Col. 3 Lines 61-67 ~ Col. 4 Lines 1-32, “The facial feature may be located using an elastic graph matching shown in FIG. 4. In the elastic graph matching technique, a captured image (block 40) is transformed into Gabor space using a wavelet transformation (block 42) which is described below in more detail with respect to FIG. 5. The transformed image (block 44) is represented by 40 complex values, representing wavelet components, per each pixel of the original image. Next, a rigid copy of a model graph, which is described in more detail below with respect to FIG. 7, is positioned over the transformed image at varying model node positions to locate a position of optimum similarity (block 46). The search for the optimum similarity may be performed by positioning the model graph in the upper left hand corner of the image, extracting the jets at the nodes, and determining the similarity between the image graph and the model graph. The search continues by sliding the model graph left to right starting from the upper-left corner of the image (block 48). When a rough position of the face is found (block 50), the nodes are individually allowed to move, introducing elastic graph distortions (block 52). A phase-insensitive similarity function is used in order to locate a good match (block 54). A phase-sensitive similarity function is then used to locate a jet with accuracy because the phase is very sensitive to small jet displacements. The phase-insensitive and the phase-sensitive similarity functions are described below with respect to FIGS. 5-8. Note that although the graphs are shown in FIG. 4 with respect to the original image, the model graph movements and matching are actually performed on the transformed image”); and
determining a plurality of expression change critical nodes according to the position information, and using sample images corresponding to the expression change critical nodes as the key images in the sample sequence (See Maurer: Figs. 9-11, Col. 8 Lines 64-67 ~ Col. 9 Lines 1-1, “Tracking error may be detected by determining whether a confidence or similarity value is smaller than a predetermined threshold (block 84 of FIG. 9). The similarity (or confidence) value S may be calculated to indicate how well the two image regions in the two image frames correspond to each other simultaneous with the calculation of the displacement of a node between consecutive image frames. Typically, the confidence value is close to 1, indicating good correspondence. If the confidence value is not close to 1, either the corresponding point in the image has not been found (e.g., because the frame rate was too low compared to the velocity of the moving object), or this image region has changed so drastically from one image frame to the next, that the correspondence is no longer well defined (e.g., for the node tracking the pupil of the eye the eyelid has been closed). Nodes having a confidence value below a certain threshold may be switched off”; and Col. 6 Lines 22-28, “After the facial features are located, the facial features may be tracked over consecutive frames as illustrated in FIG. 9. The tracking technique of the invention achieves robust tracking over long frame sequences by using a tracking correction scheme that detects whether tracking of a feature or node has been lost and reinitializes the tracking process for that node”. Note that change is tracked and initialized if change is significant, and this is mapped to the current limitation).
Regarding claim 16, Bouaziz and Rosenfeld teach all the features with respect to claim 10 as outlined above. Further, Bouaziz, Maurer, and Zhou teach that the electronic device according to claim 10, wherein generating an expression coefficient corresponding to the local facial image to be processed comprises:
inputting the local facial image to be processed into an expression coefficient prediction model having been trained, to output the expression coefficient corresponding to the local facial image to be processed through the expression coefficient prediction model (See Bouaziz: Figs. 1-3, and [0068], “FIG. 3 shows a flowchart of an optimization pipeline according to embodiments of the present disclosure. The optimization pipeline may receive input data 302 that may include color image 304 and depth map 306. The input data 302 may be organized in frames. Each frame of input data 302 may be processed using an interleaved optimization that sequentially refines tracking 308 and a model 310. The output of the tracking refinement 308 may comprise tracking parameters 312 including rigid alignment and blendshape weights per frame, which can be used to derive a virtual avatar 314 in real-time. Furthermore, a user-specific dynamic expression model 316 may be adapted during model refinement 310 based on facial characteristics of the observed user according to the input data 302 using an adaptive dynamic expression model 318. It is to be noted that the adaptive dynamic expression model 318 and the user-specific dynamic expression model 316 may correspond to the dynamic expression model 100 as shown in FIG. 1”; and [0069], “The term "real-time" used throughout this disclosure refers to a performance of a computing system or processing device subject to timing constraints, which specify operational deadlines from input or a processing event to an output or a corresponding response. Accordingly, computing or processing systems operating in real-time must guarantee a response according to strict timing conditions, for example, within a range of milliseconds. Preferably, in media systems a real-time response should be delivered without a perceivable delay for the user. For example, a graphical output should be kept at constant frame rates of at least 15 Hz with a latency to the user input of at least 150 milliseconds. Preferably, the frame rates are within a range of 20 Hz to 150 Hz, such as within two of 20, 25, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140 and 150 Hz and, most preferably, at 25 Hz. The latency may be preferably at least 160 milliseconds, preferably within a range of 10 milliseconds to 160 milliseconds, such as within two of 150, 140, 130, 120, 110, 100, 90, 80, 70, 60, 50, 40, 30, 20, and 10 milliseconds, and most preferably of 150 milliseconds. The real-time performance of embodiments of the present disclosure can be achieved by separation of the tracking refinement 308 and model refinement 310. The interactive generation of the virtual avatar 314 can be accomplished using blendshapes and the computed blendshape weights. Concurrently, the user-specific dynamic expression model may be selectively refined in order to meet the timing constraints”);
wherein the expression coefficient prediction model is trained (See Maurer: Fig. 15, and Col. 13 Lines 23-33, “An example of an avatar animation that uses facial feature tracking and classification may be shown with respect to FIG. 15. During the training phase the individual is prompted for a series of predetermined facial expressions (block 120), and sensing is used to track the features (block 122). At predetermined locations, jets and image patches are extracted for the various expressions (block 124). Image patches surrounding facial features are collected along with the jets 126 extracted from these features. These jets are used later to classify or tag facial features 128. This is done by using these jets to generate a personalized bunch graph and by applying the classification method described above”) by:
obtaining a sample sequence of a target expression action and dividing the sample sequence into a plurality of expression action intervals (See Zhou: Fig. 1-3, and [0030], “Firstly, in the present invention, a group of user images with different poses and expressions are acquired. The group of images is divided into two parts: rigid motions and non-rigid motions. The rigid motions mean that the user keeps neutral expressions and makes 15 head poses with different angles in the meantime. We use an euler angle (yaw, pitch, roll) to represent these angles: yaw is sampled from -90.degree. to 90.degree. with a sampling interval of 30.degree., keeping pitch and roll at 0.degree. in the meantime; pitch is sampled from -30.degree. to 30.degree. with a sampling interval of 15.degree. but removing 0.degree., keeping yaw and roll at 0.degree. in the meantime; roll is sampled from -30.degree. to 30.degree. with a sampling interval of 15.degree. but removing 0.degree., and keeping yaw and pitch at 0.degree. in the meantime. Noticing that we do not require that the angles of user's poses and the required angle configuration are completely exact, where probable estimation is sufficient”; and [0031], “The non-rigid motions include 15 different expressions under 3 yaw angles. These expressions are relatively large expressions, which differ greatly among different identities. These expressions are: mouth stretch, smile, brow raise, disgust, squeeze left eye, squeeze right eye, anger, jaw left, jaw right, grin, chin raise, lip pucker, lip funnel, cheek blowing and eyes closed”);
for any target sample image in the sample sequence, determining a target expression action interval in which the target sample image is located, and generating an expression coefficient corresponding to the target sample image according to an expression coefficient mapping strategy associated with the target expression action interval (See Zhou: Figs. 1-3, and [0031], “The non-rigid motions include 15 different expressions under 3 yaw angles. These expressions are relatively large expressions, which differ greatly among different identities. These expressions are: mouth stretch, smile, brow raise, disgust, squeeze left eye, squeeze right eye, anger, jaw left, jaw right, grin, chin raise, lip pucker, lip funnel, cheek blowing and eyes closed”; [0036], “The user expression blendshape model includes a neutral expression shape B.sub.0 of the user and 46 FACS expression shapes {B.sub.1, B.sub.2, . . . , B.sub.46}. These expression shapes constitute an expression linear space of the user, any expression B of the user may be obtained by a linear interpolation of basic expressions in the blendshape model”; and [0037], “Where, B.sub.0 is the neutral expression shape of the user, Bi is a basic expression shape in the user expression blendshape model, .alpha..sub.i is a coefficient of the basic expression, and B is an expression face shape obtained by interpolation”. Note that the expression coefficients are computed per image using model fitted into the specific expression group with linear interpolation, and this is mapped to “the mapping strategy”); and
training the expression coefficient prediction model using the target sample image with the generated expression coefficient (See Zhou: Figs. 1-3, and [0034], “Data preprocessing: generating a user expression blendshape model and calibrating a camera internal parameter by adopting the images with labeled 2D face feature points, and thereby obtaining 3D feature points of the images; training, by adopting the 3D feature points and the 2D images acquired in step 1, to obtain a regressor that maps 2D images to the 3D feature points”).
Regarding claim 21, Bouaziz and Rosenfeld teach all the features with respect to claim 11 as outlined above. Further, Bouaziz, Maurer, and Zhou teach that the non-transitory computer-readable storage medium according to claim 11, wherein generating an expression coefficient corresponding to the local facial image to be processed comprises:
inputting the local facial image to be processed into an expression coefficient prediction model having been trained, to output the expression coefficient corresponding to the local facial image to be processed through the expression coefficient prediction model (See Bouaziz: Figs. 1-3, and [0068], “FIG. 3 shows a flowchart of an optimization pipeline according to embodiments of the present disclosure. The optimization pipeline may receive input data 302 that may include color image 304 and depth map 306. The input data 302 may be organized in frames. Each frame of input data 302 may be processed using an interleaved optimization that sequentially refines tracking 308 and a model 310. The output of the tracking refinement 308 may comprise tracking parameters 312 including rigid alignment and blendshape weights per frame, which can be used to derive a virtual avatar 314 in real-time. Furthermore, a user-specific dynamic expression model 316 may be adapted during model refinement 310 based on facial characteristics of the observed user according to the input data 302 using an adaptive dynamic expression model 318. It is to be noted that the adaptive dynamic expression model 318 and the user-specific dynamic expression model 316 may correspond to the dynamic expression model 100 as shown in FIG. 1”; and [0069], “The term "real-time" used throughout this disclosure refers to a performance of a computing system or processing device subject to timing constraints, which specify operational deadlines from input or a processing event to an output or a corresponding response. Accordingly, computing or processing systems operating in real-time must guarantee a response according to strict timing conditions, for example, within a range of milliseconds. Preferably, in media systems a real-time response should be delivered without a perceivable delay for the user. For example, a graphical output should be kept at constant frame rates of at least 15 Hz with a latency to the user input of at least 150 milliseconds. Preferably, the frame rates are within a range of 20 Hz to 150 Hz, such as within two of 20, 25, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140 and 150 Hz and, most preferably, at 25 Hz. The latency may be preferably at least 160 milliseconds, preferably within a range of 10 milliseconds to 160 milliseconds, such as within two of 150, 140, 130, 120, 110, 100, 90, 80, 70, 60, 50, 40, 30, 20, and 10 milliseconds, and most preferably of 150 milliseconds. The real-time performance of embodiments of the present disclosure can be achieved by separation of the tracking refinement 308 and model refinement 310. The interactive generation of the virtual avatar 314 can be accomplished using blendshapes and the computed blendshape weights. Concurrently, the user-specific dynamic expression model may be selectively refined in order to meet the timing constraints”);
wherein the expression coefficient prediction model is trained (See Maurer: Fig. 15, and Col. 13 Lines 23-33, “An example of an avatar animation that uses facial feature tracking and classification may be shown with respect to FIG. 15. During the training phase the individual is prompted for a series of predetermined facial expressions (block 120), and sensing is used to track the features (block 122). At predetermined locations, jets and image patches are extracted for the various expressions (block 124). Image patches surrounding facial features are collected along with the jets 126 extracted from these features. These jets are used later to classify or tag facial features 128. This is done by using these jets to generate a personalized bunch graph and by applying the classification method described above”) by:
obtaining a sample sequence of a target expression action and dividing the sample sequence into a plurality of expression action intervals (See Zhou: Fig. 1-3, and [0030], “Firstly, in the present invention, a group of user images with different poses and expressions are acquired. The group of images is divided into two parts: rigid motions and non-rigid motions. The rigid motions mean that the user keeps neutral expressions and makes 15 head poses with different angles in the meantime. We use an euler angle (yaw, pitch, roll) to represent these angles: yaw is sampled from -90.degree. to 90.degree. with a sampling interval of 30.degree., keeping pitch and roll at 0.degree. in the meantime; pitch is sampled from -30.degree. to 30.degree. with a sampling interval of 15.degree. but removing 0.degree., keeping yaw and roll at 0.degree. in the meantime; roll is sampled from -30.degree. to 30.degree. with a sampling interval of 15.degree. but removing 0.degree., and keeping yaw and pitch at 0.degree. in the meantime. Noticing that we do not require that the angles of user's poses and the required angle configuration are completely exact, where probable estimation is sufficient”; and [0031], “The non-rigid motions include 15 different expressions under 3 yaw angles. These expressions are relatively large expressions, which differ greatly among different identities. These expressions are: mouth stretch, smile, brow raise, disgust, squeeze left eye, squeeze right eye, anger, jaw left, jaw right, grin, chin raise, lip pucker, lip funnel, cheek blowing and eyes closed”);
for any target sample image in the sample sequence, determining a target expression action interval in which the target sample image is located, and generating an expression coefficient corresponding to the target sample image according to an expression coefficient mapping strategy associated with the target expression action interval (See Zhou: Figs. 1-3, and [0031], “The non-rigid motions include 15 different expressions under 3 yaw angles. These expressions are relatively large expressions, which differ greatly among different identities. These expressions are: mouth stretch, smile, brow raise, disgust, squeeze left eye, squeeze right eye, anger, jaw left, jaw right, grin, chin raise, lip pucker, lip funnel, cheek blowing and eyes closed”; [0036], “The user expression blendshape model includes a neutral expression shape B.sub.0 of the user and 46 FACS expression shapes {B.sub.1, B.sub.2, . . . , B.sub.46}. These expression shapes constitute an expression linear space of the user, any expression B of the user may be obtained by a linear interpolation of basic expressions in the blendshape model”; and [0037], “Where, B.sub.0 is the neutral expression shape of the user, Bi is a basic expression shape in the user expression blendshape model, .alpha..sub.i is a coefficient of the basic expression, and B is an expression face shape obtained by interpolation”. Note that the expression coefficients are computed per image using model fitted into the specific expression group with linear interpolation, and this is mapped to “the mapping strategy”); and
training the expression coefficient prediction model using the target sample image with the generated expression coefficient (See Zhou: Figs. 1-3, and [0034], “Data preprocessing: generating a user expression blendshape model and calibrating a camera internal parameter by adopting the images with labeled 2D face feature points, and thereby obtaining 3D feature points of the images; training, by adopting the 3D feature points and the 2D images acquired in step 1, to obtain a regressor that maps 2D images to the 3D feature points”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights