DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 5-7, 9-12, 14, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Saragih et al. (US Patent 10636192), hereinafter Saragih, in view of Olszewski et al. (NPL: High-Fidelity Facial and Speech Animation for VR HMDs), hereinafter Olszewski.
Regarding claim 1, Saragih discloses a non-transitory computer-readable data storage medium storing program code executable by a processor to perform processing (Column 14, lines 57-67: apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability) comprising: selecting a cohort corresponding to a wearer of a head-mountable display (HMD) from a plurality of candidate cohorts that each correspond to a different facial type (Fig. 3; Fig. 5; Column 5, lines 41-57: facial tracking system 160 generates reconstructions of portions of a face of a user wearing the HMD 105, as further described below in conjunction with FIGS. 2-5. In an embodiment, the facial tracking system 160 includes one or more facial sensors and a controller, as further described below in conjunction with FIG. 2. In some embodiments, the facial tracking system 160 also includes one or more illumination sources configured to illuminate portions of the user's face within fields of view of the one or more facial sensors. Based on data from the facial sensors, the controller generates a trained model that maps positions of points identified within images captured by the facial sensors to a set of animation parameters that determines a facial animation model projecting movement of the portions of the user's face into a three dimensional model of a face presented via a virtual reality environment of the HMD 105, as further described below in conjunction with FIG. 5; Column 11, lines 7-57: one or more facial sensors 210 communicate the captured images to the controller 220, which identifies 520 points corresponding to features of the portion of the user's face within each of the captured images. Features of the portion of the user's face corresponding to boundaries of parts of the user's face. For example, identified points correspond to positions along boundaries of the user's lips or positions along the user's chin. As an example, the identified points correspond to corners of the user's lips and various locations along boundaries of the user's lips. To identify 520 the points in images of a portion of the user's face, the controller 220 obtains a model trained on a set of images of various facial expressions made by various users that include the points on portions of the users' faces identified by the users. For example, a user makes specific facial expressions, and images of portions of the user's face are captured by facial sensors 210 when the user makes each specific facial expression. The user is subsequently prompted to identify various points in the captured images corresponding to each specific facial expression. Based on the points identified by the users when various specific facial expressions were made, a machine learned model is trained to identify 520 the points from the captured images. In various embodiments, different machine learned models are trained to identify 520 points on different portions of the user's face. The controller 220 applies the trained model to images captured 510 by different facial sensors 210 to identify 520 points in images of portions of the user's face captured 510 by different facial sensors 210. For example, the controller 220 identifies 520 points of portions of the user's face included in each image captured 510 by one or more facial sensors 210… controller 220 maps 530 the identified points to a three-dimensional model of a face for generating a graphical representation of the portions of the user's face corresponding to the captured images. In some embodiments, the three-dimensional model of the face is selected from a stored library of three-dimensional models based on characteristics of the user. For example, the user specifies one or more parameters and the controller 220 selects a three-dimensional model of a face from the library that has at least a threshold amount of parameters matching the parameters specified by the user; the controller 220 identifies points on the selected three-dimensional model corresponding to the points identified 520 from the captured images to map 530 the identified points to the selected three-dimensional model. Alternatively, the controller 220 receives an image of the user's face and selects a three-dimensional model of a face having at least a threshold similarity to the image of the user's face and maps 530 the identified points to the selected three-dimensional model); capturing a set of facial images of the wearer using one or multiple cameras of the HMD (Column 10, line 63-Column 11, line 6: One or more facial sensors 210 coupled to a HMD 105 worn by a user capture 510 images of portions of the user's face. In various embodiments, multiple facial sensors 210 are coupled to the HMD 105, with each facial sensor 210 configured to capture 510 images of a different portion of the user's face. The different portions of the user's face partially overlap in some embodiments, while in other embodiments the different portions of the user's face do not overlap. A facial sensor 210 captures 510 a series of images of a portion of the user's face within a field of view of the facial sensor 210); retargeting the predicted blendshape weights for the facial expression of the wearer onto an avatar corresponding to the wearer to render the avatar with the facial expression (Column 5, lines 41-57: facial tracking system 160 generates reconstructions of portions of a face of a user wearing the HMD 105, as further described below in conjunction with FIGS. 2-5. In an embodiment, the facial tracking system 160 includes one or more facial sensors and a controller, as further described below in conjunction with FIG. 2. In some embodiments, the facial tracking system 160 also includes one or more illumination sources configured to illuminate portions of the user's face within fields of view of the one or more facial sensors. Based on data from the facial sensors, the controller generates a trained model that maps positions of points identified within images captured by the facial sensors to a set of animation parameters that determines a facial animation model projecting movement of the portions of the user's face into a three dimensional model of a face presented via a virtual reality environment of the HMD 105, as further described below in conjunction with FIG. 5; Column 11, line 58-Column 12, line 20: a facial animation model of the portions of the user's face of which the facial sensors 210 captured 510 images. In various embodiments, the facial animation model is an expression parameter comprising a parametric representation of human faces, where different parameters correspond to different expressions of the portion of the user's face included in a captured image. For example, the parametric representation of human faces is a blendshape model that models facial expressions of the user as a linear combination of blendshapes that each correspond to an expression of the portion of the user's face included in a captured image. In various embodiments, the facial animation model calculates a combination of coefficients corresponding to different human facial expressions that determine a weight of each human facial expression in a combination. When the facial animation model is a blendshape model, the facial animation model calculates a vector of blendshape coefficients that determine the weight of each expression mesh in the linear combination. Hence, a vector of blendshape coefficients is determined for each image captured by a facial sensor 210. The blendshape coefficients may be extracted based on positions of the identified points within various captured images, so the positions of the identified points within captured images determine weights corresponding to various facial expressions in the calculated vector); and displaying the rendered avatar (Fig. 6; Column 13, lines 25-40: FIG. 6 is a conceptual diagram of generating a graphical representation of a user's face while the user wears a head mounted display (HMD) 105. In the example of FIG. 6, two facial sensors 210A, 210B (also referred to individually and collectively using reference number 210). Each facial sensor 210A, 210B is positioned to capture images of a portion of the user's face while the user wears the HMD 105. In the example of FIG. 6, the facial sensors 210A, 210B each capture images 600A, 600B of a corresponding portion of the user's face that is below the HMD 105 when the user wears the HMD 105. For purposes of illustration, FIG. 6 shows an example where the facial sensor 210A captures images 600A of a user's mouth from a particular angle relative to the user's mouth, while the facial sensor 210B captures images 600B of the user's mouth from a different angle relative to the user's mouth; Column 14, lines 12-25: the controller 220 determines transforms 615 mapping the two-dimensional images 600A, 600B to three dimensions and one or more differential transformations accounting for movement of facial sensors 210A, 210B in one or more directions, as further described above in conjunction with FIG. 5. From the facial animation model 610 and the transforms 615, the controller generates a rendering model 620 that is applied to a three-dimensional model of a face to generate a graphical representation of the portions of the user's face included in the captured images 600A, 600B that is modified by the rendering model 620 to replicate facial expressions captured by the facial sensors 210A, 210A while the user wears the HMD 205).
Saragih does not explicitly disclose applying a machine learning model for the selected cohort to the captured set of facial images to predict blendshape weights for a facial expression of the wearer exhibited within the captured set of images, each candidate cohort having a differently trained machine learning model.
However, Olszewski teaches facial animation using HMDs (Abstract), further comprising applying a machine learning model for the selected cohort to the captured set of facial images to predict blendshape weights for a facial expression of the wearer exhibited within the captured set of images, each candidate cohort having a differently trained machine learning model (Fig. 17; Section 5: goal is to recover detailed 3D facial expressions from video frames. In this paper, we address this problem by representing a face as a set of facial blendshape meshes. Then, our algorithm determines the blendshape weights that best correspond to each frame. More concretely, let us assume we are given a generic blendshape model as a set of meshes b = {b0,b1,··· ,bN}. Our target expression of frame It at time t can be formulated as:…where b0 is the neutral face expression, and wt ∈ [0,1]N is a corresponding blendshape weight vector. Then, our goal is to determine the value for w that best corresponds to a given image or video frame of a human face. In this paper, we formulate this as learning a mapping function ψ, which predicts a blendshape weight vector of image It… the neutral network is a classification function that determines whether the expression in the frame is neutral or not. The output of this network is used for dampening the estimation toward the neutral expression. The final output of our model, ψ(·) combines two network outputs to predict the final blendshape weights; Section 2: training data are collected using a separate RGB-D sensor and linear blendshape models obtained from the facial animation software Faceshift [2014]. As in this work, they also adopt a deep learning framework for regression, but only poor results could be demonstrated even with user-specific training data. Instead, our system generates highly compelling speech animation by mapping the video input from cameras directed at the regions of interest directly to the appropriate facial expression controls. Furthermore, we remove the need for user-specific training data by training our system on users of varying appearance). Olszewski teaches that this will allow for production of realistic facial expressions and animations (Section 1). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Saragih with the features of above as taught by Olszewski so as to allow for production of realistic facial expressions and animations as presented by Olszewski.
Regarding claim 2, Saragih, in view of Olszewski teaches the non-transitory computer-readable data storage medium of claim 1, Olszewski discloses wherein the differently trained machine learning model for each candidate cohort is trained on facial training images of the different facial type to which the candidate cohort corresponds (Fig. 17; Section 5: goal is to recover detailed 3D facial expressions from video frames. In this paper, we address this problem by representing a face as a set of facial blendshape meshes. Then, our algorithm determines the blendshape weights that best correspond to each frame. More concretely, let us assume we are given a generic blendshape model as a set of meshes b = {b0,b1,··· ,bN}. Our target expression of frame It at time t can be formulated as:…where b0 is the neutral face expression, and wt ∈ [0,1]N is a corresponding blendshape weight vector. Then, our goal is to determine the value for w that best corresponds to a given image or video frame of a human face. In this paper, we formulate this as learning a mapping function ψ, which predicts a blendshape weight vector of image It… the neutral network is a classification function that determines whether the expression in the frame is neutral or not. The output of this network is used for dampening the estimation toward the neutral expression. The final output of our model, ψ(·) combines two network outputs to predict the final blendshape weights; Section 2: training data are collected using a separate RGB-D sensor and linear blendshape models obtained from the facial animation software Faceshift [2014]. As in this work, they also adopt a deep learning framework for regression, but only poor results could be demonstrated even with user-specific training data. Instead, our system generates highly compelling speech animation by mapping the video input from cameras directed at the regions of interest directly to the appropriate facial expression controls. Furthermore, we remove the need for user-specific training data by training our system on users of varying appearance).
Regarding claim 5, Saragih, in view of Olszewski teaches the non-transitory computer-readable data storage medium of claim 1, Saragih discloses wherein selecting the cohort corresponding to the wearer from the candidate cohorts comprises: receiving wearer selection of the different facial type of the candidate cohort corresponding to a facial type of the wearer (Column 11, lines 7-38: one or more facial sensors 210 communicate the captured images to the controller 220, which identifies 520 points corresponding to features of the portion of the user's face within each of the captured images. Features of the portion of the user's face corresponding to boundaries of parts of the user's face. For example, identified points correspond to positions along boundaries of the user's lips or positions along the user's chin. As an example, the identified points correspond to corners of the user's lips and various locations along boundaries of the user's lips. To identify 520 the points in images of a portion of the user's face, the controller 220 obtains a model trained on a set of images of various facial expressions made by various users that include the points on portions of the users' faces identified by the users. For example, a user makes specific facial expressions, and images of portions of the user's face are captured by facial sensors 210 when the user makes each specific facial expression. The user is subsequently prompted to identify various points in the captured images corresponding to each specific facial expression. Based on the points identified by the users when various specific facial expressions were made, a machine learned model is trained to identify 520 the points from the captured images. In various embodiments, different machine learned models are trained to identify 520 points on different portions of the user's face. The controller 220 applies the trained model to images captured 510 by different facial sensors 210 to identify 520 points in images of portions of the user's face captured 510 by different facial sensors 210. For example, the controller 220 identifies 520 points of portions of the user's face included in each image captured 510 by one or more facial sensors 210).
Regarding claim 6, Saragih, in view of Olszewski teaches the non-transitory computer-readable data storage medium of claim 1, Olszewski discloses wherein selecting the cohort corresponding to the wearer from the candidate cohorts comprises: applying a classifier machine learning model to the captured set of facial images of the wearer to identify the different facial type of the candidate cohort to which a facial type of the wearer corresponds (Section 5: In order to enable high fidelity facial speech animation, our CNN based method builds upon two main ideas: (1) exploiting a sequence of frames to capture temporal signals, and (2) explicitly dampening an estimation toward the neutral face when necessary because humans are sensitive to neutral expressions. Our model thus contains two sub-networks. The expression network is a regression function that estimates facial expression weights from a sequence of frames. On the other hand, the neutral network is a classification function that determines whether the expression in the frame is neutral or not. The output of this network is used for dampening the estimation toward the neutral expression. The final output of our model, ψ(·) combines two network outputs to predict the final blendshape weights).
Regarding claim 7, Saragih, in view of Olszewski teaches the non-transitory computer-readable data storage medium of claim 1, Saragih discloses wherein the processing further comprises, prior to retargeting the predicted blendshape weights for the facial expression of the wearer onto the avatar corresponding to the wearer: applying natural facial expression constraints to the predicted blendshape weights to ensure that the predicted blendshape weights do not correspond to an unnatural facial expression unlikely to be exhibitable by the wearer (Column 11, lines 39-57: controller 220 maps 530 the identified points to a three-dimensional model of a face for generating a graphical representation of the portions of the user's face corresponding to the captured images. In some embodiments, the three-dimensional model of the face is selected from a stored library of three-dimensional models based on characteristics of the user. For example, the user specifies one or more parameters and the controller 220 selects a three-dimensional model of a face from the library that has at least a threshold amount of parameters matching the parameters specified by the user; the controller 220 identifies points on the selected three-dimensional model corresponding to the points identified 520 from the captured images to map 530 the identified points to the selected three-dimensional model. Alternatively, the controller 220 receives an image of the user's face and selects a three-dimensional model of a face having at least a threshold similarity to the image of the user's face and maps 530 the identified points to the selected three-dimensional model; Column 12, lines 46-61: a differential transformation differently attenuates various weights associated with expression parameters (e.g., attenuates various weights associated with blendshapes). For example, for weights between a minimum value and a threshold value, the differential transform applies a quadratic penalty to the weights to reduce sparseness between facial expressions corresponding to different expression parameters (e.g., blendshapes). However, in the preceding example, the differential transform applies a linear penalty to weights between the threshold value and an additional threshold, which prevents attenuation by the differential transformation from attenuating the weights so differences between facial expressions corresponding to different expression parameters are reduced, resulting in an overly muted graphical representation of facial expressions of the user).
Regarding claim 9, Saragih discloses a method comprising: for each of a plurality of cohorts that each correspond to a different facial type, rendering avatar training images of avatars having the different facial type of the cohort and having facial expressions corresponding to specified blendshape weights (Fig. 3; Fig. 5; Column 5, lines 41-57: facial tracking system 160 generates reconstructions of portions of a face of a user wearing the HMD 105, as further described below in conjunction with FIGS. 2-5. In an embodiment, the facial tracking system 160 includes one or more facial sensors and a controller, as further described below in conjunction with FIG. 2. In some embodiments, the facial tracking system 160 also includes one or more illumination sources configured to illuminate portions of the user's face within fields of view of the one or more facial sensors. Based on data from the facial sensors, the controller generates a trained model that maps positions of points identified within images captured by the facial sensors to a set of animation parameters that determines a facial animation model projecting movement of the portions of the user's face into a three dimensional model of a face presented via a virtual reality environment of the HMD 105, as further described below in conjunction with FIG. 5; Column 11, lines 7-57: one or more facial sensors 210 communicate the captured images to the controller 220, which identifies 520 points corresponding to features of the portion of the user's face within each of the captured images. Features of the portion of the user's face corresponding to boundaries of parts of the user's face. For example, identified points correspond to positions along boundaries of the user's lips or positions along the user's chin. As an example, the identified points correspond to corners of the user's lips and various locations along boundaries of the user's lips. To identify 520 the points in images of a portion of the user's face, the controller 220 obtains a model trained on a set of images of various facial expressions made by various users that include the points on portions of the users' faces identified by the users. For example, a user makes specific facial expressions, and images of portions of the user's face are captured by facial sensors 210 when the user makes each specific facial expression. The user is subsequently prompted to identify various points in the captured images corresponding to each specific facial expression. Based on the points identified by the users when various specific facial expressions were made, a machine learned model is trained to identify 520 the points from the captured images. In various embodiments, different machine learned models are trained to identify 520 points on different portions of the user's face. The controller 220 applies the trained model to images captured 510 by different facial sensors 210 to identify 520 points in images of portions of the user's face captured 510 by different facial sensors 210. For example, the controller 220 identifies 520 points of portions of the user's face included in each image captured 510 by one or more facial sensors 210… controller 220 maps 530 the identified points to a three-dimensional model of a face for generating a graphical representation of the portions of the user's face corresponding to the captured images. In some embodiments, the three-dimensional model of the face is selected from a stored library of three-dimensional models based on characteristics of the user. For example, the user specifies one or more parameters and the controller 220 selects a three-dimensional model of a face from the library that has at least a threshold amount of parameters matching the parameters specified by the user; the controller 220 identifies points on the selected three-dimensional model corresponding to the points identified 520 from the captured images to map 530 the identified points to the selected three-dimensional model. Alternatively, the controller 220 receives an image of the user's face and selects a three-dimensional model of a face having at least a threshold similarity to the image of the user's face and maps 530 the identified points to the selected three-dimensional model); for each cohort, training a machine learning model based on the rendered avatar training images of the avatars having the different facial type of the cohort and based on the specified blendshape weights (Column 5, lines 41-57: facial tracking system 160 generates reconstructions of portions of a face of a user wearing the HMD 105, as further described below in conjunction with FIGS. 2-5. In an embodiment, the facial tracking system 160 includes one or more facial sensors and a controller, as further described below in conjunction with FIG. 2. In some embodiments, the facial tracking system 160 also includes one or more illumination sources configured to illuminate portions of the user's face within fields of view of the one or more facial sensors. Based on data from the facial sensors, the controller generates a trained model that maps positions of points identified within images captured by the facial sensors to a set of animation parameters that determines a facial animation model projecting movement of the portions of the user's face into a three dimensional model of a face presented via a virtual reality environment of the HMD 105, as further described below in conjunction with FIG. 5; Column 11, line 58-Column 12, line 20: a facial animation model of the portions of the user's face of which the facial sensors 210 captured 510 images. In various embodiments, the facial animation model is an expression parameter comprising a parametric representation of human faces, where different parameters correspond to different expressions of the portion of the user's face included in a captured image. For example, the parametric representation of human faces is a blendshape model that models facial expressions of the user as a linear combination of blendshapes that each correspond to an expression of the portion of the user's face included in a captured image. In various embodiments, the facial animation model calculates a combination of coefficients corresponding to different human facial expressions that determine a weight of each human facial expression in a combination. When the facial animation model is a blendshape model, the facial animation model calculates a vector of blendshape coefficients that determine the weight of each expression mesh in the linear combination. Hence, a vector of blendshape coefficients is determined for each image captured by a facial sensor 210. The blendshape coefficients may be extracted based on positions of the identified points within various captured images, so the positions of the identified points within captured images determine weights corresponding to various facial expressions in the calculated vector); selecting, for a wearer of a head-mountable display (HMD), the cohort having the different facial type to which a facial type of the wearer corresponds (Fig. 3; Fig. 5; Column 5, lines 41-57: facial tracking system 160 generates reconstructions of portions of a face of a user wearing the HMD 105, as further described below in conjunction with FIGS. 2-5. In an embodiment, the facial tracking system 160 includes one or more facial sensors and a controller, as further described below in conjunction with FIG. 2. In some embodiments, the facial tracking system 160 also includes one or more illumination sources configured to illuminate portions of the user's face within fields of view of the one or more facial sensors. Based on data from the facial sensors, the controller generates a trained model that maps positions of points identified within images captured by the facial sensors to a set of animation parameters that determines a facial animation model projecting movement of the portions of the user's face into a three dimensional model of a face presented via a virtual reality environment of the HMD 105, as further described below in conjunction with FIG. 5; Column 11, lines 7-57: one or more facial sensors 210 communicate the captured images to the controller 220, which identifies 520 points corresponding to features of the portion of the user's face within each of the captured images. Features of the portion of the user's face corresponding to boundaries of parts of the user's face. For example, identified points correspond to positions along boundaries of the user's lips or positions along the user's chin. As an example, the identified points correspond to corners of the user's lips and various locations along boundaries of the user's lips. To identify 520 the points in images of a portion of the user's face, the controller 220 obtains a model trained on a set of images of various facial expressions made by various users that include the points on portions of the users' faces identified by the users. For example, a user makes specific facial expressions, and images of portions of the user's face are captured by facial sensors 210 when the user makes each specific facial expression. The user is subsequently prompted to identify various points in the captured images corresponding to each specific facial expression. Based on the points identified by the users when various specific facial expressions were made, a machine learned model is trained to identify 520 the points from the captured images. In various embodiments, different machine learned models are trained to identify 520 points on different portions of the user's face. The controller 220 applies the trained model to images captured 510 by different facial sensors 210 to identify 520 points in images of portions of the user's face captured 510 by different facial sensors 210. For example, the controller 220 identifies 520 points of portions of the user's face included in each image captured 510 by one or more facial sensors 210… controller 220 maps 530 the identified points to a three-dimensional model of a face for generating a graphical representation of the portions of the user's face corresponding to the captured images. In some embodiments, the three-dimensional model of the face is selected from a stored library of three-dimensional models based on characteristics of the user. For example, the user specifies one or more parameters and the controller 220 selects a three-dimensional model of a face from the library that has at least a threshold amount of parameters matching the parameters specified by the user; the controller 220 identifies points on the selected three-dimensional model corresponding to the points identified 520 from the captured images to map 530 the identified points to the selected three-dimensional model. Alternatively, the controller 220 receives an image of the user's face and selects a three-dimensional model of a face having at least a threshold similarity to the image of the user's face and maps 530 the identified points to the selected three-dimensional model).
Saragih does not explicitly disclose applying the machine learning model for the selected cohort to predict blendshape weights for a facial expression of the wearer from a set of facial images captured by the HMD of the wearer when exhibiting the facial expression.
However, Olszewski teaches facial animation using HMDs (Abstract), further comprising applying the machine learning model for the selected cohort to predict blendshape weights for a facial expression of the wearer from a set of facial images captured by the HMD of the wearer when exhibiting the facial expression (Fig. 17; Section 5: goal is to recover detailed 3D facial expressions from video frames. In this paper, we address this problem by representing a face as a set of facial blendshape meshes. Then, our algorithm determines the blendshape weights that best correspond to each frame. More concretely, let us assume we are given a generic blendshape model as a set of meshes b = {b0,b1,··· ,bN}. Our target expression of frame It at time t can be formulated as:…where b0 is the neutral face expression, and wt ∈ [0,1]N is a corresponding blendshape weight vector. Then, our goal is to determine the value for w that best corresponds to a given image or video frame of a human face. In this paper, we formulate this as learning a mapping function ψ, which predicts a blendshape weight vector of image It… the neutral network is a classification function that determines whether the expression in the frame is neutral or not. The output of this network is used for dampening the estimation toward the neutral expression. The final output of our model, ψ(·) combines two network outputs to predict the final blendshape weights; Section 2: training data are collected using a separate RGB-D sensor and linear blendshape models obtained from the facial animation software Faceshift [2014]. As in this work, they also adopt a deep learning framework for regression, but only poor results could be demonstrated even with user-specific training data. Instead, our system generates highly compelling speech animation by mapping the video input from cameras directed at the regions of interest directly to the appropriate facial expression controls. Furthermore, we remove the need for user-specific training data by training our system on users of varying appearance). Olszewski teaches that this will allow for production of realistic facial expressions and animations (Section 1). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Saragih with the features of above as taught by Olszewski so as to allow for production of realistic facial expressions and animations as presented by Olszewski.
Regarding claim 10, Saragih, in view of Olszewski teaches the method of claim 9, Saragih discloses further comprising: retargeting the predicted blendshape weights for the facial expression of the wearer onto an avatar corresponding to the wearer to render the avatar with the facial expression (Column 5, lines 41-57: facial tracking system 160 generates reconstructions of portions of a face of a user wearing the HMD 105, as further described below in conjunction with FIGS. 2-5. In an embodiment, the facial tracking system 160 includes one or more facial sensors and a controller, as further described below in conjunction with FIG. 2. In some embodiments, the facial tracking system 160 also includes one or more illumination sources configured to illuminate portions of the user's face within fields of view of the one or more facial sensors. Based on data from the facial sensors, the controller generates a trained model that maps positions of points identified within images captured by the facial sensors to a set of animation parameters that determines a facial animation model projecting movement of the portions of the user's face into a three dimensional model of a face presented via a virtual reality environment of the HMD 105, as further described below in conjunction with FIG. 5; Column 11, line 58-Column 12, line 20: a facial animation model of the portions of the user's face of which the facial sensors 210 captured 510 images. In various embodiments, the facial animation model is an expression parameter comprising a parametric representation of human faces, where different parameters correspond to different expressions of the portion of the user's face included in a captured image. For example, the parametric representation of human faces is a blendshape model that models facial expressions of the user as a linear combination of blendshapes that each correspond to an expression of the portion of the user's face included in a captured image. In various embodiments, the facial animation model calculates a combination of coefficients corresponding to different human facial expressions that determine a weight of each human facial expression in a combination. When the facial animation model is a blendshape model, the facial animation model calculates a vector of blendshape coefficients that determine the weight of each expression mesh in the linear combination. Hence, a vector of blendshape coefficients is determined for each image captured by a facial sensor 210. The blendshape coefficients may be extracted based on positions of the identified points within various captured images, so the positions of the identified points within captured images determine weights corresponding to various facial expressions in the calculated vector); displaying the rendered avatar (Fig. 6; Column 13, lines 25-40: FIG. 6 is a conceptual diagram of generating a graphical representation of a user's face while the user wears a head mounted display (HMD) 105. In the example of FIG. 6, two facial sensors 210A, 210B (also referred to individually and collectively using reference number 210). Each facial sensor 210A, 210B is positioned to capture images of a portion of the user's face while the user wears the HMD 105. In the example of FIG. 6, the facial sensors 210A, 210B each capture images 600A, 600B of a corresponding portion of the user's face that is below the HMD 105 when the user wears the HMD 105. For purposes of illustration, FIG. 6 shows an example where the facial sensor 210A captures images 600A of a user's mouth from a particular angle relative to the user's mouth, while the facial sensor 210B captures images 600B of the user's mouth from a different angle relative to the user's mouth; Column 14, lines 12-25: the controller 220 determines transforms 615 mapping the two-dimensional images 600A, 600B to three dimensions and one or more differential transformations accounting for movement of facial sensors 210A, 210B in one or more directions, as further described above in conjunction with FIG. 5. From the facial animation model 610 and the transforms 615, the controller generates a rendering model 620 that is applied to a three-dimensional model of a face to generate a graphical representation of the portions of the user's face included in the captured images 600A, 600B that is modified by the rendering model 620 to replicate facial expressions captured by the facial sensors 210A, 210A while the user wears the HMD 205).
Regarding claim 11, Saragih, in view of Olszewski teaches the method of claim 9, Saragih discloses wherein the set of facial images captured by the HMD of the wearer comprise left and right eye images of facial portions of the wearer respectively including left and right eyes of the wearer and a mouth image of a lower facial portion of the wearer including a mouth of the wearer (Fig. 6; Column 10, lines 15-30: front rigid body 305 includes an optical block 118 that magnifies image light from the electronic display 115, and in some embodiments, also corrects for one or more additional optical errors (e.g., distortion, astigmatism, etc.) in the image light from the electronic display 115. The optics block 118 directs the image light from the electronic display 115 to a pupil 405 of the user's eye 410 by directing the altered image light to an exit pupil of the front rigid body 305 that is a location where the user's eye 410 is positioned when the user wears the HMD 300. For purposes of illustration, FIG. 4 shows a cross section of the right side of the front rigid body 305 (from the perspective of the user) associated with a single eye 410, but another optical block, separate from the optical block 118, provides altered image light to another eye (i.e., a left eye) of the user; Column 13, lines 25-40: FIG. 6 is a conceptual diagram of generating a graphical representation of a user's face while the user wears a head mounted display (HMD) 105. In the example of FIG. 6, two facial sensors 210A, 210B (also referred to individually and collectively using reference number 210). Each facial sensor 210A, 210B is positioned to capture images of a portion of the user's face while the user wears the HMD 105. In the example of FIG. 6, the facial sensors 210A, 210B each capture images 600A, 600B of a corresponding portion of the user's face that is below the HMD 105 when the user wears the HMD 105. For purposes of illustration, FIG. 6 shows an example where the facial sensor 210A captures images 600A of a user's mouth from a particular angle relative to the user's mouth, while the facial sensor 210B captures images 600B of the user's mouth from a different angle relative to the user's mouth), the method further comprising: for each avatar training image of an avatar having a facial expression, simulating left and right eye avatar training images in correspondence with the left and right eye images captured by the HMD and a mouth avatar training image in correspondence with the mouth image captured by the HMD (Fig. 6; Column 10, lines 15-30: front rigid body 305 includes an optical block 118 that magnifies image light from the electronic display 115, and in some embodiments, also corrects for one or more additional optical errors (e.g., distortion, astigmatism, etc.) in the image light from the electronic display 115. The optics block 118 directs the image light from the electronic display 115 to a pupil 405 of the user's eye 410 by directing the altered image light to an exit pupil of the front rigid body 305 that is a location where the user's eye 410 is positioned when the user wears the HMD 300. For purposes of illustration, FIG. 4 shows a cross section of the right side of the front rigid body 305 (from the perspective of the user) associated with a single eye 410, but another optical block, separate from the optical block 118, provides altered image light to another eye (i.e., a left eye) of the user), and wherein, for each cohort, the machine learning model is trained using the left and right eye avatar training images and the mouth avatar training image simulated for each avatar training image of an avatar having the different facial type of the cohort (Column 10, lines 30-48: controller 220 is communicatively coupled to the electronic display 115, allowing the controller 220 to provide content for to the electronic display 115 for presentation to the user (e.g., a graphical representation of one or more portions 415 of the user's face based on data captured by the facial sensor 210). Additionally or alternatively, the controller 220 is communicatively coupled to the console 110 and communicates a facial animation model for generating graphical representations of one or more portions 415 of the user's face to the console 110, which includes one or more graphical representations of portions 415 of the user's face in content provided to the electronic display 115, or generates content for presentation by the electronic display 115 based at least in part on the facial animation model received from the controller 220. Additionally, the controller 220 is communicatively coupled to the facial sensor 210, allowing the controller 220 to provide instructions to the facial sensor 210 for capturing images of the portion 415 of the user's face).
Regarding claim 12, Saragih, in view of Olszewski teaches the method of claim 9, wherein selecting, for the wearer, the cohort having the different facial type to which the facial type of the wearer corresponds comprises: receiving wearer selection of the different facial type of the cohort corresponding to the facial type of the wearer (Saragih: Column 11, lines 7-38: one or more facial sensors 210 communicate the captured images to the controller 220, which identifies 520 points corresponding to features of the portion of the user's face within each of the captured images. Features of the portion of the user's face corresponding to boundaries of parts of the user's face. For example, identified points correspond to positions along boundaries of the user's lips or positions along the user's chin. As an example, the identified points correspond to corners of the user's lips and various locations along boundaries of the user's lips. To identify 520 the points in images of a portion of the user's face, the controller 220 obtains a model trained on a set of images of various facial expressions made by various users that include the points on portions of the users' faces identified by the users. For example, a user makes specific facial expressions, and images of portions of the user's face are captured by facial sensors 210 when the user makes each specific facial expression. The user is subsequently prompted to identify various points in the captured images corresponding to each specific facial expression. Based on the points identified by the users when various specific facial expressions were made, a machine learned model is trained to identify 520 the points from the captured images. In various embodiments, different machine learned models are trained to identify 520 points on different portions of the user's face. The controller 220 applies the trained model to images captured 510 by different facial sensors 210 to identify 520 points in images of portions of the user's face captured 510 by different facial sensors 210. For example, the controller 220 identifies 520 points of portions of the user's face included in each image captured 510 by one or more facial sensors 210); or applying a classifier machine learning model to the set of facial images of the wearer to identify the different facial type of the cohort to which the facial type of the wearer corresponds (Olszewski: Section 5: In order to enable high fidelity facial speech animation, our CNN based method builds upon two main ideas: (1) exploiting a sequence of frames to capture temporal signals, and (2) explicitly dampening an estimation toward the neutral face when necessary because humans are sensitive to neutral expressions. Our model thus contains two sub-networks. The expression network is a regression function that estimates facial expression weights from a sequence of frames. On the other hand, the neutral network is a classification function that determines whether the expression in the frame is neutral or not. The output of this network is used for dampening the estimation toward the neutral expression. The final output of our model, ψ(·) combines two network outputs to predict the final blendshape weights).
Regarding claim 14, Saragih discloses a head-mountable display (HMD) comprising: one or multiple cameras to capture a set of images of a wearer of the HMD (Fig. 1; Fig. 3; Column 3, lines 10-28: FIG. 1 is a block diagram of a system environment 100 for providing virtual reality (VR) content or augmented reality (AR) content in accordance with an embodiment. The system environment 100 shown by FIG. 1 comprises a head mounted display (HMD) 105, an imaging device 135, and an input/output (I/O) interface 140 that are each coupled to a console 110. While FIG. 1 shows an example system environment 100 including one HMD 105, one imaging device 135, and one I/O interface 140, in other embodiments, any number of these components are included in the system environment 100. For example, an embodiment includes multiple HMDs 105 each having an associated I/O interface 140 and being monitored by one or more imaging devices 135, with each HMD 105, I/O interface 140, and imaging device 135 communicating with the console; Column 5, lines 58-67: imaging device 135 generates slow calibration data in accordance with calibration parameters received from the console 110. Slow calibration data includes one or more images showing observed positions of the locators 120 that are detectable by the imaging device 135. In some embodiments, the imaging device 135 includes one or more cameras, one or more video cameras, any other device capable of capturing images including one or more of the locators 120, or some combination thereof); a processor (Column 6, lines 46-54: application is a group of instructions, that when executed by a processor, generates content for presentation to the user. Content generated by an application may be in response to inputs received from the user via movement of the HMD 105 or the I/O interface 140. Examples of applications include: gaming applications, conferencing applications, video playback application, or other suitable applications); and a memory storing program code executable by the processor (Column 14, lines 57-67: apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability)) to: retarget the predicted blendshape weights for the facial expression of the wearer onto an avatar corresponding to the wearer to render the avatar with the facial expression (Column 5, lines 41-57: facial tracking system 160 generates reconstructions of portions of a face of a user wearing the HMD 105, as further described below in conjunction with FIGS. 2-5. In an embodiment, the facial tracking system 160 includes one or more facial sensors and a controller, as further described below in conjunction with FIG. 2. In some embodiments, the facial tracking system 160 also includes one or more illumination sources configured to illuminate portions of the user's face within fields of view of the one or more facial sensors. Based on data from the facial sensors, the controller generates a trained model that maps positions of points identified within images captured by the facial sensors to a set of animation parameters that determines a facial animation model projecting movement of the portions of the user's face into a three dimensional model of a face presented via a virtual reality environment of the HMD 105, as further described below in conjunction with FIG. 5; Column 11, line 58-Column 12, line 20: a facial animation model of the portions of the user's face of which the facial sensors 210 captured 510 images. In various embodiments, the facial animation model is an expression parameter comprising a parametric representation of human faces, where different parameters correspond to different expressions of the portion of the user's face included in a captured image. For example, the parametric representation of human faces is a blendshape model that models facial expressions of the user as a linear combination of blendshapes that each correspond to an expression of the portion of the user's face included in a captured image. In various embodiments, the facial animation model calculates a combination of coefficients corresponding to different human facial expressions that determine a weight of each human facial expression in a combination. When the facial animation model is a blendshape model, the facial animation model calculates a vector of blendshape coefficients that determine the weight of each expression mesh in the linear combination. Hence, a vector of blendshape coefficients is determined for each image captured by a facial sensor 210. The blendshape coefficients may be extracted based on positions of the identified points within various captured images, so the positions of the identified points within captured images determine weights corresponding to various facial expressions in the calculated vector).
Saragih does not explicitly disclose apply a machine learning model for a cohort corresponding to a facial type of the wearer to the captured set of images to predict blendshape weights for a facial expression of the wearer exhibited within the captured set of images.
However, Olszewski teaches facial animation using HMDs (Abstract), further comprising apply a machine learning model for a cohort corresponding to a facial type of the wearer to the captured set of images to predict blendshape weights for a facial expression of the wearer exhibited within the captured set of images (Fig. 17; Section 5: goal is to recover detailed 3D facial expressions from video frames. In this paper, we address this problem by representing a face as a set of facial blendshape meshes. Then, our algorithm determines the blendshape weights that best correspond to each frame. More concretely, let us assume we are given a generic blendshape model as a set of meshes b = {b0,b1,··· ,bN}. Our target expression of frame It at time t can be formulated as:…where b0 is the neutral face expression, and wt ∈ [0,1]N is a corresponding blendshape weight vector. Then, our goal is to determine the value for w that best corresponds to a given image or video frame of a human face. In this paper, we formulate this as learning a mapping function ψ, which predicts a blendshape weight vector of image It… the neutral network is a classification function that determines whether the expression in the frame is neutral or not. The output of this network is used for dampening the estimation toward the neutral expression. The final output of our model, ψ(·) combines two network outputs to predict the final blendshape weights; Section 2: training data are collected using a separate RGB-D sensor and linear blendshape models obtained from the facial animation software Faceshift [2014]. As in this work, they also adopt a deep learning framework for regression, but only poor results could be demonstrated even with user-specific training data. Instead, our system generates highly compelling speech animation by mapping the video input from cameras directed at the regions of interest directly to the appropriate facial expression controls. Furthermore, we remove the need for user-specific training data by training our system on users of varying appearance). Olszewski teaches that this will allow for production of realistic facial expressions and animations (Section 1). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Saragih with the features of above as taught by Olszewski so as to allow for production of realistic facial expressions and animations as presented by Olszewski.
Regarding claim 15, Saragih, in view of Olszewski teaches the HMD of claim 14, Saragih discloses wherein the cohort is selected from a plurality of candidate cohorts each corresponding to a different facial type and each having a differently trained machine learning model to predict the blendshape weights (Fig. 3; Fig. 5; Column 5, lines 41-57: facial tracking system 160 generates reconstructions of portions of a face of a user wearing the HMD 105, as further described below in conjunction with FIGS. 2-5. In an embodiment, the facial tracking system 160 includes one or more facial sensors and a controller, as further described below in conjunction with FIG. 2. In some embodiments, the facial tracking system 160 also includes one or more illumination sources configured to illuminate portions of the user's face within fields of view of the one or more facial sensors. Based on data from the facial sensors, the controller generates a trained model that maps positions of points identified within images captured by the facial sensors to a set of animation parameters that determines a facial animation model projecting movement of the portions of the user's face into a three dimensional model of a face presented via a virtual reality environment of the HMD 105, as further described below in conjunction with FIG. 5; Column 11, lines 7-57: one or more facial sensors 210 communicate the captured images to the controller 220, which identifies 520 points corresponding to features of the portion of the user's face within each of the captured images. Features of the portion of the user's face corresponding to boundaries of parts of the user's face. For example, identified points correspond to positions along boundaries of the user's lips or positions along the user's chin. As an example, the identified points correspond to corners of the user's lips and various locations along boundaries of the user's lips. To identify 520 the points in images of a portion of the user's face, the controller 220 obtains a model trained on a set of images of various facial expressions made by various users that include the points on portions of the users' faces identified by the users. For example, a user makes specific facial expressions, and images of portions of the user's face are captured by facial sensors 210 when the user makes each specific facial expression. The user is subsequently prompted to identify various points in the captured images corresponding to each specific facial expression. Based on the points identified by the users when various specific facial expressions were made, a machine learned model is trained to identify 520 the points from the captured images. In various embodiments, different machine learned models are trained to identify 520 points on different portions of the user's face. The controller 220 applies the trained model to images captured 510 by different facial sensors 210 to identify 520 points in images of portions of the user's face captured 510 by different facial sensors 210. For example, the controller 220 identifies 520 points of portions of the user's face included in each image captured 510 by one or more facial sensors 210… controller 220 maps 530 the identified points to a three-dimensional model of a face for generating a graphical representation of the portions of the user's face corresponding to the captured images. In some embodiments, the three-dimensional model of the face is selected from a stored library of three-dimensional models based on characteristics of the user. For example, the user specifies one or more parameters and the controller 220 selects a three-dimensional model of a face from the library that has at least a threshold amount of parameters matching the parameters specified by the user; the controller 220 identifies points on the selected three-dimensional model corresponding to the points identified 520 from the captured images to map 530 the identified points to the selected three-dimensional model. Alternatively, the controller 220 receives an image of the user's face and selects a three-dimensional model of a face having at least a threshold similarity to the image of the user's face and maps 530 the identified points to the selected three-dimensional model).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Saragih, in view of Olszewski, and further in view of Fonte et al. (US Pub. 2015/0055086), hereinafter Fonte.
Regarding claim 3, Saragih, in view of Olszewski teaches the non-transitory computer-readable data storage medium of claim 1.
Saragih, in view of Olszewski does not explicitly disclose wherein the different facial type to which each candidate cohort corresponds is a different one of a plurality of facial shapes.
However, Fonte teaches avatar manipulation and generation (Abstract), further comprising wherein the different facial type to which each candidate cohort corresponds is a different one of a plurality of facial shapes (Paragraph [0189]: Overall face size, such as area of the front of the face or volume of the head from the model; Face width; Face height; Ear positions (each ear may have different heights); Inter-pupillary distance; Size of eyes, such as area or length or height; Spacing between eyes; Asymmetries of the nose, eyes or mouth; Color of eyes; Color of hair; Amount and shape of hair; Color of skin; Ethnicity; Age; Location or local style trends; Gender; Assessment of cheekbone shape and location; Angle of forehead; Angle of cheeks; Circles under eyes; Eyebrow size and shape; Shape of face (e.g. round, square, oval, etc); Vertical position of eyes relative to center of face; Hair style (e.g. up, down, long, balding, straight, curvy); facial hair; Intensity or softness of features). Fonte teaches that this will more accurate modeling (Paragraph [0020]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Saragih, in view of Olszewski with the features of above as taught by Fonte so as to allow for more accurate modeling as presented by Fonte.
Allowable Subject Matter
Claims 4, 8, and 13 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement of reasons for allowance:
Claim 4 would be allowable over the prior art of record since the cited references taken individually or in combination fails to particularly disclose or suggest a medium comprising: wherein the facial shapes comprise an oval facial shape, a square facial shape, a round facial shape, a diamond facial shape, a rectangular facial shape, a heart facial shape, and a diamond facial shape, as presented in the environment of the remaining limitations of claim 4. It is noted that the closest prior art, Saragih, shows the limitations of parent claim 1. However, Saragih fails to disclose or suggest wherein the facial shapes comprise an oval facial shape, a square facial shape, a round facial shape, a diamond facial shape, a rectangular facial shape, a heart facial shape, and a diamond facial shape.
Claim 8 would be allowable over the prior art of record since the cited references taken individually or in combination fails to particularly disclose or suggest a medium comprising: the rendered avatar is displayed continuously over time, and wherein each of a plurality of times the blendshape weights are predicted, the processing further comprises, prior to retargeting the predicted blendshape weights for the facial expression of the wearer onto the avatar corresponding to the wearer: applying temporal consistency constraints to the predicted blendshape weights as currently predicted in comparison to as previously predicted to ensure that the predicted blendshape weights do not correspond to an unnatural change in facial expression unlikely to be exhibitable by the wearer, as presented in the environment of the remaining limitations of claim 8. It is noted that the closest prior art, Saragih, shows the non-transitory computer-readable data storage medium of claim 1, wherein the set of facial images of the wearer are captured, the machine learning for the selected cohort is applied to the captured set of images to predict the blendshape weights, the predicted blendshape weights are retargeted onto the avatar to render the avatar. However, Saragih fails to disclose or suggest the rendered avatar is displayed continuously over time, and wherein each of a plurality of times the blendshape weights are predicted, the processing further comprises, prior to retargeting the predicted blendshape weights for the facial expression of the wearer onto the avatar corresponding to the wearer: applying temporal consistency constraints to the predicted blendshape weights as currently predicted in comparison to as previously predicted to ensure that the predicted blendshape weights do not correspond to an unnatural change in facial expression unlikely to be exhibitable by the wearer.
Claim 13 would be allowable over the prior art of record since the cited references taken individually or in combination fails to particularly disclose or suggest a medium comprising: wherein the different facial type to which each cohort corresponds is a different one of a plurality of facial shapes comprising an oval facial shape, a square facial shape, a round facial shape, a diamond facial shape, a rectangular facial shape, a heart facial shape, and a diamond facial shape, as presented in the environment of the remaining limitations of claim 13. It is noted that the closest prior art, Saragih, shows the limitations of parent claim 9. However, Saragih fails to disclose or suggest wherein the different facial type to which each cohort corresponds is a different one of a plurality of facial shapes comprising an oval facial shape, a square facial shape, a round facial shape, a diamond facial shape, a rectangular facial shape, a heart facial shape, and a diamond facial shape.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW D SALVUCCI whose telephone number is (571)270-5748. The examiner can normally be reached M-F: 7:30-4:00PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XIAO WU can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW SALVUCCI/Primary Examiner, Art Unit 2613
Prosecution Insights