DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation - 35 USC § 101
The limitations “generating a three-dimensional model of the face of the subject, wherein the three-dimensional model comprises a first model feature associated with the audio-correlated facial feature and a second model feature associated with the expression-like facial feature; and providing the three-dimensional model of the face of the subject to a display in a client device running an immersive reality application.” are considered a practical application of creating and outputting a virtual 3D model synced with phonemes and visemes of a subject for facial animation.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 2-7, 10-12, 14-21 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 3, 5-7, 11-14 of U.S. Patent No. 12,159,339. Although the claims at issue are not identical, they are not patentably distinct from each other because the notion of the claims does refer to the same invention and claim 2 of the current application corresponds with claim 1 of U.S. Patent No. 12,159,339. Claim 1 of U.S. Patent No. 12,159,339 anticipates claim 2 of the current application because it includes all of the limitations of claim 2 of the current application.
Below is a limitation mapping between claim 2 of the current application and claim 1 of U.S. Patent No. 12,159,339.
Current Application
12,159,339
2. (New) A computer-implemented method, comprising: identifying, based at least in part upon an audio capture of a voice of a subject, an audio-correlated facial feature;
identifying, based at least in part upon an image capture of a portion of a face of the subject, an expression-like facial feature of the subject;
generating a three-dimensional model of the face of the subject, wherein the three- dimensional model comprises a first model feature associated with the audio-correlated facial feature and a second model feature associated with the expression-like facial feature;
1. A computer-implemented method, comprising: identifying, from an audio capture of a subject, an audio-correlated facial feature;
identifying an expression-like facial feature of the subject;
generating a three-dimensional model of the face of the subject with the synthesized mesh based on a ground truth image of the subject;
generating a first mesh for a lower portion of a face of the subject, based on the audio-correlated facial feature;
generating a second mesh for an upper portion of the face of the subject based on the expression-like facial feature; forming a synthesized mesh with the first mesh and the second mesh;
and providing the three-dimensional model of the face of the subject to a display in a client device running an immersive reality application.
and providing the three-dimensional model of the face of the subject to a display in a client device running an immersive reality application that includes the subject.
Below is part one of claim mapping between the current application and U.S. Patent No. 12,159,339
Current Application
2
3
4
5
6
7, 2
10, 2
11, 10, 2
12, 10, 2
12,159,339
Below is part two of claim mapping between the current application and U.S. Patent No. 12,159,339
Current Application
14
15
16
17
18
19
20
21
12,159,339
11
3, 1
14
13
1
3, 1
6
5
Claim 15 and 18-21 are merely a different statutory category to that of claims 1, 3, 5, 6 of U.S. Patent No. 12,159,339, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the method of a different statutory category with generic computer components performing the same function of the method.
Claims 8, 9, 13 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 of U.S. Patent No. 12,159,339 in view of Heller et al. (US 2020/0160581)(Hereinafter referred to as Heller). U.S. Patent No. 12,159,339 teaches all of the limitations of claim 8 except, wherein the upper facial feature is related to an eyebrow of the subject or an eye of the subject.
Heller teaches creating visual models subjects faces utilizing phonemes and visemes (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101])
U.S. Patent No. 12,159,339 and Heller teaches creating virtual models using audio and images, and Heller teaches that various parts of the user are identified when utilizing phoneme viseme relationships to audio and visual data and therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of U.S. Patent No. 12,159,339 with the technique of identifying a plurality of regions for correlated with viseme phoneme data of Heller such that the system could provide a realistic animation output.
U.S. Patent No. 12,159,339 teaches all of the limitations of claim 9 except wherein the audio-correlated facial feature is associated with a shape of a lip of the subject.
Heller teaches creating visual models subjects faces utilizing phonemes and visemes (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101])
U.S. Patent No. 12,159,339 and Heller teaches creating virtual models using audio and images, and Heller teaches that various parts of the user are identified when utilizing phoneme viseme relationships to audio and visual data and therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of U.S. Patent No. 12,159,339 with the technique of identifying a plurality of regions for correlated with viseme phoneme data of Heller such that the system could provide a realistic animation output.
U.S. Patent No. 12,159,339 teaches all of the limitations of claim 13 except wherein the three- dimensional model of the face of the subject is based at least in part upon information associated with the face of the subject with a neutral expression.
Heller teaches creating visual models subjects faces utilizing phonemes and visemes (neutral expression; See figure 21)(As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101])
U.S. Patent No. 12,159,339 and Heller teaches creating virtual models using audio and images, and Heller teaches that various parts of the user are identified when utilizing phoneme viseme relationships to audio and visual data and therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of U.S. Patent No. 12,159,339 with the technique of identifying a plurality of regions for correlated with viseme phoneme data of Heller such that the system could provide a realistic animation output.
Claims 2-7, 10-12, 14-21 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 3, 5-7, 11-14 of U.S. Patent No. 11,756,250. Although the claims at issue are not identical, they are not patentably distinct from each other because the notion of the claims does refer to the same invention and claim 2 of the current application corresponds with claim 1 of U.S. Patent No. 11,756,250. Claim 1 of U.S. Patent No. 11,756,250 anticipates claim 2 of the current application because it includes all of the limitations of claim 2 of the current application.
Below is a limitation mapping between claim 2 of the current application and claim 1 of U.S. Patent No. 11,756,250.
Current Application
11,756,250
2. (New) A computer-implemented method, comprising: identifying, based at least in part upon an audio capture of a voice of a subject, an audio-correlated facial feature;
identifying, based at least in part upon an image capture of a portion of a face of the subject, an expression-like facial feature of the subject;
generating a three-dimensional model of the face of the subject, wherein the three- dimensional model comprises a first model feature associated with the audio-correlated facial feature and a second model feature associated with the expression-like facial feature;
1. A computer-implemented method, comprising: identifying, from an audio capture of a subject, an audio-correlated facial feature;
identifying an expression-like facial feature of the subject;
generating a second mesh for an upper portion of a face of the subject based on the expression-like facial feature; forming a synthesized mesh with the first mesh and the second mesh; generating a three-dimensional model of the face of the subject with the synthesized mesh based on the loss value; generating a first mesh for a lower portion of a face of the subject, based on the audio-correlated facial feature;
determining a loss value of the synthesized mesh based on a ground truth image of the subject;
and providing the three-dimensional model of the face of the subject to a display in a client device running an immersive reality application.
and providing the three-dimensional model of the face of the subject to a display in a client device running an immersive reality application that includes the subject.
Below is part one of claim mapping between the current application and U.S. Patent No. 11,756,250
Current Application
2
3
4
5
6
7, 2
10, 2
11, 10, 2
12, 10, 2
11,756,250
Below is part two of claim mapping between the current application and U.S. Patent No. 11,756,250
Current Application
14
15
16
17
18
19
20
21
11,756,250
11
3, 1
14
13
1
3, 1
6
5
Claim 15 and 18-21 are merely a different statutory category to that of claims 1, 3, 5, 6 of U.S. Patent No. 11,756,250, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify the method of a different statutory category with generic computer components performing the same function of the method.
Claims 8, 9, 13 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 of U.S. Patent No. 11,756,250 in view of Heller et al. (US 2020/0160581)(Hereinafter referred to as Heller). U.S. Patent No. 11,756,250 teaches all of the limitations of claim 8 except, wherein the upper facial feature is related to an eyebrow of the subject or an eye of the subject.
Heller teaches creating visual models subjects faces utilizing phonemes and visemes (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101])
U.S. Patent No. 11,756,250 and Heller teaches creating virtual models using audio and images, and Heller teaches that various parts of the user are identified when utilizing phoneme viseme relationships to audio and visual data and therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of U.S. Patent No. 11,756,250 with the technique of identifying a plurality of regions for correlated with viseme phoneme data of Heller such that the system could provide a realistic animation output.
U.S. Patent No. 11,756,250 teaches all of the limitations of claim 9 except wherein the audio-correlated facial feature is associated with a shape of a lip of the subject.
Heller teaches creating visual models subjects faces utilizing phonemes and visemes (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101])
U.S. Patent No. 11,756,250 and Heller teaches creating virtual models using audio and images, and Heller teaches that various parts of the user are identified when utilizing phoneme viseme relationships to audio and visual data and therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of U.S. Patent No. 11,756,250 with the technique of identifying a plurality of regions for correlated with viseme phoneme data of Heller such that the system could provide a realistic animation output.
U.S. Patent No. 11,756,250 teaches all of the limitations of claim 13 except wherein the three- dimensional model of the face of the subject is based at least in part upon information associated with the face of the subject with a neutral expression.
Heller teaches creating visual models subjects faces utilizing phonemes and visemes (neutral expression; See figure 21)(As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101])
U.S. Patent No. 11,756,250 and Heller teaches creating virtual models using audio and images, and Heller teaches that various parts of the user are identified when utilizing phoneme viseme relationships to audio and visual data and therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of U.S. Patent No. 11,756,250 with the technique of identifying a plurality of regions for correlated with viseme phoneme data of Heller such that the system could provide a realistic animation output.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 2-5, 7, 8, 9, 13-16, 18-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Heller et al. (US 2020/0160581)(Hereinafter referred to as Heller).
Regarding claim 2, Heller teaches a computer-implemented method (Certain embodiments involve automatically detecting video frames that depict visemes and that are usable for generating
an animatable puppet. For example, a computing device accesses video frames depicting a person performing gestures usable for generating a layered puppet, including a viseme gesture corresponding to a target sound or phoneme. See abstract), comprising:
identifying, based at least in part upon an audio capture of a voice of a subject, an audio- correlated facial feature (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]);
identifying, based at least in part upon an image capture of a portion of a face of the subject, an expression-like facial feature of the subject (As used herein, the term "viseme" is used to refer to an image that depicts a person's face as he or she speaks a particular phoneme. See paragraph [0037]) (The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035]) ( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]);
generating a three-dimensional model of the face of the subject, wherein the three- dimensional model comprises a first model feature associated with the audio-correlated facial feature and a second model feature associated with the expression-like facial feature (Thus, embodiments described herein improve computer-implemented processes for automatically creating animations of facial expressions (e.g., visemes) that can be synchronized with appropriate sounds ( e.g., phonemes). See paragraph [0039])( creating, from at least some of the video frames, a puppet animation of the gestures, wherein an animation of the viseme corresponding to the target sound or phoneme is generated from the particular video frame; and outputting the puppet animation to a presentation device. See claim 1)( In some embodiments, the content stream obtained or received by the computer graphics system 402 includes one or more frames in which the character is performing various gestures that can be used to generate an animatable puppet. As an example, in a frame, the character is making a gesture with the character's mouth that resembles a viseme for a sound, phoneme, phone, etc. For instance, in a frame of the content stream, the character is making a gesture that resembles the character saying the "H" sound ( e.g., a frame from a set of the frames that, as a group, depict a character saying the word "Hello"). As another example, in a frame of the video stream, the character is making a smiling or frowning gesture. See paragraph [0081])( As another example, the computer graphics system 402 receives user input indicating a command to simulate or synthesize a three-dimensional model of an individual layer. The computer graphics application 440 causes the computer graphics system 402 to generate one or more augmented layers by simulating or synthesizing various poses, views, or angles of the character in the individual layer. As still another example, the computer graphics system 402 receives user input to caricaturize one or more of the layers. The computer graphics application 440 causes the computer graphics system to generate one or more augmented layers by warping, distorting, or otherwise modifying the one or more layers. See paragraph [0084]); and
providing the three-dimensional model of the face of the subject to a display in a client device running an immersive reality application (and outputting the puppet animation to a presentation device. See claim 1).
Regarding claim 3, Heller teaches the computer-implemented method of claim 2, wherein identifying the audio-correlated facial feature comprises correlating the audio capture with a geometry of a lower portion of the face of the subject (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]).
Regarding claim 4, Heller teaches The computer-implemented method of claim 2, wherein identifying the expression-like facial feature of the subject comprises correlating a facial feature with a speech feature from the audio capture of the subject (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]).
Regarding claim 5, Heller teaches The computer-implemented method of claim 2, wherein identifying the expression-like facial feature of the subject comprises selecting the expression-like facial feature based at least in part upon a prior sampling of multiple facial expressions of the subject (The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034]) ( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035]).
Regarding claim 7, Heller teaches The computer-implemented method of claim 2, wherein the expression-like facial feature is associated with an upper facial feature of the subject (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]).
Regarding claim 8, Heller teaches The computer-implemented method of claim 7, wherein the upper facial feature is related to an eyebrow of the subject or an eye of the subject (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]).
Regarding claim 9, Heller teaches The computer-implemented method of claim 2, wherein the audio-correlated facial feature is associated with a shape of a lip of the subject (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]).
Regarding claim 13, Heller teaches the computer-implemented method of claim 2, wherein the three- dimensional model of the face of the subject is based at least in part upon information associated with the face of the subject with a neutral expression (neutral expression; See figure 21).
Regarding claim 14, Heller teaches A system (In some embodiments, the computing system 2700 includes a processing device 2702 that executes program code 2705, a memory device 2704 that stores various program data 2707 computed or used by operations in the program code 2705, one or more input devices 2712, and a presentation device 2714 that displays graphical content generated by executing the program code 2705. See paragraph [0128]), comprising:
one or more processors (In some embodiments, the computing system 2700 includes a processing device 2702 that executes program code 2705, a memory device 2704 that stores various program data 2707 computed or used by operations in the program code 2705, one or more input devices 2712, and a presentation device 2714 that displays graphical content generated by executing the program code 2705. See paragraph [0128]); and
a memory storing instructions which, when executed by the one or more processors (In some embodiments, the computing system 2700 includes a processing device 2702 that executes program code 2705, a memory device 2704 that stores various program data 2707 computed or used by operations in the program code 2705, one or more input devices 2712, and a presentation device 2714 that displays graphical content generated by executing the program code 2705. See paragraph [0128]), cause the system to:
identify, based at least in part upon an audio capture of a voice of a subject, an audio- correlated facial feature (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]);
identify, based at least in part upon an image capture of a portion of a face of the subject, an expression-like facial feature of the subject (As used herein, the term "viseme" is used to refer to an image that depicts a person's face as he or she speaks a particular phoneme. See paragraph [0037]) (The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035]) ( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]);
generate a three-dimensional model of the face of the subject, wherein the three-dimensional model comprises a first model feature associated with the audio-correlated facial feature and a second model feature associated with the expression-like facial feature (Thus, embodiments described herein improve computer-implemented processes for automatically creating animations of facial expressions (e.g., visemes) that can be synchronized with appropriate sounds ( e.g., phonemes). See paragraph [0039])( creating, from at least some of the video frames, a puppet animation of the gestures, wherein an animation of the viseme corresponding to the target sound or phoneme is generated from the particular video frame; and outputting the puppet animation to a presentation device. See claim 1)( In some embodiments, the content stream obtained or received by the computer graphics system 402 includes one or more frames in which the character is performing various gestures that can be used to generate an animatable puppet. As an example, in a frame, the character is making a gesture with the character's mouth that resembles a viseme for a sound, phoneme, phone, etc. For instance, in a frame of the content stream, the character is making a gesture that resembles the character saying the "H" sound ( e.g., a frame from a set of the frames that, as a group, depict a character saying the word "Hello"). As another example, in a frame of the video stream, the character is making a smiling or frowning gesture. See paragraph [0081])( As another example, the computer graphics system 402 receives user input indicating a command to simulate or synthesize a three-dimensional model of an individual layer. The computer graphics application 440 causes the computer graphics system 402 to generate one or more augmented layers by simulating or synthesizing various poses, views, or angles of the character in the individual layer. As still another example, the computer graphics system 402 receives user input to caricaturize one or more of the layers. The computer graphics application 440 causes the computer graphics system to generate one or more augmented layers by warping, distorting, or otherwise modifying the one or more layers. See paragraph [0084]); and
provide the three-dimensional model of the face of the subject to a display in a client device running an immersive reality application (and outputting the puppet animation to a presentation device. See claim 1).
Regarding claim 15, Heller teaches The system of claim 14, wherein the one or more processors further execute instructions to correlate the audio capture with a geometry of a lower portion of the face of the subject (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]).
Regarding claim 16, Heller teaches The system of claim 14, wherein the one or more processors further execute instructions to correlate a facial feature with a speech feature from the audio capture of the subject (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]).
Regarding claim 18, Heller teaches A non-transitory computer-readable media storing computer-readable instructions that, when executed by at least one processor, cause the at least one processor to execute operations (In some embodiments, the computing system 2700 includes a processing device 2702 that executes program code 2705, a memory device 2704 that stores various program data 2707 computed or used by operations in the program code 2705, one or more input devices 2712, and a presentation device 2714 that displays graphical content generated by executing the program code 2705. See paragraph [0128]) comprising:
identifying, based at least in part upon an audio capture of a voice of a subject, an audio- correlated facial feature (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]);
identifying, based at least in part upon an image capture of a portion of a face of the subject, an expression-like facial feature of the subject (As used herein, the term "viseme" is used to refer to an image that depicts a person's face as he or she speaks a particular phoneme. See paragraph [0037]) (The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035]) ( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]);
generating a three-dimensional model of the face of the subject, wherein the model comprises a first model feature associated with the audio-correlated facial feature and a second model feature associated with the expression-like facial feature (Thus, embodiments described herein improve computer-implemented processes for automatically creating animations of facial expressions (e.g., visemes) that can be synchronized with appropriate sounds ( e.g., phonemes). See paragraph [0039])( creating, from at least some of the video frames, a puppet animation of the gestures, wherein an animation of the viseme corresponding to the target sound or phoneme is generated from the particular video frame; and outputting the puppet animation to a presentation device. See claim 1)( In some embodiments, the content stream obtained or received by the computer graphics system 402 includes one or more frames in which the character is performing various gestures that can be used to generate an animatable puppet. As an example, in a frame, the character is making a gesture with the character's mouth that resembles a viseme for a sound, phoneme, phone, etc. For instance, in a frame of the content stream, the character is making a gesture that resembles the character saying the "H" sound ( e.g., a frame from a set of the frames that, as a group, depict a character saying the word "Hello"). As another example, in a frame of the video stream, the character is making a smiling or frowning gesture. See paragraph [0081])( As another example, the computer graphics system 402 receives user input indicating a command to simulate or synthesize a three-dimensional model of an individual layer. The computer graphics application 440 causes the computer graphics system 402 to generate one or more augmented layers by simulating or synthesizing various poses, views, or angles of the character in the individual layer. As still another example, the computer graphics system 402 receives user input to caricaturize one or more of the layers. The computer graphics application 440 causes the computer graphics system to generate one or more augmented layers by warping, distorting, or otherwise modifying the one or more layers. See paragraph [0084]); and
providing the three-dimensional model of the face of the subject to a display in a client device running an immersive reality application (and outputting the puppet animation to a presentation device. See claim 1).
Regarding claim 19, Heller teaches The non-transitory computer-readable media storing computer-readable instructions of claim 18 that, when executed by the at least one processor, cause the processor to execute operations further comprising: correlating the audio capture with a geometry of a lower portion of the face of the subject (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]).
Regarding claim 20, Heller teaches The non-transitory computer-readable media storing computer-readable instructions of claim 18 that, when executed by the at least one processor, cause the processor to execute operations further comprising: correlating a facial feature with a speech feature from the audio capture of the subject (As used herein, the term "phoneme" is used to refer to a sound that is a perceptually distinct unit of one or more words in a particular language. See paragraph [0038])(The following non-limiting example is provided to introduce certain embodiments. In this example, a viseme detection engine accesses video frames depicting a person performing gestures that can be used for generating a layered puppet, along with an input audio dataset that includes audio of one or more target phonemes. One or more gestures depicted in the video frames is a viseme corresponding to a target sound or phoneme. For instance, if viseme detection engine is used to capture visemes for the phonemes "Aa," "D," "Oh," and "Ee," a computing device equipped with a video camera and a microphone can prompt a user to speak the word "Adobe," and then record the user's face and voice as the users says "Adobe." See paragraph [0034])( Continuing with this example, to extract the visemes from the recorded frames, the viseme detection engine accesses a reference audio dataset with an annotation identifying a reference audio portion as having the target sound or phoneme. For instance, the reference audio dataset could include several audio portions of the phonemes "Aa," "D," "Oh," and "Ee," along with respective annotations identifying the reference audio portions as including the phonemes. The viseme detection engine applies a dynamic time warping operation, or other suitable audio analysis, to the reference audio dataset and the input audio dataset. The audio analysis allows the viseme detection engine to match a particular audio data portion from the input audio dataset to a particular annotated reference audio portion. The viseme detection engine retrieves the annotation for the matched reference audio portion ( e.g., the phoneme "Oh"), identifies a video frame corresponding to the matched input audio portion, and tags the identified video frame with the retrieved annotation. In this manner, the viseme detection engine obtains a video frame that depicts a user speaking the phoneme "Oh." A suitable computer graphics application can use the tagged video frame to create one or more frames of a corresponding puppet animation of the gestures. For instance, if the puppet animation involves a face speaking the "Oh" phoneme, the tagged "Oh" frame can be used as the basis for animating the face for that gesture. See paragraph [0035])( For example, the computer graphics application 440 causes the computer graphics system 402 to identify or extract the character's face by identifying various regions or features of the character's face. In some examples, the computer graphics application 440 causes the computer graphics system 402 to identify a location or position of the various regions or features of the character's face. As an example, the computer graphics system 402 identifies the character's features including, for example, the character's eyes, mouth, nose, eyebrow, lips, oral cavity, skin, ears, chin, etc., and a corresponding location or position of the character's identified features. For example, FIG. 21 depicts an example of identified features of a character that can be used to generate a layered animatable puppet, in accordance with one or more embodiments. In the example depicted in FIG. 21, the computer graphics application 440 causes the computer graphics system 402 to identify the character's eyebrows, lips, nose, eyes, chin, jaw line, etc. See paragraph [0101]).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 6, 17, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Heller et al. (US 2020/0160581)(Hereinafter referred to as Heller) in view of Daniel Cudeiro et al. (“Capture, Learning , and Synthesis of 3D speaking Styles”, 2019)(Hereinafter referred to as Cudeiro).
Regarding claim 6, Heller teaches The computer-implemented method of claim 2, but is silent to wherein identifying the expression-like facial feature of the subject comprises using a sampling of multiple facial expressions collected during a training session of a second subject speaking words.
Cudeiro teaches creating a generic audio driven facial animation using a plurality of speakers to train a neural net (Audio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, models, and standard evaluation metrics. To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers. We then train a neural net work on our dataset that factors identity from facial motion. The learned model, VOCA (Voice Operated Char acter Animation) takes any speech signal as input—even speech in languages other than English—and realistically animates a wide range of adult faces. Conditioning on subject labels during training allows the model to learn a variety of realistic speaking styles. See abstract).
Heller and Cudeiro teach of facial animation and Cudeiro teaches that a plurality of speakers can be used to train the model to provide realistic animation to a wide variety of adult faces, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of Heller with the training techniques of Cudeiro such that the system could provide realistic animation to a wide variety of adult faces.
Regarding claim 17, Heller teaches The system of claim 14, but is silent to wherein the one or more processors further execute instructions to select the expression-like facial feature based at least in part upon a prior sampling of multiple facial expressions of the subject.
Cudeiro teaches creating a generic audio driven facial animation using a plurality of speakers to train a neural net (Audio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, models, and standard evaluation metrics. To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers. We then train a neural net work on our dataset that factors identity from facial motion. The learned model, VOCA (Voice Operated Char acter Animation) takes any speech signal as input—even speech in languages other than English—and realistically animates a wide range of adult faces. Conditioning on subject labels during training allows the model to learn a variety of realistic speaking styles. See abstract).
Heller and Cudeiro teach of facial animation and Cudeiro teaches that a plurality of speakers can be used to train the model to provide realistic animation to a wide variety of adult faces, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of Heller with the training techniques of Cudeiro such that the system could provide realistic animation to a wide variety of adult faces.
Regarding claim 21, Heller teaches The non-transitory computer-readable media storing computer-readable instructions of claim 18, but is silent to that, when executed by the at least one processor, cause the processor to execute operations further comprising: selecting the expression-like facial feature based at least in part upon a prior sampling of multiple facial expressions of the subject.
Cudeiro teaches creating a generic audio driven facial animation using a plurality of speakers to train a neural net (Audio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, models, and standard evaluation metrics. To address this, we introduce a unique 4D face dataset with about 29 minutes of 4D scans captured at 60 fps and synchronized audio from 12 speakers. We then train a neural net work on our dataset that factors identity from facial motion. The learned model, VOCA (Voice Operated Char acter Animation) takes any speech signal as input—even speech in languages other than English—and realistically animates a wide range of adult faces. Conditioning on subject labels during training allows the model to learn a variety of realistic speaking styles. See abstract).
Heller and Cudeiro teach of facial animation and Cudeiro teaches that a plurality of speakers can be used to train the model to provide realistic animation to a wide variety of adult faces, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of Heller with the training techniques of Cudeiro such that the system could provide realistic animation to a wide variety of adult faces.
Claim(s) 10-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Heller et al. (US 2020/0160581)(Hereinafter referred to as Heller) in view of van Vuuren et al. (US 2020/0126283)(Hereinafter referred to as Vuuren).
Regarding claim 10, Heller teaches The computer-implemented method of claim 2, but is silent to wherein the three- dimensional model of the face of the subject is based at least in part upon a mesh, wherein the mesh comprises a lower portion of the face of the subject and an upper portion of the face of the subject.
Vuuren teaches the ability to define polygonal meshes in specific regions for facial features (define, on the three-dimensional model, a polygon mesh in a region of at least one facial feature; adjust parameters on the three-dimensional model, the region of the at least one facial feature corresponding to at least one facial feature in the input image; and display the three-dimensional model with the face of the input image projected onto the three-dimensional model See paragraph [0022]).
Heller and Vuuren teach of three-dimensional models and viseme and phonemes and Vuuren teaches that the three-dimensional model can be a mesh for specific regions of facial features, therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system of Heller with the region based mesh techniques of Vuuren such that the system could operationalize facial expressions without having to stitch the input image (Vuuren; the manipulation of the polygon mesh (and/or the adjustment of parameters in the 3D model) to achieve or operationalize facial expressions, the movement of facial features (e.g., eyes, mouth, etc.), etc., without need to cut or stitch the input image See paragraph [0064]).
Regarding claim 11, Heller in view of Vuuren teaches the computer-implemented method of claim 10, wherein the lower portion of the face of the subject in the mesh is based at least in part upon the audio-correlated facial feature (Vuuren; the manipulation of the polygon mesh (and/or the adjustment of parameters in the 3D model) to achieve or operationalize facial expressions, the movement of facial features (e.g., eyes, mouth, etc.), etc., without need to cut or stitch the input image See paragraph [0064])(Vuuren; Novel tools and techniques are provided for implementing three-dimensional facial modeling and visual speech synthesis. See abstract).
Regarding claim 12, Heller in view of Vuuren teaches The computer-implemented method of claim 10, wherein the upper portion of the face of the subject in the mesh is based at least in part upon the expression-like facial feature of the subject (Vuuren; the manipulation of the polygon mesh (and/or the adjustment of parameters in the 3D model) to achieve or operationalize facial expressions, the movement of facial features (e.g., eyes, mouth, etc.), etc., without need to cut or stitch the input image See paragraph [0064])(Vuuren; Novel tools and techniques are provided for implementing three-dimensional facial modeling and visual speech synthesis. See abstract).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Li et al. (US 2017/0243387)(Hereinafter referred to as Li), generally relates to creating a 3D model using Phonemes and Visemes (See paragraph [0060]).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS R WILSON whose telephone number is (571)272-0936. The examiner can normally be reached M-F 7:30-5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (572)-272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICHOLAS R WILSON/Primary Examiner, Art Unit 2611
Prosecution Insights