Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see remarks, filed 13 April 2026, with respect to the rejection(s) of claims 1-3, 5-9, 18 and 36 and 10, 11, 27 and 38 under 35 USC 102 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground of rejection is made in view of Li et al. (US 11,417,041 B2).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-9, 18 and 36 are rejected under 35 U.S.C. 103 as being unpatentable over Kang et al. (US 20220044463 A1) in view of Li et al. (US 11,417,041 B2).
Regarding claim 1, Kang et al. discloses a mouth shape-based method for generating a
face image (to enable, according to the expression parameter, an animation character to make
an expression corresponding to the first speech, para. 0009), comprising:
acquiring audio data to be recognized (obtaining a first speech, the first speech comprising a plurality of speech frames, para. 0025) and a preset face image (For a set animation character whose expression (for example, a mouth shape) may be adjusted by adjusting the expression parameter, the animation character may be adjusted by using the determined expression parameter, to generate an animation character responding to the speech, para. 0028);
determining an audio feature of the audio data to be recognized (a first speech including
a plurality of speech frames is obtained, linguistics information corresponding to a speech frame
in the first speech may be determined, para. 0030);
wherein the audio feature comprises a speech speed feature and a semantic feature (he
PPG (that is, the linguistics information) is spliced with a pre-annotated sentiment vector to
obtain a final feature, so that the expression parameter corresponding to the speech frame in
the first speech is determined according to the PPG and the sentiment vector corresponding to
the first speech, para. 0063); and
performing, according to the speech speed feature and the semantic feature, processing
on the preset face image, to generate a face image having a mouth shape (a set animation
character whose expression (for example, a mouth shape) may be adjusted by adjusting the
expression parameter, the animation character may be adjusted by using the determined
expression parameter, to generate an animation character corresponding to the speech, para.
0028).
Kang et al. does not expressly disclose the speech speed feature is used for representing a changing speed of phonemes in the audio data to be recognized, and the semantic feature is used for representing meanings expressed by the phonemes in the audio data to be recognized.
Li et al. teaches that speaking style may include speech intensity, pitch, emphasis, syllable duration, and/or other characteristics, col. 7, lines 29-33 and speech content may include structure (e.g., phonemes, words, sentences) and/or semantic meaning.
Kang et al. in view of Li et al. are analogous art because they are from the similar problem solving area of mouth shape modulation. At the time of the invention, it would have been obvious to a person of ordinary skill in the art to add the syllable duration speech of Li et al. to the modulation of Kang et al. in order to obtain speech speed features. The motivation for doing so would be to customize speech modulation.
Regarding claim 2, Kang et al. discloses the method according to claim 1, wherein the
determining the audio feature of the audio data to be recognized comprises:
determining, according to a preset first feature extraction model, a speech speed feature
of the audio data to be recognized (the sentiment is represented by using the sentiment vector,
para. 0064);
wherein the first feature extraction model is used for extracting the speech speed feature
from the audio data to be recognized (Speech2Face system includes four parts. A first part is to
train an ASR model for PPG extraction, para. 0053); and
determining, according to a preset second feature extraction model, a semantic feature
of the audio data to be recognized (the sentiment is represented by using the sentiment vector,
para. 0064);
wherein the second feature extraction model is used for extracting the semantic feature
from the audio data to be recognized (after a PPG of the speech frame is obtained by using the
trained ASR model, the PPG (that is, the linguistics information) is spliced with a pre-annotated
sentiment vector to obtain a final feature, so that the expression parameter corresponding to the
speech frame in the first speech is determined according to the PPG and the sentiment vector
corresponding to the first speech, para. 0063).
Regarding claim 3, Kang et al. discloses the method according to claim 2, wherein the
determining, according to the presetfirst feature extraction model, the speech speed feature of
the audio data to be recognized comprises:
inputting the audio data to be recognized into the preset first feature extraction model for
feature extraction, to obtain a phonetic posterioriorgram feature of the audio data to be
recognized (linguistics information being used for identifying a distribution possibility that the
speech frame in the first speech pertains to phonemes, para. 0008);
wherein the phonetic posterioriorgram feature represents information about a phoneme
category of the audio data to be recognized (a combination of two or more of a phonetic
posterior gram (PPG), para. 0052); and
determining, according to the phonetic posterioriorgram feature of the audio data to be
recognized, the speech speed feature of the audio data to be recognized (Examiner articulates
that (PPG) is a time-varying representation of a speech signal that shows the probability
distribution over different phonemes (speech sounds) at each moment in time.)
Regarding claim 5, Kang et al. discloses the method according to claim 2, wherein the
determining, according to the preset second feature extraction model, the semantic feature of
the audio data to be recognized comprises:
inputting the audio data to be recognized into the preset second feature extraction model
for feature extraction, to obtain output semantic feature of the audio data to be recognized (a
vector having 218 dimensions is obtained, and is spliced with a 4-dimensional sentiment vector
of the first speech to obtain a feature vector having 222 dimensions, which is subsequently used
as an input to the neural network mapping model, para. 0064)
Regarding claim 6, Kang et al. discloses the method according to claim 1, wherein the
performing, according to the speech speed feature and the semantic feature, the processing on
the preset face image, to generate the face image having the mouth shape comprises: inputting
the speech speed feature and the semantic feature into a preset model for determining a mouth
shape of a face for processing (a feature vector having 222 dimensions, which is subsequently
used as an input to the neural network mapping model, para. 0064), and generating, according
to a result obtained from the processing and the preset face image, the face image having the
mouth shape (to drive the animation character to make an expression (a mouth shape), para.
0067).
Regarding claim 7, Kang et al. discloses the method according to claim 6, wherein the
inputting the speech speed feature and the semantic feature into the preset model for
determining the mouth shape of the face for the processing, and the generating, according to
the result obtained from the processing and the preset face image, the face image having the
mouth shape comprise:
performing, based on the preset model for determining the mouth shape of the face,
splicing processing on the speech speed feature and the semantic feature, to obtain a spliced
feature of the audio data to be recognized; wherein the spliced feature represents the speech
speed feature and the semantic feature (the PPG (that is, the linguistics information) is spliced
with a pre-annotated sentiment vector to obtain a final feature, so that the expression parameter
corresponding to the speech frame in the first speech is determined according to the PPG and
the sentiment vector corresponding to the first speech, para. 0063);
performing, according to a convolutional layer in the preset model for determining the
mouth shape of the face, feature extraction on the spliced feature, to obtain a face driving
parameter (determined by using the neural network mapping model, manners of determining the
speech frame set are different and quantities of speech frames in the speech frame set are
different according to different neural network mapping models used, para. 0071);
wherein the face driving parameter is used for representing a parameter required to
drive a mouth shape change in a face image; and performing, according to the face driving
parameter, image rendering on the preset face image, to generate the face image having the
mouth shape ( an expression parameter may be determined according to the first speech, to
drive the animation character to make an expression (a mouth shape), para. 0067).
Regarding claim 8, Kang et al. discloses the method according to claim 7, wherein the
face driving parameter is a weight parameter of a blend shape (an expression parameter may
be determined according to linguistics information, to accurately drive an animation character to
make an expression corresponding to the first speech, para. 0030); and
the performing, according to the face driving parameter, the image rendering on the
preset face image, to generate the face image having the mouth shape comprises:
determining, according to the weight parameter of the blend shape, facial three-
dimensional mesh data corresponding to the preset face image (a designed 3D animation
character to make an expression corresponding to the first speech (for example, S304), para.
0053);
wherein the facial three-dimensional mesh data is data representing a three-dimensional
mesh model of a facial surface on a face image (see figure 2); and performing, according to the
facial three-dimensional mesh data, image rendering on the preset face image, to generate the
face image having the mouth shape (a phoneme is a minimum phonetic unit obtained through
division according to a natural attribute of a speech, analysis is performed according to a
pronunciation action in a syllable, and an action (for example, a mouth shape) forms a
phoneme, para. 0045).
Regarding claim 9, Kang et al. discloses the method according to any one of claims 1 to
8, further comprising: if it is determined that a value represented by the speech speed feature of
the audio data to be recognized is less than a preset speech speed threshold value, performing,
according to the semantic feature, processing on the preset face image, to generate the face
image having the mouth shape (he expressions may include a facial expression and a body
posture expression. The facial expression may include, for example, a mouth shape, para.
0060).
Claim 18, a mouth shape-based apparatus claim, is rejected for the same reason as
claim 1.
Claim 36, a non-transitory computer readable storage medium claim, is rejected for the
same reason as claim 1.
Claims 10, 11, 27 and 38 are rejected under 35 U.S.C. 103 as being unpatentable over Chae et al. (US 20220358703 A1) in view of Li et al. (US 11,417,041 B2).
Regarding claim 10, Chae et al. discloses a method for training a model for determining
a mouth shape of a face, comprising:
acquiring image data to be trained and a preset face image (the person background
image may be an image used during a training process, para. 0045);
wherein the image data to be trained comprises audio data to be trained and a face
image to be trained, and the face image to be trained has a mouth shape corresponding to the
audio data to be trained (extract a voice feature vector from the speech audio signal, para.
0008);
determining an audio feature of the audio data to be trained (extract a voice feature
vector, para. 0008);
wherein the audio feature comprises a speech speed feature and a semantic feature;
performing, according to the speech speed feature, the semantic feature, and the preset face
image, training on an initial model for determining a mouth shape of a face, and obtaining a face
image having a mouth shape (adjust learning parameters (e.g., the loss function or the Softmax
function) so that the generated speech video (i.e., a video in which the portion related to the
speech are reconstructed through the audio part) is similar to the original speech video, para.
0044); and if the face image having the mouth shape and the face image to be trained are
consistent, determining that a trained model for determining a mouth shape of a face is obtained
(decoder 108 may compare the generated speech video with the original speech video (i.e., an
answer value) and thus adjust learning parameters (e.g., the loss function or the Softmax
function), para. 0044).
Chae et al. does not expressly disclose the speech speed feature is used for representing a changing speed of phonemes in the audio data to be trained, and the semantic feature is used for representing meanings expressed by the phonemes in the audio data to be trained.
Li et al. teaches that speaking style may include speech intensity, pitch, emphasis, syllable duration, and/or other characteristics, col. 7, lines 29-33.
Chae et al. in view of Li et al. are analogous art because they are from the similar problem solving area of mouth shape modulation. At the time of the invention, it would have been obvious to a person of ordinary skill in the art to add the syllable duration speech of Li et al. to the modulation of Chae et al. in order to obtain speech speed features. The motivation for doing so would be to customize speech modulation.
Regarding claim 11, Chae et al. discloses the method according to claim 10, wherein the
determining the audio feature of the audio data to be trained comprises: determining, according
to a preset first feature extraction model, a speech speed feature of the audio data to be trained;
wherein the first feature extraction model is used for extracting the speech speed feature from
the audio data to be trained; and determining, according to a preset second feature extraction
(second encoder 104 is a machine learning model that is trained to extract a voice feature
vector using a speech audio signal as an input. Here, the speech audio signal corresponds to
an audio part of the person background image (i.e., an image in which a person is speaking)
input to the first encoder 102. In other words, in a video in which a person speaks (hereinafter,
referred to as a "speech video"), a video part thereof may be input to the first encoder 102, and
an audio part thereof may be input to the second encoder 104. The second encoder 104 may
include at least one convolutional layer, para. 0040) model, a semantic feature of the audio data
to be trained; wherein the second feature extraction model is used for extracting the semantic
feature from the audio data to be trained.
Claim 27, an apparatus claim, is rejected for the same reason as claim 10.
Claim 38, a non-transitory computer readable storage medium claim, is rejected for the
same reason as claim 10.
Allowable Subject Matter
Claims 4, 12, 13 and 15-17 are objected to as being dependent upon a rejected base
claim, but would be allowable if rewritten in independent form including all of the limitations of
the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THOMAS J LETT whose telephone number is (571)272-7464. The examiner can normally be reached Mon-Fri 9-6 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571) 272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THOMAS J LETT/Primary Examiner, Art Unit 2611
Prosecution Insights