Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claims 1-20 were submitted. Claims 1, 14, and 20 are independent and are all method Claims. Claims 13 and 14-20 are directed to visual TTS and were withdrawn from consideration by an election on 12/18/2025. Claims 1-11 and 13 remain pending and under examination of which Claim 1 is independent. Claims 1 and 13 are amended.
This Application was published as U.S. 20240105160.
Apparent priority: 8 June 2021 (earlier of two KR priorities).
Applicant’s amendments and arguments are considered but are either unpersuasive or moot in view of the new grounds of rejection that, if presented, were necessitated by the amendments to the Claims.
This action is Final.
Response to Amendments and Arguments
Applicant’s arguments are directed to the amended language and are moot in view of the modified grounds of rejection.
Claim 1 is amended as follows:
1. A method for generating a synthesis voice for text, the method being performed by one or more processors and comprising:
acquiring a text-to-speech synthesis model trained to generate a synthesis voice for a training text, wherein the text-to-speech synthesis model is configured to generate and output the synthesis voice based on reference voice data and a training style tag represented by natural language;
receiving a target text;
acquiring a style tag represented by natural language; and
inputting the style tag and the target text into the text-to-speech synthesis model and acquiring a synthesis voice for the target text, the synthesis voice generated and output by the text-to-speech synthesis model, wherein the synthesis voice comprises voice style features related to the style tag.
Claim 13 is amended as follows:
13. The method according to claim 1, where the style tag is input through an API call, and wherein the receiving of the target text is via the API call.
Applicant argues that Bakis does not teach a “TTS analysis model” and does not include the word “model” and therefore does not teach the “TTS speech synthesis model” of the Claim. Response 11-12.
Applicant next argues that Shekhar mere teaches a “speech neural network” and contends that such “is not reasonably equivalent to a TTS speech synthesis model.” Response 12-13.
In Reply, f
First, reliance on a single word is misplaced and when a term or phrase is of significant to the invention and carries a special meaning, such meaning needs to be claimed with particularity inside the Claim language. Thus, if a particular meaning is intended by “model” it needs to be placed inside the Claim and reference made to support in the Specification.
Second, absent a specific definition, a claim term/phrase is accorded its broadest reasonable interpretation in view of the Specification and the broadest reasonable interpretation of a “TTS model” includes a “TTS engine” or a “TTS neural network.” The following are examples of support in the Specification all of which are consistent with the interpretation accorded by the Examiner. The “TTS model” of the Claim receives text and generates speech as do the engine of Bakis and the NN of Shekhar.
“[0057] … For example, the synthetic speech generation system may be configured to input one or more sentences and speech style characteristics into an artificial neural network text-to-speech synthesis model and generate outputted speech data for the one or more sentences which reflects the speech style characteristics. Such a synthetic speech generation system may be executed by any computing device such as a user terminal or a system accessible from the user terminal.” “[0064] … According to an embodiment, the synthetic speech generation system may input one or more sentences and speech style characteristics into the artificial neural network text-to-speech synthesis model to generate outputted speech data reflecting the speech style characteristics and provide it through the user interface. The synthetic speech may be generated based on the outputted speech data.”
The “TTS model” of the instant Application, under its BRI, is taught by Bakis or Shekhar. Because the later dependent Claims specify the model to be a NN model, Shekhar teaches the Claims that specify the model as a NN model.
Additionally, any type of TTS, whether you call it an engine or a model or a system or by another name, almost inherently includes an acoustic model, and optionally a language model, to convert the text to phonemes and to sound. That is how TTS works with the exception of very old concatenative TTS systems that concatenated pieces of audio instead of generating the audio/speech. Even concatenation, which has long been defunct, requires its own model. TTS cannot operate without at least an acoustic model and often also includes a language model. Thus, emphasis on the word “model” is misplaced. See the Conclusion section for references added that discuss the inside of a TTS and show that any TTS requires a model and model/system/engine are the same absent some specific definition in the claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3 and 7-10 are rejected under 35 U.S.C. 103 as being unpatentable over Bakis (U.S. 20050096909) in view of Shekhar (U.S. 20220028367).
Regarding Claim 1, Bakis teaches:
1. A method for generating a synthesis voice for text, the method being performed by one or more processors and [Bakis is titled: “System and Method for Expressive Tex-to-Speech.” Figure 5 shows the hardware including the “Processor 290.”]
comprising:
acquiring a text-to-speech synthesis model trained to generate a synthesis voice for a training text, [Bakis, Figure 3, “TTS Developer Device 158” provides the TTS model to the “TTS Device 154.”] wherein the text-to-speech synthesis model is configured to generate and output the synthesis voice [Bakis, Figure 3, “Text-to-speech (TTS) device 154” and Figure 4, “text-to-speech (TTS) engine 280” as part of the TTS Device 154. Figure 5, TTS Engine 280.] based on reference voice data and a training style tag represented by natural language; [Bakis, Figure 3, the “TTS Designer Device 152” provides the “Speech Style Sheet 110” to the “TTS Device 154.”]
receiving a target text; [Bakis, Figure 1, “identify text to convert to speech 102.”]
acquiring a style tag represented by natural language; and [Bakis, Figure 1, “select one or more speech style sheets defining desired speech characteristics 104.”]
inputting the style tag and the target text into the text-to-speech synthesis model and acquiring a synthesis voice for the target text, the synthesis voice generated and output by the text-to-speech synthesis model, [Bakis, Figures 3 and 5, output of the synthesized voice from the speaker at 160 and Figure 4, output of the voice from the IVR device 160.] wherein the synthesis voice comprises reflecting voice style features related to the style tag. [Bakis, Figure 3 shows the input of the “speech style sheet 110” and the “user interface device 156” from which the text is obtained to the “TTS Device 154.” Figure 1, “Associated the text with selected speech style sheets 106” and “convert the text to expressive speech using low level markups defined by the selected speech style sheets 108.” Figure 3 shows the speaker 160 for output of the synthesized voice.]
Bakis is an early reference cited to indicate that the generation of expressive/emotive speech has been well-known. Bakis does not teach the training of a model or the use of trained model and the style sheets are input.
Shekhar teaches:
acquiring a text-to-speech synthesis model trained to generate a synthesis voice for a training text, [Shekhar, Figure 5, “trained expressive speech neural network 514.”] wherein the text-to-speech synthesis model is configured to generate and output the synthesis voice [Shekhar, Figures 1 and 2, “TTS system 104.”] based on reference voice data and a training style tag represented by natural language; [Shekhar, Figure 5, “training text 502” leading to “predicted context-based speech map 506” and “ground truth 510”/ “training style tag.” “[0099] The expressive audio generation system 106 can utilize the loss function 508 to determine the loss (i.e., error) resulting from the expressive speech neural network 504 by comparing the predicted context-based speech map 506 with a ground truth 510 (e.g., a ground truth context-based speech map)….”]
receiving a target text; [Shekhar, Figures 2, 3A and 3B show inputs of “Input Text 202/302/306” and so do Figures 4a and 4B.]
acquiring a style tag represented by natural language; and [Shekhar, Figure 2, the “expressive audio generation system 106” generates the “context-based speech map 206” / “style tag” of the Claim.]
inputting the style tag and the target text into the text-to-speech synthesis model and acquiring a synthesis voice for the target text, the synthesis voice generated and output by the text-to-speech synthesis model, wherein the synthesis voice comprises [Shekhar, Figure 1, “expressive audio 208.” Figure 6, “expressive audio 606.” Figure 10, “generating expressive audio 1010.”] voice style features related to the style tag. [Shekhar, Figure 2, “Expressive Audio 208” is generated from the “context-based speech map 206” / “style tag. “[0064] As further illustrated in FIG. 2, the expressive audio generation system 106 can generate expressive audio 208 for the input text. In particular, the expressive audio generation system 106 can generate the expressive audio 208 using the context-based speech map 206. Accordingly, the expressive audio 208 can incorporate the one or more style features associated with the input text 202.”]
Bakis and Shekhar pertain to generation of emotive and expressive speech and it would have been obvious to combine the model training aspects of the Shekhar which is a recent reference with the older method of Bakis to bring the TTS approach up to date with modern methods. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Regarding Claim 2, Bakis teaches:
2. The method according to claim 1, wherein the acquiring the style tag includes:
providing a user interface to input the style tag; and [Bakis, Figures 3 and 4 show the input of the “speech style sheet 110” to the “TTS Device/Engine 154/280” through the user or developer interface devices which may be the same thing. “[0051] … In some embodiments, the TTS developer device 158 and the user interface device 156 may be or include the same device….” ]
acquiring at least one style tag represented in natural language through the user interface. [Bakis, Figure 2 shows the style sheet 110 including different voice styles each of which correspond to a gender, volume, tone, pitch, timbre, timing, prosody, and even foreign accent and are categorized into Happy and Serious for example. “[0057] … The TTS developer may then use the IVR developer device 158 such as a PC, to access a user interface device 156 such as a GUI. An example of such a user interface device 156 may be, for example, a software-implemented browser application. Using the user interface device 156, the TTS developer may access the style sheets 110 created by the TTS designer….” “[0021] As used herein, the term "voice style" generally refers to a manner and/or style of speech. Voice styles may define either and/or both of a low level markup and a high level markup associated with a particular style or manner of speaking. For example, a voice style may be "happy", "annoyed", "playful", "formal", "Engineering" or "hoarse". Some voice styles (like "happy", for example) may define expressive styles or manners of speech such as how a "happy" voice sounds. In some embodiments, a voice style such as "happy" may indicate that text to be converted to speech be preceded (or succeeded) by a particular word or phrase. Other voice styles (like "Engineering", for example) may define one or more pronunciation rules associated with a style, manner, or category of speech. The low level markup defined by a particular voice style may correspond and/or be related or associated with a high level markup identifying, representing, and/or indicating the particular voice style.” Markup refers to eXtensible Markup Language (XML) and includes tags: “[0019] As used herein, the term "text" generally refers to any text, character, symbol, other visual indicator, or any combination thereof. For example, text may be or include structured information such as extensible markup language (XML) or other markup, tags, and/or programming code. In some embodiments for example, text may include XML markup defining airline flight characteristics to be spoken in an interactive voice response (IVR) system.”]
Regarding Claim 3, Bakis teaches:
3. The method according to claim 2, wherein the acquiring the style tag includes:
outputting a style tag recommendation list including a plurality of candidate style tags represented by natural language to the user interface; and [Bakis, “Systems and methods are provided for expressive text-to-speech which include identifying text to convert to speech, selecting a speech style sheet from a set of available speech style sheets, the speech style sheet defining desired speech characteristics, marking the text to associate the text with the selected speech style sheet, and converting the text to speech having the desired speech characteristics by applying a low level markup associated with the speech style sheet.” Abstract. Figure 1, 104 is the selection step which implies the recommendation/suggestion/provision of a set of style sheets to select from. Note that the Claim provides no criterion for the recommendation and this type of specificity is provided in Claim 4 for which a different reference is cited.]
acquiring at least one candidate style tag selected from the style tag recommendation list as the style tag for the target text. [Bakis, Figure 1, 106, “[0032] Processing continues at 106 where the developer marks the selected and/or identified text to associate it with the speech style sheet. This may further include associating the text with one or more voice styles and one or more voice-types associated with the speech style sheet. For example, the developer may use a mouse or other pointing device to select a toolbar button associated with a voice style associated with the speech style sheet. Continuing the example introduced above, the selected style sheet labeled "Aviation" may contain definitions of several voice styles (labeled "formal" and "informal"). Further, each voice style may be associated with the voice-types "Female" and "Male". When the speech style sheet labeled "Aviation" is loaded and/or selected, a pull-down list of the available voice-types, and toolbar buttons associated with the two available voice styles appear. The developer may choose (from the pull-down list) the voice-type labeled "Female" for example. The selected text will then be associated with the voice-type labeled "Female", and will thus be spoken in a female voice in accordance with the low level markup defined by the voice-type. The developer may also select the toolbar button "formal" to associate the selected text with the voice style labeled "formal". The selected text would then be spoken in accordance with the combination of low level markups defined by the respective voice style and voice-type selected by the developer (a formal female voice).”]
Regarding Claim 7, Bakis teaches:
7. The method according to claim 1, wherein the acquiring the style tag includes:
receiving a selection for a preset; and [Bakis teaches that a number of characteristics together correspond to a “speech style sheet”/preset like “aviation” in [0031] and in [0032] provided above and the user/developer can select “aviation” from the menu and all the collective voice-styles and voice-types associated with this preset style sheet come up. (See [0179] of the instant Application for a definition of “preset” which corresponds to the teachings of Bakis.)]
acquiring a style tag included in the preset as the style tag for the target text. [Bakis, [0033]-[0034]] and Figure 1, 106 and 108 “[0035] According to some embodiments, the style sheet selected may contain definitions of speech properties and/or other speech rules including pronunciation rules. In the current example for instance, the speech style sheet "Aviation" may contain rules defining how certain characters, words, and/or phrases should be pronounced to comply with speech relating to the category or field of aviation. …”]
Regarding Claim 8, Bakis teaches:
8. The method according to claim 1, wherein the text-to-speech synthesis model is configured to generate the synthesis voice for the target text reflecting the voice style features, based on features of reference voice data related to the style tag. [Bakis, Figure 2 showing the characteristics of the voice styles and voice fonts that are selected. Figure 1, 108. The paragraphs provided above generate speech based on the features that are specified in the style sheet and the XML tag of the style sheet. See Figure 2 for all the features such as volume, tone, timbre, pitch, timing, and prosody that are modified based on the selected voice.]
Regarding Claim 9, Bakis is not a neural network era reference and Shekhar uses neural networks and therefore teaches the uses of embeddings
Shekharteaches:
9. The method according to claim 1, wherein the text-to-speech synthesis model is configured to acquire embedding features for the style tag, and generate the synthesis voice for the target text reflecting the voice style features based on the acquired embedding features. [Shekhar, Figures 3A and 3B teach the generation of the “contextual word embeddings 304/310” which include the style tags/tokens and in Figures 4A, 4B, and 5 are placed into a context-based speech map.]
Rationale as provided for Claim 1 because Shekhar was combined for teaching the use of an NN which involves formation of embeddings.
Regarding Claim 10, Bakis is not training a neural network and does not mention a loss function.
Shekhar teaches:
10. The method according to claim 1, wherein the text-to-speech synthesis model is trained to minimize a loss between a first style feature extracted from the voice data and a second style feature extracted from the training style tag. [Shekhar, Figure 5, training of a neural network model involves minimizing/optimizing a loss function. “[0099] The expressive audio generation system 106 can utilize the loss function 508 to determine the loss (i.e., error) resulting from the expressive speech neural network 504 by comparing the predicted context-based speech map 506 with a ground truth 510 (e.g., a ground truth context-based speech map). The expressive audio generation system 106 can back propagate the determined loss to the expressive speech neural network 504 (as shown by the dashed line 512) to optimize the model by updating its parameters/weights. In particular, the expressive audio generation system 106 can back propagate the determined loss to each channel of the expressive speech neural network 504 (e.g., the character-level channel, the word-level channel, and, in some instances, the speaker identification channel) as well as the decoder of the expressive speech neural network 504 to update the respective parameters/weights of that channel. In some embodiments, the expressive audio generation system 106 back propagates the determined loss to each component of the expressive speech neural network (e.g., the character-level encoder, the word-level encoder, the location-sensitive attention mechanism, etc.) update the parameters/weights of that component individually. Consequently, with each iteration of training, the expressive audio generation system 106 gradually improves the accuracy with which the expressive speech neural network 504 can generate context-based speech maps for input texts (e.g., by lowering the resulting loss value). As shown, the expressive audio generation system 106 can thus generate the trained expressive speech neural network 514.”]
Rationale as provided for Claim 1 because Shekhar was combined for teaching the training of a NN.
Claims 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over Bakis and Shekhar in view of Aguillar (U.S. 20210104245) and further in view of Xiu (U.S. 20220059122).
Regarding Claim 4, Bakis does not teach identification of mood from text.
Shekhar teaches:
4. The method according to claim 3, wherein the outputting the style tag recommendation list to the user interface includes:
identifying at least one of emotion or mood represented in the target text; [Shekhar teaches detection of emotion from text and based on the words of the text. Figure 2, “input text 202” to “expressive audio generation system 106.” “[0021] As mentioned above, in one or more embodiments, the expressive audio generation system utilizes a neural network having a multi-channel architecture—such as an expressive speech neural network—to analyze an input text. In particular, the expressive audio generation system can utilize the channels of the expressive speech neural network to analyze the character-level information of an input text separately from the word-level information of an input text. ….”]
determining the plurality of candidate style tags related to at least one of the identified emotion or mood; and [Shekhar teaches that the “style tokens” that are learned from the words and characters of the text correspond to Emotion among other things. “[0005] … The disclosed systems further utilize a contextual word-level style predictor of the deep learning model to separately encode contextual information of the input text. Specifically, the disclosed systems can use contextual word embeddings to learn style tokens that correspond to various style features (e.g., emotion, pitch, modulation, etc.)….” The style tokens / style tags correspond to different features. This limitation’s language is taught by Shekhar.]
outputting the style tag recommendation list including the determined plurality of candidate style tags to the user interface. [Shekhar outputs the style tokens/ style tags by incorporating them into the synthesized speech and does not output a list to the user interface. The steps are done internally. However, Figures 2, 4A and 4B show a “context-based speech map 206/430/472” which includes the emotion/style/tokens/tags and Figure 5 shows that during the training a “predicted context-based speech map 506” is generated that teaches the “recommendation list” of the Claim and is modified by the training process by the “ground truth 510” iteratively. “[0098] As shown in FIG. 5, the expressive audio generation system 106 implements the training by providing a training text 502 to the expressive speech neural network 504. The training text 502 includes a plurality of words and a plurality of associated characters. Further, as shown, the expressive audio generation system 106 utilizes the expressive speech neural network 504 to generate a predicted context-based speech map 506 based on the training text 502. ...” “[0099] The expressive audio generation system 106 can utilize the loss function 508 to determine the loss (i.e., error) resulting from the expressive speech neural network 504 by comparing the predicted context-based speech map 506 with a ground truth 510 (e.g., a ground truth context-based speech map). … Consequently, with each iteration of training, the expressive audio generation system 106 gradually improves the accuracy with which the expressive speech neural network 504 can generate context-based speech maps for input texts (e.g., by lowering the resulting loss value). As shown, the expressive audio generation system 106 can thus generate the trained expressive speech neural network 514.” “[0064] As further illustrated in FIG. 2, the expressive audio generation system 106 can generate expressive audio 208 for the input text. In particular, the expressive audio generation system 106 can generate the expressive audio 208 using the context-based speech map 206. Accordingly, the expressive audio 208 can incorporate the one or more style features associated with the input text 202.”]
Figures 13 and 14 of the instant Application show what the Applicant intends by output of a tag recommendation list.
Aguillar teaches:
…
determining the plurality of candidate style tags related to at least one of the identified emotion or mood; and [[Aguillar detects sentiment from audio (not text) or a mixture of text and audio (Figure 7) and generates but does output an n-best list of sentiment categories in Figure 5, 530, 540. “[0111] The attention model 520 may output a score 530 indicating a likelihood of the utterance corresponding to a sentiment category 540. The attention model 520 may output model output data including an indicator of a sentiment or a N-best list of scores and corresponding sentiment category. The sentiment detection component 275 may predict from multiple sentiment categories, including but not limited to, happiness, sadness, anger and neutral. … In an example embodiment, the sentiment categories may be broad such as positive, neutral, and negative or may be more precise such as angry, happy, distressed, surprised, disgust, or the like.” “[0138] The utterance attention component 718 may output score 720 indicating a sentiment category 740 for the user audio data 512….”]
Bakis/Shekhar and Aguillar pertain to emotion detection and it would have been obvious to combine the method of Aguillar which generates an n-best list of associated emotions with the system of combination in order to have the benefits of n-best lists that are common in other types of NLP recognition processing.This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Aguillar does not teach output of the generated n-best to the user interface.
Xiu teaches:
outputting the style tag recommendation list including the determined plurality of candidate style tags to the user interface. [Xiu, Figure 6 shows the “generating an emotion record 622” and “displaying to the user or third party 626” and Figure 6 shows the display of the emotion record on the screen of a phone. The emotions are the style tags or can be converted to style tags.]
Bakis/Shekhar/Aguillar and Xiu pertain to emotion detection and it would have been obvious to combine the method of Xiu which displays a record of different emotions at different times to the user or third parties with the system of combination that generates an n-best list of associated emotions in order to display the n-best list of the emotions. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Regarding Claim 5, Bakis does not teach identification of mood from text.
Shekhar teaches:
5. The method according to claim 3, wherein the outputting the style tag recommendation list to the user interface includes:
determining the plurality of candidate style tags based on a style tag usage pattern of the user; and [Shekhar, Figure 4B, includes a speaker identification portion (466, 456, 464) and adds the speaker identity (profile) to the data that is used by the decoder 470 to come up with the “context-based speech map 472” that includes the style tags/tokens. “[0092] Further, as shown in FIG. 4B, the speaker identification channel 456 can generate a speaker identity feature vector 464 based on speaker-based input 466. Indeed, in one or more embodiments, the expressive audio generation system 106 provides the speaker-based input 466 to the expressive speech neural network 450 along with the input text 458 to generate expressive audio that captures the style (e.g., sound, tonality, etc.) of a particular speaker. As mentioned above, the speaker-based input 466 can identify a particular speaker from among a plurality of available speakers, include details describing the speaker (e.g., age, gender, etc.), or include a sample of speech to be mimicked. In one or more embodiments, the speaker-based input 466 can include, or be associated with, a particular language or accent to be incorporated into the resulting expressive audio. I….”]
outputting the style tag recommendation list including the determined plurality of candidate style tags to the user interface.
Shekhar uses the tags/tokens to generate expressive speech and does not output to the user interface. It also does not teach generating a list associated with the same sentence.
Aguillar as applied to Claim 4 above teaches generating an n-best list of emotions but does not teach display of this list.
Xiu as applied to Claim 4 above, and in Figures 6 and 7, teaches display of a list of emotions pertaining to different events.
The combination teaches the display of the n-best list to a user. Once the n-best list is generated, displaying it would be trivial except if the Claim were to do more with the displayed list which it is currently not doing. Further specifying what happens after the n-best list is shown and how it is shown would make this combination inappropriate.
Rationale as provided for Claim 4.
PNG
media_image1.png
574
386
media_image1.png
Greyscale
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Bakis and Shekhar in view of Morimoto (U.S. 20170323013).
This Claim pertains to the embodiment shown in Figure 14 of the instant Application:
PNG
media_image2.png
238
584
media_image2.png
Greyscale
[0171] FIG. 14 is a diagram illustrating a style tag recommendation list output based on the user-input information. As illustrated in FIG. 14, in response to detecting a partial input (i.e., “soft”) 1410 for the style tag of the second sentence, a processor of the system for generating a synthesis voice may automatically complete at least one candidate style tag that begins with or includes “soft”, and output a style tag recommendation list 1420 including at least one automatically completed candidate style tag to the user interface. When a selection input for one of the style tag recommendation list 1420 is received from the user, a style tag corresponding to the selection input may be acquired as a style tag of the second sentence.
Regarding Claim 6, Bakis does not teach using autocomplete to complete a partial indication of emotion. Neither does Shekhar. Bakis and Shekhar however teach the output of the generated emotions.
Morimoto teaches:
6. The method according to claim 2, wherein the providing the user interface includes:
detecting a partial input of natural language related to the style tag; [Morimoto, Figure 1, “emotion extraction unit 135” and “emotion evaluation unit 136.” Figure 4, “input new data S401” to “extract emotional expressions from the extracted data elements S403.” “[0109] When a voice itself is to be analyzed, the voice is divided into partial voices of a specified length and the partial voices are used as objects to be analyzed. For example, if a voice stating “this film is interesting” is obtained, the data evaluation system 100 can extract the partial voice “interesting” ….” This system can operate based on partial input.]
automatically completing at least one candidate style tag including the partial input; and [Morimoto, “[0109] When a voice itself is to be analyzed, the voice is divided into partial voices of a specified length and the partial voices are used as objects to be analyzed. For example, if a voice stating “this film is interesting” is obtained, the data evaluation system 100 can extract the partial voice “interesting” from the relevant voice and generate its emotion marker on the basis of the evaluation result of that partial voice….” The emotion of “interesting” is deduced from a partial voice input.]
outputting the automatically completed at least one candidate style tag through the user interface. [Morimoto, Figure 1, “presentation unit 139” connected to “display unit 150.” Figure 2 showing the user interface 200.]
Bakis/Shekhar and Morimoto pertain to detection of emotion from voice and text and it would have been obvious to modify the system of combination to include the detection of emotion from a partial input of the voice or indication of emotion to cover the situations where the input is incomplete but can be guessed from the partial input. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Bakis and Shekhar in view of Shakeri (U.S. 11790884).
Regarding Claim 11, Bakis, Figure 2, shows “prosody” as one feature 122n listed on the speech style sheet 110. Shekhar teaches the use of contextual embeddings that include expressive style but does not expressly include the word “prosody” which is just the sound of speech.
Shakeri teaches:
11. The method according to claim 1, wherein the text-to-speech synthesis model is configured to extract sequential prosodic features from the style tag and generate the synthesis voice for the target text reflecting the sequential prosodic features as the voice style features. [Shakeri teaches that speech style and prosody are the same or at least that speech style includes prosody and acoustic information are input as a sequence such that the prosodic information comes in a sequence of features. “A “speech audio generator” as used in some embodiments described herein, is a software module that receives an indication of an utterance and outputs speech audio corresponding to the indication. Various characteristics of the output speech audio may be varied by speech audio generator modules described herein, e.g. speech content, speaker identity, and speech style (for example the prosody of the output speech).” 3:35-43. “Additionally, or alternatively, the user may be asked to mimic an example of speech audio 107 such that the player speaks the same words as the example, in a speech style (e.g. prosody) similar to that of the example.” 7:12-17. “… In addition, each training example may further comprise speech style features (e.g. prosodic features) for the speech sample. …” 8:1-5. “The acoustic features may comprise a sequence of vectors, each vector representing acoustic information in a short time period, e.g. 50 milliseconds.” 6:40-43.]
Bakis/Shekhar and Shakeri pertain to generation of expressive speech and part of the expression is included in prosody (See Bakis figure 2) and therefore it would have been obvious to combine the express teachings of Shakeri wrt to prosody with the system of combination to cover more aspects of emotive speech. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Bakis and Shekhar in view of Mariano (U.S. 20170186418) and further in view of Bocchieri (U.S. 20120130709).
Regarding Claim 13, Bakis and Shekhar do not mention an API call.
Mariano teaches:
13. The method according to claim 1, wherein the style tag is input through an API call and wherein the receiving of the target text is via the API call. [Mariano teaches that in lieu of tagging the text with the style an API Call may be added to the application but does not consider it a good way: “[0045] In some conventional implementations, an application may be configured to provide input to a TTS system, but may not be configured to take advantage of all of the speech styles supported by the TTS system. Using the example of FIG. 2 to illustrate, an application may be configured to provide input text to style-specific linguistic analysis components 210A and 210B to produce neutral and joyful style speech output, but not to generate didactic style speech. In such conventional implementations, configuring the application to take advantage of an additional speech style may necessitate significant modifications to the application (e.g., to introduce API calls, etc.), testing of the application, testing of the integration of the application and TTS system, etc. By contrast, with some embodiments of the invention, enabling an application to take advantage of an additional output speech style may be accomplished by merely modifying the application to insert an additional type of style indication (e.g., tag) into input text.”]
Bakis/Shekhar and Mariano pertain to generation of expressive speech and it would have been obvious to combine the API call of Mariano with the system of combination in order to have a variation in the implementation. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Mariano does not teach input of text through an API call.
Bocchieri teaches:
wherein the receiving of the target text is via the API call. [Bocchieri is directed to generation of LM and AM models that can be used in speech recognition and therefore includes input of text corresponding to speech and teaches that he communications are performed via API calls. “[0011] Disclosed are systems, methods, and non-transitory computer-readable storage media for generating speech models from the perspective of a server and from a client device. The server receives a standard feature stream and/or an optional proprietary feature stream, transcriptions, and parameter values as inputs from a network client independent of and/or without access to or specific knowledge of internal operations of the automatic speech recognition system. However, the network client may have general knowledge of the available tools and functionalities via the API. The system processes the inputs to train an acoustic model and a language model. Then the server transmits the acoustic model and the language model to the network client. The client communicates with the server via an API. The client provides text, such as transcriptions of the input speech. The input speech can be recorded live from a user or can be selected from a database of previously recorded speech. The client device extracts features from the input speech and the input text based on configuration parameters and transmits, via an API call, the features, the input speech, the input text, and configuration parameter values to the server. Later, the client receives from the server an acoustic model and a language model generated based on the features or the input speech, the input text, and the configuration parameter values.” “[0033] The system then extracts features from the input speech and the input text based on configuration parameters (404). The system transmits, via an API call, the features, the input speech, the input text, and configuration parameter values to the server (406). The configuration parameter values can indicate one or more specific task, application, or desired use for the requested models. The ASR server can process the input speech, text, features, and so forth for a significant amount of time. While many API calls in other applications may result in a near-instantaneous response, the server may take several hours, days, or longer to generate an AM and LM in response to the API call. Thus, the system may wait for a long time for the server to respond to the API call. In one variation, the client does not keep a constant communication channel open with the server, such as an HTTP or HTTPS session or other persistent session, while waiting for the response to the API call from the server.”]
Bakis/Shekhar/Mariano and Bocchieri pertain to speech processing and it would have been obvious to combine the API call of Boccieri which is used for receiving all types of input including text with the system of combination in order to have a variation in the implementation. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
TTS and Model:
Fang (U.S. 20230298564):
“Embodiments of this application provide a speech synthesis method performed by an electronic device. The method includes: acquiring a target text to be synthesized into a speech; generating hidden layer features and prosodic features of the target text, and predicting pronunciation duration of characters in the target text using an acoustic model corresponding to the target text; generating acoustic features corresponding to the target text based on the hidden layer features, the prosodic features and the pronunciation duration; and synthesizing a target speech corresponding to the target text according to the acoustic features. Using the solution provided by the embodiments of this application is beneficial to reducing the difficulty of speech synthesis.” Abstract.
Oh (U.S. 20070106514):
“[0017] Another aspect of the present invention provides a speech synthesis method for adjusting a speech style, comprising the steps of: (a) receiving a sentence with a marked friendliness level; (b) selecting a prosodic model based on the marked friendliness level of the sentence; and (c) generating a synthesized speech of the sentence with the marked friendliness level by obtaining speech segments from a synthesis unit database on the basis of the selected prosodic model, the synthesis unit database storing speech segments for each friendliness level.”
Gururani (U.S. 20210151029):
“[0001] Text-to-speech systems are systems that emulate human speech by processing text and outputting a synthesized utterance of the text. However, conventional text to speech systems may produce unrealistic, artificial sounding speech output and also may not capture the wide variation of human speech. Techniques have been developed to produce more expressive text-to-speech systems, however many of these systems do not enable fine-grained control of the expressivity by a user. In addition, many systems for expressive text-to-speech use large, complex models requiring a significant number of training examples and/or high-dimensional features for training.” “[0013] … In addition, by being trained using low-dimensional representations, models described in this specification can better disentangle between different aspects of speech style, thus providing a user with more control when generating expressive speech from text.”
Claims 4-5:
Yang (U.S. 20210366462), Figure 9 determines emotion from the text of a received message. “[0172] In addition, the TTS device 100 according to an embodiment of the present invention may further include a semantic analysis module, a context analysis module, and an emotion determination module, and the semantic analysis module,…”Yang, Figure 10 shows an emotion vector that includes the plurality of emotions (neutral, love, happy, anger, sad) that is detected in this input text. “[0176] For example, FIG. 10 is a diagram for explaining an emotion vector according to an embodiment of the present invention. FIG. 10, if a received message M1 is “Love you”, the TTS device 100 calculates a first emotion vector through the syntactic analysis module, and the first emotion vector may be a vector in which weights are set to a plurality of emotion elements EA1, EA2, EA3, EA4, and EA5). For example, the first emotion vector is an emotion vector containing emotion classification information that is generated by assigning a weight “0” to neutral EA1, anger EA4 and sad EA5, a weight “0.9” to love EA2, and a weight “0.1” to happy EA3 with respect to the message M1 of “Love you”. Yang, Figures 17a and 17b show the output of the suggested/recommended emotions E to the user for his selection. “[0215] FIGS. 17A and 17B shows an example of transmitting a message with emotion classification information set therein according to an embodiment of the present invention.” “[0216] Referring to FIG. 17A, the transmitting device 12 may execute an application for writing a text message and the text message may be input. In this case, an emotion setting menu Es may be provided. The emotion setting menu Es may be displayed as an icon. When the emotion setting menu ES is selected after the text message is input, the transmitting device 12 may display at least one emotion item on a display so that a user can select emotion classification information with respect to the input text message. According to an embodiment, the at least one emotion item may be provided as a pop-up window. According to an embodiment, at least one candidate emotion item frequently used by the user may be recommended as the at least one emotion item”
Claim 9:
Shakeri (U.S. 11790884), Figure 1, “speech audio generator 110” is a TTS and “speech script 108” is the text that is converted to audio of a particular style: “… In some examples, the speech audio generator comprises a text-to-speech module configured to receive text data representing speech content, player identifier data for the player, and optionally, speech style features, and to output speech audio in the voice of the player.” 5:27-33. Figure 2, “synthesizer 202” and Figure 3, “speech content data 301” indicate TTS. “The data representing speech content may comprise text data. The text data may be any digital data representing text. …” 6:1-2. “Methods and systems described herein also enable the learning of an accurate representation of a speaker's voice in the form of a speaker embedding. By learning a suitable speaker embedding for each speaker, and inputting this along with source acoustic features into the voice convertor module, the performance of the source speech audio (e.g. prosody) may be retained while realistically transforming the voice of the source speech audio into that of the player.” 4:47-55.
Claim 10:
Shakeri (U.S. 11790884) Figures 5, 6, and 7 are directed to model training and each include a “model trainer 508/608/708.” Minimizing a loss function between the training data and input data is a feature of training. The loss is the error and must be optimized/minimized: “Model trainer 608 receives the predicted target acoustic features 607 and the “ground-truth” target acoustic features 602, and updates the parameters of acoustic feature encoder 605 and acoustic feature decoder 606 in order to optimize an objective function. The objective function comprises a loss in dependence on the predicted target acoustic features 607 and the ground-truth target acoustic features 602. For example, the loss may measure a mean-squared error between the predicted target acoustic features 607 and the ground-truth target acoustic features 602….” 16:1-10.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to FARIBA SIRJANI whose telephone number is (571)270-1499. The examiner can normally be reached 9 to 5, M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Fariba Sirjani/
Primary Examiner, Art Unit 2659
Prosecution Insights