DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Terminal Disclaimer
The Double Patenting rejection of the claims (6/30/26) is hereby withdrawn in light of the approved Terminal Disclaimer filed 9/30/26.
Continued Examination under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 9/30/26 has been entered.
Response to Arguments
Applicant's arguments filed 9/30/26 have been fully considered but they are not persuasive.
Regarding the 35 U.S.C. 103 rejection of the claims, Applicant argues that Meyer’s markers describe where a sentence or phrase begins and ends, but does not provide the claimed classification of content relationship, and as such argues that Meyer as well as additional reference Kim fail to disclose limitation “wherein the pre-processing module is configured to classify the relationship between sentences according to whether the unit text input is in a continuous relationship with the content of the previous sentence, whether the unit text input is in a relationship in which new content completely different from the content of the previous sentence starts, whether the unit text input is in a relationship to additionally explain the content of the previous sentence, or whether the unit text input is in a relationship to repeatedly explain the content of the previous sentence”, and further that the combination would not have been taught by the references (Arguments, pg. 7 - pg. 11, first para.).
Examiner respectfully disagrees as Meyer discloses its front end 250 as generating markers 254 from text input 240, where the generated markers indicate the locations of the beginnings ([begin sentence] marker) and endings ([end sentence] marker) of sentences in the received input text 240 (para. [0049]-[0050]; para. [0067]) that includes adjacent sentences (fig. 3A; para. [0061]), corresponding to an indication or a classification of the text according to relationships between a previous sentence (i.e. a marking of the ending of the previous sentence) and the next sentence (i.e. a marking of the beginning of the next sentence), consistent with Applicant’s original specification disclosure of a “relationship” (pg. 14, para. [0047]), and as a result, “wherein the pre-processing module is configured to classify the relationship between sentences according to …whether the unit text input is in a relationship in which new content completely different from the content of the previous sentence starts” in limitation “wherein the pre-processing module is configured to classify the relationship between sentences according to whether the unit text input is in a continuous relationship with the content of the previous sentence, whether the unit text input is in a relationship in which new content completely different from the content of the previous sentence starts, whether the unit text input is in a relationship to additionally explain the content of the previous sentence, or whether the unit text input is in a relationship to repeatedly explain the content of the previous sentence”.
Furthermore, in response to Applicant’s argument (Arguments, pg. 11, second - sixth para.) that there is no teaching, suggestion, or motivation to combine the references, including arguments regarding modifying the combination (i.e. the combined references) to perform steps in the claim language, Examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, given Meyer’s lack of a teaching of training its synthesis system, it would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Kim with the method of Meyer in arriving at the missing features of Meyer, because such combination would have resulted in generating output speech data that simulates a speech of a target speaker that reflects the articulatory feature, the emotion feature, and/or the prosody feature of the target speaker (Kim, para. [0067]). Therefore, Examiner maintains that the combination of the references discloses the language recited in claims 1 and 12 as well as the claims dependent therefrom.
Regarding the dependent claims 5, 6 and 9-11 rejected with additional references Arik and Stanton, Applicant argues that Arik and Stanton fail to disclose the above argued limitation regarding content-relationship classification as recited in independent claims 1 and 12 and as such argues that the references fail to disclose language recited in the dependent claims (Arguments, pg. 11, eighth para. - pg. 13).
Examiner respectfully disagrees as the combination of Meyer and Kim disclose the language recited in the independent claims as provided above, and absent any specific argument as to why the cited portions of Arik and Stanton fail to disclose or suggest limitations recited in the dependent claims, Examiner maintains that the rejections of the dependent clams are appropriate.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
1. Claims 1-4, 7, 8 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Meyer et al US 2011/0202344 A1 (“Meyer”) in view of Kim et al US 2020/0342852 A1 (“Kim”)
Per claim 1, Meyer discloses an apparatus for synthesizing speech, which is a computing apparatus that includes one or more processors and a memory storing one or more programs executed by the one or more processors, the apparatus comprising:
a pre-processing module configured to classify at least one of a position within a sentence and a relationship between sentences for each of unit texts input (fig. 2; fig. 3A; The example text input 150 is a plain text transcription of desired speech output 110, submitted to the TTS engine as, "Arriving at 221 Baker St. Please enjoy your visit." …, para. [0012]; Markers 254 may indicate in any suitable way the locations of various lexical, syntactic and/or prosodic boundaries and/or events that may be inferred from text input 240. For example, markers 254 may indicate the locations of boundaries between words … Markers 254 may also indicate the locations of words and/or word sequences of particular text normalization types, such as dates, times, currency, addresses, natural numbers, digit sequences and the like …, para. [0049]; Markers 254 generated from text input 240 by front-end 250 may be used by synthesis system 200 in further processing to select appropriate audio recordings 232 for rendering text input 240 as speech…., para. [0050]; For example, markers 330 include [begin sentence] and [end sentence] markers …, para. [0067]; para. [0077]-[0079]); and
a speech synthesis module configured to synthesize speech uttering the unit text based on the input unit text and at least one of the position within the sentence and the relationship between the sentences of each of the unit texts, wherein the relationship between sentences is regarding what kind of relationship each unit text has with a content of a previous sentence (fig. 3A; Markers 254 may also indicate the locations of the beginnings and endings of sentences …, para. [0049]; Markers 254 generated from text input 240 by front-end 250 may be used by synthesis system 200 in further processing to select appropriate audio recordings 232 for rendering text input 240 as speech…., para. [0050]; para. [0061]; For example, markers 330 include [begin sentence] and [end sentence] markers that may be derived from certain capitalizations and punctuation marks in text input 310 …, para. [0067]),
wherein the pre-processing module is configured to classify the relationship between sentences according to whether the unit text input is in a continuous relationship with the content of the previous sentence, whether the unit text input is in a relationship in which new content completely different from the content of the previous sentence starts, whether the unit text input is in a relationship to additionally explain the content of the previous sentence, or whether the unit text input is in a relationship to repeatedly explain the content of the previous sentence (para. [0049]-[0050]; para. [0061]; For example, markers 330 include [begin sentence] and [end sentence] markers that may be derived from certain capitalizations and punctuation marks in text input 310 …, para. [0067], end and beginning sentence markers inserted into text that include adjacent sentences (fig. 3A) as implying classifying the relationship between sentences according to whether the unit text input is in a relationship in which new content completely different from the content of the previous sentence starts),
Meyer does not explicitly disclose wherein the speech synthesis module is trained to synthesize speech of a predetermined speaker according to the at least one of the position within the sentence and the relationship between the sentences of each of the unit texts, so that speech characteristics of the predetermined speaker according to a context of the predetermined speaker can be reflected in a speech synthesis
However, this feature is taught by Kim (fig. 3; the attention 320 generates position information of an input text from which a speech is to be synthesized …, para. [0066]; the encoder 310 and the decoder 330 included in the speech synthesizer 230 may receive the articulatory feature, the emotion feature, and the prosody feature of the speaker. Herein, the articulatory feature, the emotion feature, and the prosody feature may be a speaker embedding vector … the encoder 310, the attention 320, and the decoder 330 included in the speech synthesizer 230 may configure a single artificial neural network text-to-speech synthesis model that simulates a speech of a target speaker by reflecting the articulatory feature, the emotion feature, and/or the prosody feature of the target speaker…. the single artificial neural network text-to-speech synthesis model configured by the speech synthesizer 230 may be trained using a sequence-to-sequence model (seq2seq) …, para. [0067]; the attention 724 included in the decoder 720 of FIG. 7 may correspond to the attention 320 of FIG. 3…., para. [0087]; the input text may include a sentence …, para. [0088]; the position information may be information regarding which location of the input text is being converted into a speech …, para. [0096])
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Kim with the method of Meyer in arriving at the missing features of Meyer, because such combination would have resulted in generating output speech data that simulates a speech of a target speaker by reflecting the articulatory feature, the emotion feature, and/or the prosody feature of the target speaker (Kim, para. [0067])
Per claim 2, Meyer in view of Kim discloses the apparatus according to claim 1,
Meyer discloses wherein the preprocessing module comprises: a classification unit configured to classify the at least one of the position within the sentence and the relationship between the sentences for each of the unit texts input (para. [0077]-[0078]); and
a classification marking unit configured to mark the classification symbol on each unit text according to the at least one of the position within the sentence and the relationship between the sentences of each of the unit texts (para. [0077]-[0078]).
Per claim 3, Meyer in view of Kim discloses the apparatus according to claim 2,
Meyer discloses wherein the classification unit is configured to classify the position within the sentence with respect to a beginning part and an end part of the unit text (para. [0077]-[0078]); and
the classification marking unit is configured to mark the classification symbol at the beginning part and the end part of the unit text (para. [0077]-[0078]).
Per claim 4, Meyer in view of Kim discloses the apparatus according to claim 3,
Meyer discloses wherein the unit text is obtained by dividing a sentence into several parts (Front-end 250 may process and/or analyze text input 240 to determine the sequence of words and/or sounds represented by the text …, para. [0047]); and
the classification marking unit is configured to omit the classification symbol according to the position within the sentence of the unit text (fig. 3A; fig. 3B).
Per claim 7, Meyer in view of Kim discloses the apparatus according to claim 1,
Meyer discloses: wherein the pre- processing module includes: a classification unit configured to classify the at least one of the position within the sentence and the relationship between the sentences for each of the unit texts input (fig. 2; fig. 3A; para. [0012]; para. [0067]; para. [0077]-[0079];
and a classification marking unit configured to generate the classification mark information according to the at least one of the position within the sentence and the relationship between the sentences of each of the unit texts (para. [0012]; para. [0049]; para. [0067]; para. [0077]-[0079]).
Per claim 8, Meyer in view of Kim discloses the apparatus according to claim 7, wherein the classification unit is configured to classify the position within the sentence with respect to a beginning part and an end part of the unit text (para. [0077]-[0078]); and
the classification marking unit is configured to generate the classification mark information with respect to the start part and the end part of the unit text (para. [0077]-[0078]).
Per claim 12, Meyer in view of Kim discloses a method for synthesizing speech performed on a computing apparatus that includes one or more processors and a memory storing one or more programs executed by the one or more processors, the method comprising:
classifying at least one of a position within a sentence and a relationship between sentences for each of unit texts input (fig. 2; fig. 3A; The example text input 150 is a plain text transcription of desired speech output 110, submitted to the TTS engine as, "Arriving at 221 Baker St. Please enjoy your visit." …, para. [0012]; Markers 254 may indicate in any suitable way the locations of various lexical, syntactic and/or prosodic boundaries and/or events that may be inferred from text input 240. For example, markers 254 may indicate the locations of boundaries between words … Markers 254 may also indicate the locations of words and/or word sequences of particular text normalization types, such as dates, times, currency, addresses, natural numbers, digit sequences and the like …, para. [0049]; For example, markers 330 include [begin sentence] and [end sentence] markers …, para. [0067]; para. [0077]-[0079]); and
synthesizing, using a speech synthesis module, speech uttering the unit text based on the input unit text and at least one of the position within the sentence and the relationship between sentences of each of the unit texts, wherein the relationship between sentences is regarding what kind of relationship each unit text has with a content of a previous sentence (fig. 3A; Markers 254 may also indicate the locations of the beginnings and endings of sentences …, para. [0049]; Markers 254 generated from text input 240 by front-end 250 may be used by synthesis system 200 in further processing to select appropriate audio recordings 232 for rendering text input 240 as speech…., para. [0050]; para. [0061]; For example, markers 330 include [begin sentence] and [end sentence] markers that may be derived from certain capitalizations and punctuation marks in text input 310 …, para. [0067]),
wherein the classifying comprises classifying the relationship between sentences according to whether the unit text input is in a continuous relationship with the content of the previous sentence, whether the unit text input is in a relationship in which new content completely different from the content of the previous sentence starts, whether the unit text input is in a relationship to additionally explain the content of the previous sentence, or whether the unit text input is in a relationship to repeatedly explain the content of the previous sentence (para. [0049]-[0050]; para. [0061]; For example, markers 330 include [begin sentence] and [end sentence] markers that may be derived from certain capitalizations and punctuation marks in text input 310 …, para. [0067], end and beginning sentence markers inserted into text that include adjacent sentences (fig. 3A) as implying classifying the relationship between sentences according to whether the unit text input is in a relationship in which new content completely different from the content of the previous sentence starts),
Meyer does not explicitly disclose wherein the speech synthesis module is trained to synthesize speech of a predetermined speaker according to the at least one of the position within the sentence and the relationship between the sentences of each of the unit texts, so that speech characteristics of the predetermined speaker according to a context of the predetermined speaker can be reflected in a speech synthesis
However, this feature is taught by Kim (fig. 3; the attention 320 generates position information of an input text from which a speech is to be synthesized …, para. [0066]; the encoder 310 and the decoder 330 included in the speech synthesizer 230 may receive the articulatory feature, the emotion feature, and the prosody feature of the speaker. Herein, the articulatory feature, the emotion feature, and the prosody feature may be a speaker embedding vector … the encoder 310, the attention 320, and the decoder 330 included in the speech synthesizer 230 may configure a single artificial neural network text-to-speech synthesis model that simulates a speech of a target speaker by reflecting the articulatory feature, the emotion feature, and/or the prosody feature of the target speaker…. the single artificial neural network text-to-speech synthesis model configured by the speech synthesizer 230 may be trained using a sequence-to-sequence model (seq2seq) …, para. [0067]; the attention 724 included in the decoder 720 of FIG. 7 may correspond to the attention 320 of FIG. 3…., para. [0087]; the input text may include a sentence …, para. [0088]; the position information may be information regarding which location of the input text is being converted into a speech …, para. [0096])
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Kim with the method of Meyer in arriving at the missing features of Meyer, because such combination would have resulted in generating output speech data that simulates a speech of a target speaker by reflecting the articulatory feature, the emotion feature, and/or the prosody feature of the target speaker (Kim, para. [0067]).
2. Claims 5 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Meyer in view of Kim as applied to claim 2 above, and further in view of Arik et al US 2019/0122651 A1 (“Arik”)
Per claim 5, Meyer in view of Kim discloses the apparatus according to claim 2,
Kim discloses wherein the speech synthesis module includes: an encoder configured to receive each of the unit texts and generate an embedding vector for each of the unit texts received (para. [0088]-[0089])
Meyer in view of Kim does not explicitly disclose wherein the speech synthesis module includes: an encoder configured to receive each of unit texts marked with the classification symbol and generate an embedding vector for each of the unit texts received or a decoder configured to synthesize speech for each unit text based on the embedding vector for each of the unit texts
However, these features are taught by Arik:
wherein the speech synthesis module includes: an encoder configured to receive each of unit texts marked with the classification symbol and generate an embedding vector for each of the unit texts received (para. [0040]; an input text in converted (205) into trainable embedding representations using an embedding model, such as text embedding model 110 …, para. [0043]; Text Preprocessing, para. [0045]; Replace spaces between words with special separator characters which indicate the duration of pauses …, para. [0050]; a phoneme-only model requires a preprocessing step to convert words to their phoneme representations …, para. [0054]; the text embedding model 110 may comprise a phoneme-only model …, para. [0055]) and
a decoder configured to synthesize speech for each unit text based on the embedding vector for each of the unit texts (a Deep Voice 3 architecture 100 uses residual convolutional layers in an encoder 105 to encode text into per-timestep key and value vectors 120 for an attention-based decoder 130. In one or more embodiments, the decoder 130 uses these to predict the mel-scale log magnitude spectrograms 142 that correspond to the output audio …, para. [0040])
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Arik with the method of Meyer in arriving at the missing features of Meyer, because such combination would have resulted in providing speech with better quality and naturalness (Arik, Abstract; para. [0063]).
Per claim 6, Meyer in view of Kim and Arik discloses the apparatus according to claim 5,
Arik discloses: an attention unit configured to receive an embedding vector for each unit text from the encoder and generate an attention map by determining an attention weight for each unit text based on the embedding vector (para. [0043]; para. [0061]),
wherein the decoder is configured to receive an embedding vector and an attention map for each unit text from the attention unit, and synthesize speech for each unit text based on the embedding vector and the attention map (The attention value vectors may be computed from attention key vectors and text embeddings. to jointly consider the local information in he and the long-term context information in hk. The key vectors hk are used by each attention block to compute attention weights …, para. [0061]).
3. Claims 9-11 are rejected under 35 U.S.C. 103 as being unpatentable over Meyer in view of Kim as applied to claim 2 above, and further in view of Stanton et al US 2021/0035551 A1 (“Stanton”)
Per claim 9, Meyer in view of Kim discloses the apparatus according to claim 7,
Kim discloses wherein the speech synthesis module includes: a first encoder configured to receive each of the unit texts and generate a first embedding vector for each of the unit texts (para. [0088]-[0089]);
Meyer in view of Kim does not explicitly disclose a second encoder configured to receive classification mark information for each of the unit texts and generate a second embedding vector for the classification mark information, a combining unit configured to generate a combination vector by combining the first embedding vector and the second embedding vector or a decoder configured to synthesize speech for each of the unit texts based on the combination vector.
However, these features are taught by Stanton:
a second encoder configured to receive classification mark information for each of the unit texts and generate a second embedding vector for the classification mark information (The context features may include … or speech tags (e.g., noun, verb, adjective, etc.) …, para. [0037]; Context Features, Style Embedding, fig. 6B; para. [0072]; para. [0080]);
a combining unit configured to generate a combination vector by combining the first embedding vector and the second embedding vector (para. [0037]; Input Text Sequence, Text Encoder, Context Features, Style Embedding, Concatenator, fig. 6B; para. [0072]; para. [0080]); and
a decoder configured to synthesize speech for each of the unit texts based on the combination vector (para. [0037]; Input Text Sequence, Text Encoder, Context Features, Style Embedding, Concatenator, fig. 6B; para. [0072]; para. [0080]); and a decoder configured to synthesize speech for each of the unit texts based on the combination vector (Decoder, fig. 6B; provides the concatenation to the decoder 658 of the TTS model 650 for conversion into synthesized speech …, para. [0072])
It would have been obvious to one of ordinary skill in the art before the effective filing of the invention to combine the teachings of Stanton with the method of Meyer in view of Kim in arriving at the missing features of Meyer in view of Kim, because such combination would have resulted in synthesizing expressive speech having a natural sounding prosody/style based on the input text and the context features (Stanton, para. [0088]).
Per claim 10, Meyer in view of Kim and Stanton discloses the apparatus according to claim 9,
Meyer discloses wherein the classification mark information is indexed according to the at least one of the position within the sentence and the relationship between the sentences (para. [0012]; para. [0049]; para. [0077]-[0078]); and
Stanton suggests the second encoder is configured to generate a one-hot encoding vector for the classification mark information of each of the unit texts, and generate the second embedding vector by embedding the one-hot encoding vector (Each character in the sequence of characters can be represented as a one-hot vector and embedded into a continuous vector. That is, the subsystem 102 can represent each character in the sequence as a one-hot vector and then generate an embedding, i.e., a vector or other ordered collection of numeric values, of the character before providing the sequence as input to the seq2seq network 106, para. [0041]; the encoder 652, the attention module 656, and the decoder 658 may collectively correspond to the seq2seq recurrent neural network 106 …, para. [0080], context embedding 612 provided to seq2seq network 106/model 650 as suggesting limitation).
Per claim 11, Meyer in view of Kim and Stanton discloses the apparatus according to claim 9,
Stanton suggests: an attention unit configured to receive the combination vector for each unit text from the combining unit, and generate an attention map by determining an attention weight for each unit text received (fig. 6B; element 656),
wherein the decoder is configured to receive a combination vector and an attention map for each unit text from the attention unit, and synthesize speech for each unit text based on the combination vector and the attention map (fig. 6B, elements 656,658; the attention module 512 outputs a set of combination weights 516, 516a-n that represent the contribution of each style token 514 to the encoded prosody embedding PE 450. The attention module 512 may determine the combination weights 516 by normalizing the style tokens 514 via a softmax activation …, para. [0068]; the attention module 656 is configured to convert the concatenation 655 to a fixed-length context vector 657 for each output step of the decoder 658 to produce the output audio signal 670 …, para. [0080], attention modules as calculating weights/map for input, attention module 656 as providing output to decoder 658 as suggesting limitation).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See PTO 892 form.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OLUJIMI A ADESANYA whose telephone number is (571)270-3307. The examiner can normally be reached Monday-Friday 8:30-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richemond Dorvil can be reached at 571-272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OLUJIMI A ADESANYA/Primary Examiner, Art Unit 2658
