DETAILED ACTION
This office action is in response to Applicant’s Request for Continued Examination (RCE), received on 02/13/2026. Claims 1-2, 4-5, 7-10, 12-13, 15-16, 19, 21, 23-24 have been amended. Claims 1-5, 7-13, 15-16, 18-25 are pending and have been considered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/13/2026 has been entered.
Response to Arguments
Applicant’s arguments, see pg. 12, filed 02/13/2026, with respect to the “Claim Objections” have been fully considered and are persuasive. The objection of claim 15 has been withdrawn.
Applicant's arguments filed 02/13/2026, see pgs. 12-13, with respect to have been fully considered but they are not persuasive.
Applicant’s representative asserts, “Claims 1-5, 7, 8, 10-13, 15, 16, 18, 20, 23, and 24 are rejected under 35 U.S.C. § 101 as allegedly being directed to an abstract idea without significantly more. Applicant respectfully disagrees. However, solely in the interest of advancing prosecution, Applicant amends independent claims 1 and 12. Additionally, Applicant respectfully submits that amended independent claims 1 and 12 are not directed to an abstract idea and even assuming arguendo they are, amended independent claims 1 and 12 include significantly more than any alleged abstract idea. Consequently, Applicant respectfully requests that the Office withdraw the § 101 rejections of claims 1-5, 7, 8, 10-13, 15, 16, 18, 20, 23, and 24.
Under the USPTO's Subject Matter Eligibility Update of August 2025, the Office has clarified that the 'Mental Process' exception should not be stretched to cover AI-specific data structures that have no human analog. Amended Claim 21 recites generating 'spectrograms' using a 'decoder' and processing them with a 'vocoder.' A mel-spectrogram is a complex, high- dimensional frequency representation of audio that a human cannot mentally generate, visualize, or process to create audio waves. As such, these limitations recite specific machine-centric data structures that remove the claim from the realm of mental processes.”
In response, the examiner would like to refer to the status of the claim, in view of Applicant’s arguments and/or amendments. Specifically, the examiner respectfully asserts that claim 21 was previously indicated to be containing eligible subject matter (see pg. 16 of Final action mailed 12/19/2026). The examiner agrees with Applicant’s remarks around the eligibility of claim 21, but as the other independent claims have not had similar amendments entered, these claims remain ineligible for reasons which will be explained below. Claims 9 and 19 remain eligible for reasons described in the previous action.
Applicant's arguments filed 02/13/2026, see pgs. 13-15, with respect to “Rejections based on 35 U.S.C. 103” (for independent claim 1) have been fully considered but they are not persuasive.
Applicant’s representative asserts, “…that Zhang and Gao, whether taken alone or in combination, do not teach or suggest, at least, ‘receiving first input data representative of one or more first voice characteristics of a first speaker, the one or more first voice characteristics disentangling from a first accent of the first speaker; [and] receiving second input data representative of one or more second voice characteristics corresponding to a second accent learned from speech from one or more second speakers in a target language, the one or more second voice characteristics disentangled from voice characteristics of one or more second speakers,’ as amended claim 1 recites.
In the rejection of previously presented independent claim 1, the Office acknowledges that Zhang does not teach or suggest ‘the one or more characteristics associated with disentangling a first accent of the first speaker’ and ‘determining a second accent learned using one or more instances of speech from one or more second speakers, the one or more instances of speech being in a target language.’ Office Action, p. 17. Consequently, Applicant respectfully submits that Zhang further does not teach or suggest ‘receiving first input data representative of one or more first voice characteristics of a first speaker, the one or more first voice characteristics disentangling from a first accent of the first speaker; [and] receiving second input data representative of one or more second voice characteristics corresponding to a second accent learned from speech from one or more second speakers in a target language, the one or more second voice characteristics disentangled from voice characteristics of one or more second speakers,’ as amended claim 1 recites.
Gao does not remedy the deficiencies of Zhang. In the rejection of previously presented independent claim 1, the Office cites Gao as allegedly teaching, ‘the one or more characteristics associated with disentangling a first accent of the first speaker’ and ‘determining a second accent learned using one or more instances of speech from one or more second speakers, the one or more instances of speech being in a target language.’ Id., p. 18. However, when describing any type of disentangling, Gao states:
After training the style embedding, a TTS voice may be trained alone with that style embedding to allow disentangling of speech content and style. Once trained, the speech system may apply a style detected from a user query to a customized TTS voice that may thus respond in a style appropriate to the detected style.
Gao, col. 32,11. 6-11.
As shown, Gao describes disentangling speech content from style. Id. Gao does not teach or suggest disentangling ‘one or more first voice characteristics’ of speech from ‘a second voice characteristic’ of the speech, as the speech content in Gao is not a ‘voice characteristic.’ Therefore, Gao further does not teach or suggest performing disentangling of a ‘voice characteristic corresponding to ... [an] accent’ to determine ‘one or more [other] voice characteristics.’ Consequently, Gao does not teach or suggest ‘receiving first input data representative of one or more first voice characteristics of a first speaker, the one or more first voice characteristics disentangling from a first accent of the first speaker; [and] receiving second input data representative of one or more second voice characteristics corresponding to a second accent learned from speech from one or more second speakers in a target language, the one or more second voice characteristics disentangled from voice characteristics of one or more second speakers,’ as amended claim 1 recites.
For similar reasons, Applicant respectfully submits that Zhang and Gao, whether taken alone or in combination, do not teach or suggest, ‘generating, using a vocoder of one or more models and based at least on the one or more first voice characteristics, the one or more second voice characteristics, and the text, audio data representative the text spoken in the target language,’ as amended claim 1 recites.”
In response, the examiner would like to refer to the broadest reasonable interpretation (BRI) of the claims as currently amended in view of the prior art. Specifically, Applicant’s arguments against Zhang not teaching the “receiving first input data” and “receiving second input data” are unpersuasive because they are merely alleging that Zhang does not teach the recited elements without explaining how or why.
Applicant's arguments fail to comply with 37 CFR 1.111(b) because they amount to a general allegation that the claims define a patentable invention without specifically pointing out how the language of the claims patentably distinguishes them from the references.
Applicant's arguments do not comply with 37 CFR 1.111(c) because they do not clearly point out the patentable novelty which he or she thinks the claims present in view of the state of the art disclosed by the references cited or the objections made. Further, they do not show how the amendments avoid such references or objections.
With regard to Applicant’s arguments against Gao, the examiner would like to refer to the BRI of “voice characteristic” as currently claimed in view of the combination of Zhang in view of Gao. [0025] of the instant app defines characteristics associated with voice to be “a timbre, a volume, a pace, a pitch, a resonance, and/or any other attributes that may be associated with the voice”, wherein the characteristics do not include accent. Based on this definition, the examiner would like to turn towards Gao. The cited section of Gao discloses “style embedding[s]” which are used to separate speech content from style. These style embeddings of Gao are taken in view of the style encodings of Zhang, wherein Zhang discloses style encodings to “describe the prosody information of the speaker” ([0085]), wherein prosody information is disclosed to represent pause durations ([0064]). Taking this in view of the instant application’s voice characteristic definition (including pace), the examiner respectfully asserts that the style encodings of Gao in view of Zhang track to voice characteristics as currently claimed. Further, the examiner respectfully asserts that “disentangling ‘one or more first voice characteristics’ of speech from ‘a second voice characteristic’ of the speech” is not something which is claimed. In claim 1, the only disentangling claimed is with respect to one or more first/second voice characteristics of one or more first/second speakers. There is no connection between the first and second voice characteristics in the disentangling; therefore, Applicant’s arguments against the following claim elements are unpersuasive as the examiner respectfully asserts that the combination of Zhang in view of Gao discloses the “receiving first input data” and “receiving second input data” in view of the plurality of second styles of Gao. The examiner asserts that extending “receiving” operations disclosed by the art to “second” instances of the same operations/objects does not prevent the operations from being performed by the same art as the first and second ‘receiving’ operations are not related.
Applicant's arguments filed 02/13/2026, see pgs. 15-18, with respect to the rejection of independent claim 12 (Zhang in view of Gao) have been fully considered but they are not persuasive.
Applicant’s representative asserts, “Applicant respectfully submits that Zhang and Gao, whether taken alone or in combination, do not teach or suggest, at least, ‘determin[ing], using one or more models and based at least on the second accent and text, one or more durations associated with one or more phonemes corresponding to text; determin[ing], using one or more models and based at least on the one or more durations associated with the one or more phonemes, one or more features associated with speech; [and] determin[ing], using the one or more models and based at least on the one or more durations associated with the one or more phonemes and the one or more features, audio data representative of the speech in the target language and spoken based at least on the second accent,’ as amended claim 12 recites.
In the rejection of previously presented independent claim 12, the Office cites Zhang as allegedly teaching, ‘determine, using one or more models and based at least on the one or more characteristics and the second accent, one or more features associated with the speech; determine, using the one or more models and based at least on the one or more features, audio data representative of speech in the target language and spoken based at least on the second accent.’ Office Action, pp. 26 and 27. Additionally, in the rejection of previously presented dependent claim 18, the Office cites Zhang as allegedly teaching, ‘determine an alignment between one or more embeddings associated with text and one or more phonemes based at least on the second accent associated with the target language, wherein the one or more features are further determined based at least on the alignment.’ Id., p. 29.
However, the cited portion Zhang as allegedly teaching the ‘aligning’ describes that the style encoding, the text, encoding, and the speaker encoding are fused. Zhang, para. [0086]. Zhang does not teach or suggest determining ‘one or more durations associated with one or more phonemes’ of the text, let alone based on an ‘accent.’ For instance, the only duration described in Zhang is the pause duration between each word segment. Id., para. [0067]. However, Zhang does not teach or suggest a ‘duration’ associated with a phoneme.
Therefore, Zhang further does not teach or suggest determining ‘one or more features associated with speech’ and/or generating the speech based on ‘one or more durations associated with one or more phonemes.’ Consequently, Zhang does not teach or suggest ‘determin[ing], using one or more models and based at least on the second accent and text, one or more durations associated with one or more phonemes corresponding to text; determin[ing], using one or more models and based at least on the one or more durations associated with the one or more phonemes, one or more features associated with speech; [and] determin[ing], using the one or more models and based at least on the one or more durations associated with the one or more phonemes and the one or more features, audio data representative of the speech in the target language and spoken based at least on the second accent,’ as amended claim 12 recites.
Gao does not remedy the deficiencies of Zhang. For instance, the only duration that Gao describes is the duration of an audio output. Cao, col. 32, 11.22-55. However, Gao does not teach or suggest ‘one or more durations associated with one or more phonemes.’ Consequently, the combination of Zhang and Gao does not teach or suggest ‘determin[ing], using one or more models and based at least on the second accent and text, one or more durations associated with one or more phonemes corresponding to text; determin[ing], using one or more models and based at least on the one or more durations associated with the one or more phonemes, one or more features associated with speech; [and] determin[ing], using the one or more models and based at least on the one or more durations associated with the one or more phonemes and the one or more features, audio data representative of the speech in the target language and spoken based at least on the second accent,’ as amended claim 12 recites.”
In response, the examiner would like to refer to previously uncited sections of Gao. Specifically, the examiner agrees with Applicant’s assertions that Zhang does not disclose the argued phoneme duration claim element, but respectfully disagrees with Applicant’s assertions that “the only duration that Gao described is the duration of audio output.” Referring to Fig. 6 of Gao, there can be seen Prosody Models 630 which determine duration and frequency of text as part of the linguistic features ([Col. 32, Lines 30-55]), wherein Gao also discloses grapheme-to-phoneme conversion ([Col. 12, Lines 25-35]) which occurs before the TTS operation (as seen in Fig. 2). This indicates a duration determination of phonemes to be disclosed in Gao. Further, considering the Prosody Models 630 receive both linguistic features representing duration and style embeddings representing accent ([Col. 41, Lines 5-15]), there is a strong indication that the style embeddings are used in combination with the linguistic features for determining the duration of phonemes using the prosody model. Gao explicitly discloses this: [Col. 35, Lines 30-35] the linguistic features, along with any conditional features such as style embedding 615 or speaker ID, may be used to produce the prosody features, such as duration.
Applicant’s arguments, see pgs. 18-19, filed 02/13/2026, with respect to the rejection(s) of claim(s) 21 under 35 U.S.C. 103 (Zhang in view of Gao) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Elias et al. (US-20220122582-A1), hereinafter Elias. Elias discloses “A method for training a non-autoregressive TTS model includes receiving training data that includes a reference audio signal and a corresponding input text sequence. The method also includes encoding the reference audio signal into a variational embedding that disentangles the style/prosody information from the reference audio signal and encoding the input text sequence into an encoded text sequence. The method also includes predicting a phoneme duration for each phoneme in the input text sequence and determining a phoneme duration loss based on the predicted phoneme durations and a reference phoneme duration. The method also includes generating one or more predicted mel-frequency spectrogram sequences for the input text sequence and determining a final spectrogram loss based on the predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence” (abstract). See updated rejection below.
Applicant's arguments filed 02/13/2026, with respect to the rejections of dependent claims 3, 13, 20, 22 and 9 and 19 have been fully considered but they are not persuasive. In view of the examiner’s above rationale maintaining and/or updating rejections of the independent claims, Applicant’s arguments against dependent claims are not persuasive as the examiner asserts that the cited art teaches every element of the claims. See updated rejections below.
Claim Objections
Claim 1 is objected to because of the following informalities: the “generating, using a vocoder…” step reads, “audio data representative the text spoken in the target language”. The examiner believes “of” should be added between “representative” and “the” to clarify the language of the claim. Appropriate correction is required.
Claim Rejections - 35 USC § 101
Claim(s) 1-5, 7-8, 10-13, 15-16, 18, 20, 23-24 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Independent claim 1 recites:
receiving first input data representative of one or more first voice characteristics of a first speaker, the one or more first voice characteristics disentangling from a first accent of the first speaker;
receiving second input data representative of one or more second voice characteristics corresponding to a second accent learned from speech from one or more second speakers in a target language, the one or more second voice characteristics disentangled from voice characteristics of one or more second speakers;
determining text associated with the target language;
generating, using a vocoder one or more models and based at least on the one or more first voice characteristics, the one or more second voice characteristics, and the text, audio data representative the text spoken in the target language; and
causing output of the generated audio data.
Independent claim 12 recites:
one or more processing units to:
determine one or more voice characteristics of a first speaker, the one or more voice characteristics associated with disentangling a first accent of the first speaker;
determine a second accent learned using one or more instances of speech from one or more second speakers, the one or more instances of speech being in a target language;
determine, using one or more models and based at least on the second accent and text, one or more durations associated with one or more phonemes corresponding to text,
determine, using one or more models and based at least on the one or more durations associated with the one or more phonemes, one or more features associated with speech;
determine, using the one or more models and based at least on the one or more durations associated with the one or more phonemes and the one or more features, audio data representative of the speech in the target language and spoken based at least on the second accent; and
cause output of the speech represented by the audio data.
These limitations, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. For example, the claim(s) read(s) on translating text into a different language based on a received target language, while pronouncing that translation in a certain way, i.e. based on identity/accent information. That is, other than reciting “a vocoder of one or more models” (Claims 1, 12), “one or more models” (claim 12), and “one or more processors” (Claims 12), nothing in the claim element precludes the step from practically being performed in the mind.
For example, consider a bilingual speaker impersonating someone speaking in a different, first language, i.e. to relay news, current events, etc., in a second language to their friends, coworkers, etc. The impersonator will listen to their desired information, which can be transcribed using pen and paper, wherein the listening operation will provide the impersonator with identity and accents, i.e. one or more characteristics (accents will contain feature information, e.g. accents represent specific pronunciations wherein each pronunciation, phoneme, represents a feature), of the speaker in the cases of famous, well-known speakers. This operation can be extended to a second, i.e. target, language/voice to determine a second accent given the listener is fluent in both languages and will be able to identify regional accents in both. Further, similar to the listening operation for determining first voice characteristic, this can be extended to a second input data without precluding the mental process classification. Based on the auditorily received audio and transcribed text associated with the audio, the impersonator can mentally prepare what they would sound like doing their impersonation, i.e. generate/determine audio data representative of the speech. Further, they can practice the speaking in the impersonation to themselves, indicating a generation/determination without a “output”. Incorporating a vocoder of one or more models to perform this task is assigning the mental process to a generic computing component. This does not provide an inventive step. When the impersonator has perfected their act, they can then present (and/or record using generic recording objects) their finalized impersonation to be shown to others, representing the output step. The step of translating received text in a first language into a second, target language is a well-known mental process to those who are at least bilingual speakers. Similarly, determining accents in multiple languages is a well-known operation associated with listening for those who are fluent in both languages. Further, the additional “determine…one or more durations associated with one or more phonemes” of independent claim 12 is also a mental process. Listening to audio allows for a user to determine lengths of phonemes as the audio is processed through the brain, wherein the mentally determined lengths can be written down in association with a transcript using pen and paper.
All of these steps can be performed in the mind and/or using pen and paper. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea (Step 2A, Prong one, Yes).
This judicial exception is not integrated into a practical application because the addition of generically recited computer elements does not add a meaningful limitation to the abstract idea because they amount to simply implementing the abstract idea on a computer. The claims are directed to an abstract idea. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception ( Step 2A, Prong two, No). As discussed above, with respect to integration of an abstract idea into a practical application, the additional element of “receiving”, “determining”, “generating”, “causing” are merely for the purpose of data gathering, storing, processing, and/or insignificant extra-solution activity that amount to no more than mere instructions to apply the exception using a generic computer component. Paragraph(s) [0071] of the instant application disclose(s) applying the method to a generic computing device such as a PC, i.e. “computing device” which does not distinguish itself from a “laptop”, “desktop”, etc. Mere instructions to apply the exception using a generic computer component cannot provide an inventive concept. Therefore, the claims are not patent eligible (Step 2B, No).
Similarly, dependent claim(s) 2-5, 7-8, 10-11, 13, 15-16, 18, 20, 23-24 include additional steps that are considered “insignificant extra-solution activity to judicial exception” because they fail to provide meaningful significance that goes beyond generally linking the use of an abstract idea to a particular technological environment.
For example, claim 2 reads on determining features associated with the to-be generated speech, wherein those features are gathered from the characteristics, text, and accent, using those determined features to generate audio data. Determining features of speech based on the identity of the speaker and/or the accent of the speaker is a mental process. For example, if the accent of the speaker is determined to be “Southern American” (as would be determined through listening), the person listening to the speech, i.e. the impersonator (continuing the example from the independent claims), will know to draw out the vowels near the end of words and/or shorten the ends of words, i.e. “yellow” would be pronounced as “yella”. Similarly, the impersonator speaking using this accent would be representative of generated audio data based on the features, i.e. gathered from an accent.
Claim 3 reads on the features associated with the speech being one or more of energies, frequencies, or phoneme durations. Determining the length of specific phonemes is a mental process, i.e. how impersonators pick up new manners of speaking through listening. Determining the energies and/or frequencies of signals are considered to be well-known, routine, and/or conventional mathematical operations which can be applied using generic methods. Consider:
Joublin et al. (US-20080234959-A1), which discloses conventional methods of determining fundamental frequencies of harmonic signals, [0003].
Hoek (US-6266003-B1), which discloses conventional sinusoidal analysis methods for determining the frequency of the sinusoids, [Col. 1, Lines 50-65].
Nuutinen et al. (US-20030016771-A1), which discloses calculating the energy of a signal using methods well known in the field, [0027].
Dominique et al. (US-20080205329-A1), which discloses an energy calculation which can be implemented using well-known methods, [0051].
Therefore, the well-known extra-solution activity cannot be considered inventive.
Claim 4 reads on determining a speaker’s identity and receiving data based on the identity. Determining the identity of a person is a mental process. For example, consider an impersonator who wants to sound like Ozzy Osbourne. Based on the impersonator’s awareness of the identity, they will know to apply an English accent, thereby mentally identifying characteristics of speech specific to English accents. Based on these mentally determined characteristics, the impersonator can then speak them into life, i.e. generating audio data based on the characteristics. If the speaker sounds “correct” or similar to an expected identity, the listener will decide to listen.
Claim 5 reads on learning speech characteristics by disentangling the accent from additional, second speech in an additional, second language. The process of separating accent from speech is a mental process, regardless of the language being spoken in as “additional”. Listening to speech, a user familiar with both languages will understand the accent and can separate them mentally, i.e. “The cat ran away”, which the listener would have no problem transcribing with pen and paper, said with a New York accent.
Claim 7 reads on generating an alignment between embeddings associated with the text and phonemes, wherein the generated audio data is based on the alignment, accent, and characteristics. Embedding input text is a well-known operation considered to be extra-solution activity. Consider:
Chiu et al. (US-7260771-B2), which discloses well known methods for converting text into encodings, i.e. embeddings, [Col. 8, Lines 25-45].
Yang et al. (US-20210209304-A1), which discloses well-known word embedding models, [0130].
Therefore, embedding text does not provide an inventive step. Further, aligning embeddings and phonemes, i.e. graphemes to phonemes, is a mental process. The object of pronunciation is to appropriately pair text to its associated spoken sound. Further, performing this alignment based on accent and/or identity (deemed to be determined through mental processes) indicates a further mental process to know how to align the text with phonemes, i.e. length of pronunciation of words, different pronunciations of words, etc., which is dependent upon a knowledge of how different accents/identities pronounce words (determined through the mental process of listening to speakers with these accents).
Claim 8 reads on determining durations associated with phonemes corresponding to text based on the text and second voice characteristics. As previously disclosed, determining phoneme durations is a mental process. Generating audio based on a known length of time for phonemes to be pronounced is the mental process of speaking in a certain way with a certain cadence based on written speaking instructions.
Claim 10 reads on preserving the voice or timbre associated with a speaker. Preserving the voice and/or timbre of a speaker is the process of impersonation. Impersonators determine their pronunciations mentally.
Claim 11 reads on generating the audio data without additional audio data associated with the speaker in the target language. Generating audio data without additional audio data, i.e. just based on a received text, accent, and identity, is a mental process. An impersonator can be provided with identity, accent, and text to be spoken on a piece of paper and told to provide their best effort at producing the combination. Their attempt would be generated/spoken using mentally determined pronunciations.
Claim 13 reads on the features associated with the speech being one or more of energies, frequencies, or phoneme durations. Determining the length of specific phonemes is a mental process, i.e. how impersonators pick up new manners of speaking through listening. Determining the energies and/or frequencies of signals are considered to be well-known, routine, and/or conventional mathematical operations which can be applied using generic methods. Consider:
Joublin et al. (US-20080234959-A1), which discloses conventional methods of determining fundamental frequencies of harmonic signals, [0003].
Hoek (US-6266003-B1), which discloses conventional sinusoidal analysis methods for determining the frequency of the sinusoids, [Col. 1, Lines 50-65].
Nuutinen et al. (US-20030016771-A1), which discloses calculating the energy of a signal using methods well known in the field, [0027].
Dominique et al. (US-20080205329-A1), which discloses an energy calculation which can be implemented using well-known methods, [0051].
Therefore, the well-known extra-solution activity cannot be considered inventive.
Claim 15 reads on determining the identity of the speaker based on the determined speech characteristics. As previously disclosed, determining the identity of a speaker is a mental process. A user can listen to audio and identify that speaker as a man, woman, child, etc.
Claim 16 reads on learning speech characteristics by disentangling the accent from additional, second speech in an additional, second language. The process of separating accent from speech is a mental process, regardless of the language being spoken in as “additional”. Listening to speech, a user familiar with both languages will understand the accent and can separate them mentally, i.e. “The cat ran away”, which the listener would have no problem transcribing with pen and paper, said with a New York accent.
Claim 18 reads on generating an alignment between embeddings associated with the text and phonemes, wherein the generated audio data is based on the alignment, accent, and identity. Embedding input text is a well-known operation considered to be extra-solution activity. Consider:
Chiu et al. (US-7260771-B2), which discloses well known methods for converting text into encodings, i.e. embeddings, [Col. 8, Lines 25-45].
Yang et al. (US-20210209304-A1), which discloses well-known word embedding models, [0130].
Therefore, embedding text does not provide and inventive step. Further, aligning embeddings and phonemes, i.e. graphemes to phonemes, is a mental process. The object of pronunciation is to appropriately pair text to its associated spoken sound. Further, performing this alignment based on accent and/or identity (deemed to be determined through mental processes) indicates a further mental process to know how to align the text with phonemes, i.e. length of pronunciation of words, different pronunciations of words, etc., which is dependent upon a knowledge of how different accents/identities pronounce words (determined through the mental process of listening to speakers with these accents).
Claim 20 reads on the system of the claimed invention being comprised in at least one of a plurality of generic computing environments. Applying a mental process to a generic environment does not introduce and inventive step.
Claim 23 reads on the speech refraining from being spoken using the first accent based on the characteristics associated with the accent. Determining to not speak in a particular accent is a mental determination.
Claim 24 reads on the speech refraining from being spoken using the first accent based on the characteristics associated with the accent. Determining to not speak in a particular accent is a mental determination.
Therefore, these claims are also not patent eligible.
Claims 9, 19, 21-22, 25 have been deemed to contain eligible subject matter due to the specificity with which the outputs of these claims are generated. Specifically, one could argue that generation of a spectrogram and/or “audio data representative of speech” using decoders and vocoders is well-known extra-solution activity; however, the generation of these elements based upon the combination of identity of speaker, text associated with a target language, and accent associated with the target language introduces enough input detail to prevent an assertion that these steps are simply well-known extra-solution activity. There is a clear method defined in these claims which distinguishes itself from other audio/audio representation generation methods considered to be well-known. Claims 22, 25 are eligible as they are dependent upon eligible base claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4-5, 7-8, 10-12, 15-16, 18, 23-24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (US-20220375453-A1), hereinafter Zhang, in view of Gao et al. (US-11562744-B1), hereinafter Gao.
Regarding claim 1, Zhang discloses: a method comprising:
receiving first input data representative of one or more first voice characteristics of a first speaker ([0090] The speaker identifier is input into the style encoder, to obtain a corresponding style feature for the speaker, [0099] The style encoding (output by the style network) may be obtained by inputting the linguistic features of the text samples and the speaker identifiers of the speech samples into the style network, [Generating style features, i.e. characteristics, of a speaker based on a speaker identifier indicates the features are received corresponding to a voice of a first speaker, i.e. represented through the identifier]).
Zhang does not disclose:
the one or more first voice characteristics disentangling from a first accent of the first speaker; and,
receiving second input data representative of one or more second voice characteristics corresponding to a second accent learned from speech from one or more second speakers in a target language, the one or more second voice characteristics disentangled from voice characteristics of one or more second speakers.
Gao discloses:
the one or more first voice characteristics disentangling from a first accent of the first speaker ([Col. 32, Lines 5-10] After training the style embedding, a TTS voice may be trained alone with that style embedding to allow disentangling of speech content and style, [In view of the previously disclosed style encodings of Zhang which could be substituted for the style embedding of Gao without a change in functionality to the disentangling of speech content and style of Gao]); and,
receiving second input data representative of one or more second voice characteristics corresponding to a second accent learned from speech from one or more second speakers in a target language ([Col. 32, Lines 65-67] for example, the output may choose between a British or an American accent), [Col. 41, Lines 10-15] At step 820, the assistant system 140 may determine a first style of the voice input based on one or more first features extracted from the voice input and/or the input text. Examples of these first features may include volume, emphasis, pronunciation, accent, [In view of the two accents claimed, i.e. British/American, indicating either of these two accents to be a second input from a second speaker as compared to voice input received for the other accent in order to be learned and chosen. Further, in view of the above excerpt regarding choosing an output style, indicating the chosen style to be a target style with an associated target language, i.e. English, see “user-specific customization” defined in Col. 32, Lines 60-65 indicating the user to be specifying the target language, wherein the plurality of accents indicates at least two speakers with second voice characteristics]),
the one or more second voice characteristics disentangled from voice characteristics of one or more second speakers ([In view of the previously disclosed disentanglement of Gao which could be extended to a second style (see “second style”, [Col. 41, Lines 5-45], which is necessarily learned via second voice using the operations of Zhang in view of Gao)]).
Zhang and Gao are considered analogous art within text-to-speech systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang to incorporate the teachings of Gao, because of the novel way to use a semi-supervised approach for determining style representations for synthetic speech, improving the communication abilities of human-machine speech interactions (Gao, [Col. 33, Lines 55-67, Col. 34, Lines 1-15]).
Zhang further discloses:
determining text associated with the target language ([0029] target text to be synthesized and an identifier of a speaker are obtained, [0030] the text to be synthesized may be any text in any language. The language may be, for example, Chinese, English, Japanese, and the like, [0036] a target language to which the target text belongs).
Gao further discloses:
generating, using a vocoder of one or more models and based at least on the one or more first voice characteristics, the one or more second voice characteristics, and the text, audio data representative the text spoken in the target language ([Fig. 6, Neural Vocoder 650], [Col. 35, Lines 30-40] the input text may first be converted to linguistic features. Then, the linguistic features, along with any conditional features such as style embedding 615 or speaker ID, may be used to produce the prosody features, such as duration and F0. Linguistic features combined with prosody features may then be used to generate spectral acoustic features and, at the last stage, conditional neural vocoder 650 takes in the spectral features to synthesize an output audio waveform 660, [As can be seen in Fig. 6, inputting linguistic features, i.e. first voice characteristics, wherein the linguistic features are based on text, and a style embedding, i.e. second voice characteristics, indicates generated waveform output, i.e. a visualization of audio data, which is dependent upon first and second voice characteristics in addition to text]).
Zhang further discloses:
causing output of the generated audio data ([0040] the output is a synthesized speech [Output of synthesized speech from an acoustic spectrum indicates speech generated from audio data, i.e. the acoustic spectrum, in view of the waveform of Gao]).
Regarding claim 2, Zhang in view of Gao discloses: the method of claim 1.
Zhang further discloses:
determining, using the one or more models ([In view of the previously disclosed speech synthesis model of Zhang]) and based at least on the one or more first voice characteristics, the text associated with the target language, and the one or more second voice characteristics corresponding to the second accent, one or more features associated with the speech ([0073] the target text to be synthesized and the identifier of the speaker are obtained; the pronunciation information of at least one character included in the target text is obtained; phonemes contained in the at least one character, and tones for syllables or words combined by the phonemes are determined based on the pronunciation information of at least one character in the target text; the suffix is added to each of the phonemes based on the type of the target language to which the target text belongs, and tone encoding of the tones is determined; feature items in the linguistic features are generated based on the suffixed phonemes and the tone encoding [Obtaining target text, identification of the speaker (represented through characteristic embeddings), and pronunciation, i.e. accent, information to be used to generate features indicates a determination of features based on characteristics, text, and accent inputs, wherein they are associated with a target language]);
wherein the generating the audio data is based at least on the one or more features associated with the speech ([0073] speech synthesis is performed based on the linguistic features of the target text and the identifier of the speaker to obtain a target speech).
Regarding claim 4, Zhang in view of Gao discloses: the method of claim 1.
Zhang further discloses:
determining an identity associated with the first speaker ([0023] an identifier of a speaker are first obtained);
wherein the receiving the first input data is based on the identity ([0099] speaker encoding corresponding to the speakers may be obtained by inputting the speaker identifiers of the speech samples into the second encoder of the training model, [Determining a speaker encoding indicates that encoding is representative of a plurality of identity characteristics, i.e. first voice characteristics, associated with the voice of the first speaker based on a determined identity]).
Regarding claim 5, Zhang in view of Gao discloses: the method of claim 1.
Gao further discloses:
wherein the one or more speech characteristics associated with the voice of the speaker are learned by disentangling the first accent from second speech made by the speaker in a second language ([Col. 31, Line 25] to train the multi-style TTS system, [Col. 32, Lines 5-10] After training the style embedding, a TTS voice may be trained alone with that style embedding to allow disentangling of speech content and style [Performing this operation after training indicates a speech characteristic learning operation to know how to disentangle the style from words, wherein the training is performed for a “multi-style” TTS system, indicating at least a first accent/language and an additional, i.e. second, accent (style) and/or language, ([Col. 19, Lines 55-65]) of Gao describe a translation task]).
Regarding claim 7, Zhang in view of Gao discloses: the method of claim 1.
Zhang further discloses:
generating, based at least on the text associated with the target language ([In view of the previously disclosed text associated with the target language of Zhang]), an alignment between one or more embeddings associated with the text and one or more phonemes ([0094] The acoustic text encoding may be input into the attention mechanism module, and may be aligned with the linguistic features input into the first encoder via the attention mechanism module, so as to obtain prosody information, [0085] prosody information when the speaker narrates the target text, i.e., the cadence and prosody of the speaker when narrating the target text, which is a macro-reflection of… a duration, [0023] linguistic features of the target text are generated by performing feature extraction on the pronunciation information of the at least one character [An encoding tracks to an embedding, Linguistic features are representative of phonemes, i.e. pronunciations]);
wherein the generating the audio data is further based at least on the alignment ([Fig. 7, Speaker Encoding], [0082] speaker encoding of the speaker is obtained by inputting the identifier of the speaker, [Speaker encodings track to characteristics], [Fig. 7, Style Encoding], [0085] The style encoding may describe the prosody information¸ [Prosody information can be reasonably understood to represent a rate/manner of speaking, i.e. an alignment between embeddings, e.g. graphemes, and timing of those spoken representations of graphemes, e.g. phonemes], [Fig. 7, Text Encoding], [0038] feature extraction may be performed on the pronunciation information of at least one character in the target text to generate linguistic features of the target text, [0049] pronunciation information may include information such as a phoneme, a syllable, a word, a tone, a stress, and a rhotic accent, [0099] Text Encoding corresponding to the linguistic features of the text samples, [Performing extraction on information which contains accent information indicates the text encoding is representative of accent. Further, sending all of this information into a decoder resulting in an acoustic spectrum indicates a generating of audio data using identity, alignment between embeddings and phonemes, and accent, wherein the accent is a second accent in view of the multiple accents defined in Zhang/Gao]).
Regarding claim 8, Zhang in view of Gao discloses: the method of claim 7.
Gao further discloses:
determining, based at least on the text associated with the target language and the one or more second voice characteristics corresponding to the second accent, one or more durations associated with one or more phonemes corresponding to the text ([Fig. 6, Duration of Prosody Model 630 which receives Linguistic Features based on Text 620 and Style Embedding 615], [Determining a duration of features, wherein the input to the duration detection is text and style embedding, i.e. second voice characteristics]),
wherein generating the audio data is further based at least on the one or more durations ([Fig. 6, Duration sent from Prosody Models 630 to Acoustic Models 640 which are used to generation output Waveform 660]).
Regarding claim 10, Zhang in view of Gao discloses: the method of claim 1.
Zhang further discloses:
wherein the speech preserves at least one of a voice associated with the first speaker or a timbre associated with the first speaker ([0041] for a Chinese-speaking speaker A, a target speech of the target text spoken by speaker A in English may be obtained by performing speech synthesis based on the identifier of speaker A and linguistic features of the target text in English, [0083] wherein the speaker encoding may describe the timbre feature of the speaker [Gathering a target speech based on the identifier of the speaker, i.e. speaker A, to generate a translated speech spoken by speaker A indicates a preservation of at least a voice of speaker A, which could be reasonably considered to also contain timbre information, i.e. the generated voice of a speaker will inherently have timbre information associated with the generation, wherein the same speaking voice indicates the same vocal timbre as well. Further, generating a spectrum based on timbre, i.e. speaker encoding (see Fig. 7), indicates that timbre information is preserved]).
Regarding claim 11, Zhang in view of Gao discloses: the method of claim 1.
Zhang further discloses:
wherein the generating the audio data is performed without second audio data associated with the first speaker in the target language ([Fig. 7, Text Encoding, Speaker Encoding, Style Encoding], [In view of the system of Fig. 7 of Zhang, which only receives speaker encodings, i.e. timbre information, text encoding, i.e. linguistic pronunciation information, and style encoding, i.e. prosody, as inputs to generate an acoustic spectrum, i.e. audio data representative of speech, indicating this operation is performed without additional, second audio data, i.e. additional, second spectrums]).
Regarding claim 12, Zhang discloses: a system comprising:
one or more processors ([0132] computing unit 1101 include, but are not limited to, a central processing unit (CPU)) to:
determine one or more voice characteristics of a first speaker ([0090] The speaker identifier is input into the style encoder, to obtain a corresponding style feature for the speaker, [0099] The style encoding (output by the style network) may be obtained by inputting the linguistic features of the text samples and the speaker identifiers of the speech samples into the style network, [Generating style features, i.e. characteristics, of a speaker based on a speaker identifier indicates the features are determined corresponding to a voice of a first speaker, i.e. represented through the identifier]).
Zhang does not disclose:
the one or more voice characteristics associated with disentangling a first accent of the first speaker; and,
determine a second accent learned using one or more instances of speech from one or more second speakers, the one or more instances of speech being in a target language; and,
determine, using one or more models and based at least on the second accent and text, one or more durations associated with one or more phonemes corresponding to text.
Gao discloses:
the one or more voice characteristics associated with disentangling a first accent of the first speaker ([Col. 32, Lines 5-10] After training the style embedding, a TTS voice may be trained alone with that style embedding to allow disentangling of speech content and style, [In view of the previously disclosed style encodings of Zhang which could be substituted for the style embedding of Gao without a change in functionality to the disentangling of speech content and style of Gao]); and,
determine a second accent learned using one or more instances of speech from one or more second speakers, the one or more instances of speech being in a target language ([Col. 32, Lines 65-67] for example, the output may choose between a British or an American accent), [Col. 41, Lines 10-15] At step 820, the assistant system 140 may determine a first style of the voice input based on one or more first features extracted from the voice input and/or the input text. Examples of these first features may include volume, emphasis, pronunciation, accent, [In view of the two accents claimed, i.e. British/American, indicating either of these two accents to be a second accent from a second speaker as compared to voice input received for the other accent]), the one or more instances of speech being in a target language ([In view of the above excerpt regarding choosing an output style, indicating the chosen style to be a target style with an associated target language, i.e. English, see “user-specific customization” defined in Col. 32, Lines 60-65 indicating the user to be specifying the target language]); and
determine, using one or more models and based at least on the second accent and text, one or more durations associated with one or more phonemes corresponding to text ([Fig. 6, Duration of Prosody Model 630 which receives Linguistic Features based on Text 620 and Style Embedding 615], [Determining a duration of features, wherein the input to the duration detection is text and style embedding, i.e. second voice characteristics corresponding to a second accent. Further, wherein Gao discloses grapheme-to-phoneme (as would be applied to input text [Col. 12, Lines 20-20]) indicating the duration is with respect to phonemes]).
Zhang and Gao are considered analogous art within text-to-speech systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang to incorporate the teachings of Gao, because of the novel way to use a semi-supervised approach for determining style representations for synthetic speech, improving the communication abilities of human-machine speech interactions (Gao, [Col. 33, Lines 55-67, Col. 34, Lines 1-15]).
Zhang further discloses:
determine, using one or more models ([0074] a speech synthesis model may be used to perform speech synthesis based on the linguistic features of the target text and the identifier of the speaker) and based at least on the one or more durations associated with the one or more phonemes, one or more features associated with speech ([0073] the target text to be synthesized and the identifier of the speaker are obtained; the pronunciation information of at least one character included in the target text is obtained; phonemes contained in the at least one character, and tones for syllables or words combined by the phonemes are determined based on the pronunciation information of at least one character in the target text; the suffix is added to each of the phonemes based on the type of the target language to which the target text belongs, and tone encoding of the tones is determined; feature items in the linguistic features are generated based on the suffixed phonemes and the tone encoding [Obtaining target text, identification of the speaker, i.e. characteristics, and pronunciation, i.e. accent, information to be used to generate features indicates a determination of features based on identity, text, and accent inputs, wherein they are associated with a target language (with an associated second accent), wherein any of the pronunciation information can track to variables as previously disclosed, i.e. adding suffixes to pronunciation information, i.e. phonemes, indicates the suffix is a determined feature based on the pronunciation]);
determine, using the one or more models and based at least on the one or more durations associated with the one or more phonemes and the one or more features ([In view of the previously disclosed models and features of Zhang]), audio data representative of the speech in the target language and spoken based at least on the second accent ([0088] The outputs of the first encoder, the second encoder and the style network are connected to the input of the decoder. The inputs to the speech synthesis model may be linguistic features of the text and a speaker identifier (i.e., Input Speaker), and the output may be an acoustic spectrum of a speech [An acoustic spectrum tracks to audio data representative of speech associated with text in the target language, wherein the target language will inherently have a second accent as compared to the original language]); and
cause output of the speech represented by the audio data ([0040] the output is a synthesized speech [Output of synthesized speech from an acoustic spectrum indicates speech generated from audio data, i.e. the acoustic spectrum]).
Regarding claim 15, Zhang in view of Gao discloses: the system of claim 12.
Zhang further discloses:
wherein the one or more processors are further to:
determine an identity associated with the first speaker ([0023] an identifier of a speaker are first obtained, [An identifier of a speaker is representative of the speaker’s identity]);
wherein the one or more voice characteristics of the first speaker are based at least on the identity associated with the first speaker ([0099] speaker encoding corresponding to the speakers may be obtained by inputting the speaker identifiers of the speech samples into the second encoder of the training model, [Determining a speaker encoding indicates that encoding is representative of a plurality of identity characteristics associated with the voice of the speaker]).
Regarding claim 16, Zhang in view of Gao discloses: the system of claim 15.
Gao further discloses:
wherein the one or more voice characteristics of the first speaker are learned by disentangling the first accent from second speech made by the first speaker in a second language ([Col. 31, Line 25] to train the multi-style TTS system, [Col. 32, Lines 5-10] After training the style embedding, a TTS voice may be trained alone with that style embedding to allow disentangling of speech content and style [Performing this operation after training indicates a speech characteristic learning operation to know how to disentangle the style from words, wherein the training is performed for a “multi-style” TTS system, indicating at least a first accent/language and an additional, i.e. second, accent (style) and/or language, ([Col. 19, Lines 55-65]) of Gao describe a translation task]).
Regarding claim 18, Zhang in view of Gao discloses: the system of claim 12.
Zhang further discloses:
wherein the one or more processors are further to:
determine an alignment between one or more embeddings associated with text and one or more phonemes based at least on the second accent associated with the target language ([0086] At 607, the style encoding, the text encoding and the speaker encoding are fused to obtain fused encoding, [A fusing operation is equivalent to an alignment operation, wherein the alignment between embeddings and phonemes is represented as the style encoding, and the accent information is based on the text encoding (see Fig. 7), in view of the plurality of languages defined in Zhang, any of which can represent a target language with an inherent associated accent]),
wherein the one or more features are further determined based at least on the alignment ([In view of the previously disclosed excerpt of Zhang, it is apparent that the fused encodings are representative of the features determined through the fusing, i.e. alignment]).
Regarding claim 23, Zhang in view of Gao discloses: the method of claim 1.
Gao further discloses:
wherein the speech refrains from being spoken using the first accent of the first speaker based at least on the one or more first voice characteristics being associated with disentangling the first accent ([Fig. 8, 840 “Generate the voice response based on one or more of the second features of the second [st]yle”], [Generating a voice response having a second style with associated second features indicates the response to refrain from being spoken using the first accent of the first speaker based on the second characteristics]).
Regarding claim 24, Zhang in view of Gao discloses: the system of claim 12.
Gao further discloses:
wherein the speech refrains from being spoken using the first accent of the first speaker based at least on the one or more voice characteristics being associated with disentangling the first accent ([Fig. 8, 840 “Generate the voice response based on one or more of the second features of the second [st]yle”], [Generating a voice response having a second style with associated second features indicates the response to refrain from being spoken using the first accent of the first speaker based on the second characteristics]).
Claim(s) 3, 13, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Gao, further in view of Finkelstein et al. (US-20220051654-A1), hereinafter Finkelstein.
Regarding claim 3, Zhang in view of Gao discloses: the method of claim 2.
Zhang in view of Gao does not disclose:
wherein the one or more features associated with the speech comprise one or more of:
one or more energies associated with the speech;
one or more frequencies associated with the speech; or
one or more phoneme durations associated with the speech.
Finkelstein discloses:
wherein the one or more features associated with the speech comprise one or more of:
one or more energies associated with the speech ([0046] The model 222a may jointly predict, for each syllable of given input text 320, a duration of the syllable and pitch (F0) and energy (C0) contours for the syllable);
one or more frequencies associated with the speech ([0043] generates an intermediate output audio signal 201 that may include a sequence of mel-frequency spectrograms inherently possessing the intended prosody for the input text utterance 320 [Generating mel-frequency spectrograms indicates a required frequency feature identification to generate the spectrogram]); or
one or more phoneme durations associated with the speech ([0034] the prosodic features may include phoneme durations and fixed-length frames of pitch and energy sampled from the reference audio signal [Wherein the input audio of Zhang could be used as the reference audio signal of Finkelstein without a change in functionality to the feature determination of Finkelstein]).
Zhang, Gao, and Finkelstein are considered analogous art within speech prosody transfer for generating synthetic speeches. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang in view of Gao, to incorporate the teachings of Finkelstein, because of the novel way to produce realistic, synthesized speech using textual input and prosody features not conveyed in textual input, but determined through domain/identity, allowing for multiple, diverse representations of output based on the same input (Finkelstein, [0002]).
Regarding claim 13, Zhang in view of Gao discloses: the system of claim 12.
Zhang in view of Gao does not disclose:
wherein the one or more features associated with the speech comprise one or more of:
one or more energies associated with the speech; or
one or more frequencies associated with the speech.
Finkelstein discloses:
wherein the one or more features associated with the speech comprise one or more of:
one or more energies associated with the speech ([0046] The model 222a may jointly predict, for each syllable of given input text 320, a duration of the syllable and pitch (F0) and energy (C0) contours for the syllable); or
one or more frequencies associated with the speech ([0043] generates an intermediate output audio signal 201 that may include a sequence of mel-frequency spectrograms inherently possessing the intended prosody for the input text utterance 320 [Generating mel-frequency spectrograms indicates a required frequency feature identification to generate the spectrogram]).
Zhang, Gao, and Finkelstein are considered analogous art within speech prosody transfer for generating synthetic speeches. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang in view of Gao, to incorporate the teachings of Finkelstein, because of the novel way to produce realistic, synthesized speech using textual input and prosody features not conveyed in textual input, but determined through domain/identity, allowing for multiple, diverse representations of output based on the same input (Finkelstein, [0002]).
Regarding claim 20, Zhang in view of Gao discloses: the system of claim 12.
Zhang further discloses:
wherein the system is comprised in at least one of:
a system for performing simulation operations ([0140] artificial intelligence is a study of making computers to simulate certain thinking processes);
a system for performing deep learning operations ([0139] The disclosure relates to a field of computer technologies, particularly to a field of artificial intelligence technologies such as deep learning);
a system for performing conversation AI operations ([0140] artificial intelligence software technologies mainly include computer vision, speech recognition technology, natural language processing technology [Natural language processing and speech recognition are reasonably considered to be conversational operations]);
a system for generating synthetic data ([0004] performing speech synthesis [Speech/audio is a form of data]); and,
a system implemented at least partially using cloud computing resources ([0138] The cloud server is also known as a cloud computing server or cloud host).
Gao further discloses:
wherein the system is comprised in at least one of:
a system for performing voice conferencing ([Col. 38, Lines 54-55] Speech queries from multiple speakers were recorded by letting them read the queries freely in a conference room); and,
a system implemented at least partially in a data center ([Col. 54, Lines 64-65] span multiple data centers).
Zhang in view of Gao does not disclose:
wherein the system is comprised in at least one of:
a system implementing a gaming application.
Finkelstein discloses:
wherein the system is comprised in at least one of:
a system implementing a gaming application ([0071] gaming applications).
Zhang, Gao, and Finkelstein are considered analogous art within speech prosody transfer for generating synthetic speeches. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang in view of Gao to incorporate the teachings of Finkelstein, because of the novel way to produce realistic, synthesized speech using textual input and prosody features not conveyed in textual input, but determined through domain/identity, allowing for multiple, diverse representations of output based on the same input (Finkelstein, [0002]).
The examiner would like to note that due to the disjunctive nature of the claim, not all systems defined in the claim require a mapping.
Claim(s) 9, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Gao, further in view of Pekar et al. (US-20220189455-A1), hereinafter Pekar.
Regarding claim 9, Zhang in view of Gao discloses: the method of claim 1.
Zhang further discloses:
wherein the generating the audio data comprises:
generating, using a decoder of the one or more models and based at least on the one or more first voice characteristics, the text associated with the target language, and the one or more second voice characteristic corresponding to the second accent ([Fig. 7, Decoder], [A decoder which receives a speaker encoding, i.e. identity characteristics, text encoding, i.e. text, and a style encoding, i.e. accent]), one or more spectrograms associated with the speech ([0088] The outputs of the first encoder, the second encoder and the style network are connected to the input of the decoder. The inputs to the speech synthesis model may be linguistic features of the text and a speaker identifier (i.e., Input Speaker), and the output may be an acoustic spectrum of a speech. The acoustic spectrum, for example, may be a Mel spectrum, [A mel spectrum tracks to a mel-spectrogram]).
Zhang in view of Gao does not disclose:
generating, using the vocoder of the one or more models and based at least on the one or more spectrograms, the audio data representative of the speech associated with the text in the target language and spoken based at least on the second accent.
Pekar discloses:
generating, using a vocoder of the one or more models and based at least on the one or more spectrograms, the audio data ([0039] Based on the input, the trained neural network determines or generates, as output, a set of phone durations and a set of acoustic features (e.g., fundamental frequency, mel-spectrogram, spectral envelope, or the like). The set of phone durations and the set of acoustic features may be provided as input to a vocoder (i.e., voice encoder) for synthesizing a waveform for a speech signal [“phone” tracks to a phoneme], [0039] The vocoder generates the waveform that corresponds to an audio of the speech sample, included in the received first request. The translated audio is in the target in the target language in the voice of the target speaker, [0046] The synthesis server 106 may receive a request for speech synthesis that includes a target text document and a speaker ID of the target speaker (i.e., one of the Indian speakers). Based on the received request, the synthesis server 106 generates speech in American English in a voice of the Indian speaker, [A target, i.e. second, language (American English) in the voice of a target speaker as compared to the plurality of possible accents disclosed in [0028]-[0029] indicates the Indian accent to be a second accent compared to any of the other defined accents]).
Zhang, Gao, and Pekar are considered analogous art within cross-lingual speech synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang in view of Gao to incorporate the teachings of Pekar, because of the novel way to separately represent all phonemes across languages as one-hot vectors, achieving dimensional reduction of data, further allowing for improved phoneme matching across languages without expert knowledge (Pekar, [0022]).
Regarding claim 19, Zhang in view of Gao discloses: the system of claim 12.
Zhang further discloses:
wherein the generating the audio data comprises:
generating, using a decoder of the one or more models and based at least on the one or more durations associated with the one or more phonemes and the one or more features ([Fig. 7, Decoder], [A decoder which receives a speaker encoding, i.e. identity, text encoding, i.e. text, and a style encoding, i.e. accent, any of which can represent features as previously disclosed]), one or more spectrograms associated with the speech ([0088] The outputs of the first encoder, the second encoder and the style network are connected to the input of the decoder. The inputs to the speech synthesis model may be linguistic features of the text and a speaker identifier (i.e., Input Speaker), and the output may be an acoustic spectrum of a speech. The acoustic spectrum, for example, may be a Mel spectrum, [A mel spectrum tracks to a mel-spectrogram]).
Zhang in view of Gao does not disclose:
generating, using a vocoder of the one or more models and based at least on the one or more spectrograms, the audio data representative of the speech in the target language and spoken based at least on the second accent.
Pekar discloses:
generating, using a vocoder of the one or more models and based at least on the one or more spectrograms ([0039] Based on the input, the trained neural network determines or generates, as output, a set of phone durations and a set of acoustic features (e.g., fundamental frequency, mel-spectrogram, spectral envelope, or the like). The set of phone durations and the set of acoustic features may be provided as input to a vocoder (i.e., voice encoder) for synthesizing a waveform for a speech signal [“phone” tracks to a phoneme]), the audio data representative of the speech in the target language and spoken based at least on the second accent ([0039] The vocoder generates the waveform that corresponds to an audio of the speech sample, included in the received first request. The translated audio is in the target in the target language in the voice of the target speaker, [0046] The synthesis server 106 may receive a request for speech synthesis that includes a target text document and a speaker ID of the target speaker (i.e., one of the Indian speakers). Based on the received request, the synthesis server 106 generates speech in American English in a voice of the Indian speaker, [A target, i.e. second, language (American English) in the voice of a target speaker as compared to the plurality of possible accents disclosed in [0028]-[0029] indicates the Indian accent to be a second accent compared to any of the other defined accents]).
Zhang, Gao, and Pekar are considered analogous art within cross-lingual speech synthesis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang in view of Gao to incorporate the teachings of Pekar, because of the novel way to separately represent all phonemes across languages as one-hot vectors, achieving dimensional reduction of data, further allowing for improved phoneme matching across languages without expert knowledge (Pekar, [0022]).
Claim(s) 21-22, 25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Gao, further in view of Elias et al. (US-20220122582-A1), hereinafter Elias.
Regarding claim 21, Zhang discloses:
one or more processors ([0132] a graphics processing unit (GPU), various dedicated AI computing chips, various computing units that run machine learning model algorithms, and a digital signal processor (DSP), and any appropriate processor, controller and microcontroller) comprising processing circuitry to:
receive one or more voice characteristics of a first speaker ([0090] The speaker identifier is input into the style encoder, to obtain a corresponding style feature for the speaker, [0099] The style encoding (output by the style network) may be obtained by inputting the linguistic features of the text samples and the speaker identifiers of the speech samples into the style network, [Generating style features, i.e. characteristics, of a speaker based on a speaker identifier indicates the features are determined corresponding to a voice of a first speaker, i.e. represented through the identifier]).
Zhang does not disclose:
the one or more voice characteristics associated with disentangling a first accent of the first speaker; and,
receive a second accent learned using one or more instances of speech from one or more second speakers, the one or more instances of speech being in a target language.
Gao discloses:
the one or more voice characteristics associated with disentangling a first accent of the first speaker ([Col. 32, Lines 5-10] After training the style embedding, a TTS voice may be trained alone with that style embedding to allow disentangling of speech content and style, [In view of the previously disclosed style encodings of Zhang which could be substituted for the style embedding of Gao without a change in functionality to the disentangling of speech content and style of Gao]); and,
receive a second accent learned using one or more instances of speech from one or more second speakers ([Col. 32, Lines 65-67] for example, the output may choose between a British or an American accent), [Col. 41, Lines 10-15] At step 820, the assistant system 140 may determine a first style of the voice input based on one or more first features extracted from the voice input and/or the input text. Examples of these first features may include volume, emphasis, pronunciation, accent, [In view of the two accents claimed, i.e. British/American, indicating either of these two accents to be a second accent from a second speaker as compared to voice input received for the other accent]), the one or more instances of speech being in a target language ([In view of the above excerpt regarding choosing an output style, indicating the chosen style to be a target style with an associated target language, i.e. English, see “user-specific customization” defined in Col. 32, Lines 60-65 indicating the user to be specifying the target language]).
Zhang and Gao are considered analogous art within text-to-speech systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang to incorporate the teachings of Gao, because of the novel way to use a semi-supervised approach for determining style representations for synthetic speech, improving the communication abilities of human-machine speech interactions (Gao, [Col. 33, Lines 55-67, Col. 34, Lines 1-15]).
Zhang further discloses:
generate, using a decoder of one or more models and based at least on the one or more voice characteristics and the second accent and text, one or more spectrograms associated with speech corresponding to the text ([0102] obtains the acoustic spectrum of the target speech by decoding the fused encoding with the decoder of the speech synthesis model, [0088] The acoustic spectrum, for example, may be a Mel spectrum, [Generating an acoustic spectrum, i.e. mel-spectrogram, based on fused encodings, wherein the fused encodings combine characteristics at least characteristics and accent, wherein the input to Zhang is text, indicates the spectrogram is generated based on at least the characteristics and second accent and text]); and,
cause output of the speech represented by the audio data ([0040] the output is a synthesized speech [Output of synthesized speech from an acoustic spectrum indicates speech generated from audio data, i.e. the acoustic spectrum]).
Zhang in view of Gao does not disclose:
receive, using the one or more models and based at least on the second accent, an alignment between one or more portions of the text and the one or more spectrograms associated with the speech.
Elias discloses:
receive, using the one or more models and based at least on the second accent, an alignment between one or more portions of the text and the one or more spectrograms associated with the speech ([0047] the phoneme-level fine-grained VAE network may align the reference mel-frequency spectrogram sequence 202 with each phoneme in a sequence of phonemes extracted from the input text sequence 206 and encode a sequence of phoneme-level variational embeddings 220, [Aligning a spectrogram with phonemes extracted from a text sequence indicates the alignment to be between the spectrogram and portions of text (represented as phonemes)]).
Zhang, Gao, and Elias are considered analogous art within text-to-speech which disentangles style/prosody information from content. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang in view of Gao to incorporate the teachings of Elias, because of the novel way to use a non-autoregressive neural TTS model augmented with a variational autoencoder for synthesizing speech, allowing for the disentangling of latent representations/states from reference audio which cannot be represented by input text, enabling for synthesized speech produced through TTS systems to sound more similar to a reference input audio. Further, it would have been obvious to include mel-frequency spectrograms as input to the synthesizer so that the TTS model can be trained to convert input text utterances into sequence having intended style learned via a prior variational embedding, reducing the amount of required data to convert styles of input text in deployment (Elias, [0036]-[0037]).
Gao further discloses:
generate, using a vocoder of the one or more models and based at least on the alignment and the one or more spectrograms, audio data representative of the speech associated with the text in the target language and spoken based at least on the second accent ([Col. 32, Line 37] neural vocoders may output a speech waveform, [Col. 32, Lines 45-50] the style extraction unit may be added to the prosody model, though it could be added directly to the acoustic model or to any subsequent model, [In view of the vocoder being part of the acoustic model, see Col. 32, Lines 35-40, indicating the speech waveform, i.e. in view of the spectrogram of Zhang, generated is based at least on the style embedding containing a second accent in a second language (see translation task of [Col. 19, Lines 55-65]) which can be added to the acoustic model]).
Regarding claim 22, Zhang in view of Gao, further in view of Elias discloses: the one or more processors of claim 21.
Zhang further discloses:
wherein the one or more processors are comprised in at least one of:
a system for performing simulation operations ([0140] artificial intelligence is a study of making computers to simulate certain thinking processes);
a system for performing deep learning operations ([0139] The disclosure relates to a field of computer technologies, particularly to a field of artificial intelligence technologies such as deep learning);
a system for performing conversation AI operations ([0140] artificial intelligence software technologies mainly include computer vision, speech recognition technology, natural language processing technology [Natural language processing and speech recognition are reasonably considered to be conversational operations]);
a system for generating synthetic data ([0004] performing speech synthesis [Speech/audio is a form of data]); and,
a system implemented at least partially using cloud computing resources ([0138] The cloud server is also known as a cloud computing server or cloud host).
Gao further discloses:
wherein the system is comprised in at least one of:
a system for performing voice conferencing ([Col. 38, Lines 54-55] Speech queries from multiple speakers were recorded by letting them read the queries freely in a conference room); and,
a system implemented at least partially in a data center ([Col. 54, Lines 64-65] span multiple data centers).
The examiner would like to note that due to the disjunctive nature of the claim, not all systems defined in the claim require a mapping.
Regarding claim 25, Zhang in view of Gao, further in view of Elias discloses: the one or more processors of claim 21.
Zhang further discloses:
wherein the one or more processors are further to:
determine text associated with the target language ([0034] target text may be obtained by querying based on the target language to which the target text belongs),
wherein the one or more spectrograms are further generated based at least on the text ([0098] decoding the fused output by a decoder in the training model to obtain a predicted acoustic spectrum, [In view of the fused encoding of Fig. 7 which has a clear dependence upon the text encoding]), and wherein the speech is associated with the text ([0113] target text to be synthesized, [Target text to be synthesized indicates the synthesized speech to be associated with the target text]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wang et al. (US-20240274122-A1) discloses “An expressive speech translation system may process source speech in a source language and output synthesized speech in a target language while retaining vocal performance characteristics such as intonation, emphasis, rhythm, style, and/or emotion. The system may receive a transcript of the source speech, translate it, and generate transcript data. To generate the synthesized speech, the system may process the transcript data with a language embedding representing language-dependent speech characteristics of the target language, a speaker embedding representing speaker-dependent voice identity characteristics of a speaker, and a performance embedding representing the vocal performance characteristics of the source speech. The system may control the duration of segments of the synthesized speech to better align with corresponding segments of the source speech for the purpose of dubbing multimedia content with synthesized speech in a language different from that of the original audio.” (abstract). See entire document.
Li et al. (US-20250349282-A1) discloses “Systems and methods are provided for machine learning models configured as zero-shot personalized text-to-speech models which comprise a feature extractor, a speaker encoder, and a text-to-speech module. The feature extractor is configured to extract acoustic features and prosodic features from new target reference speech associated with the new target speaker. The speaker encoder is configured to generate a speaker embedding corresponding to the new target speaker based on the acoustic features extracted from the new target reference speech. The text-to-speech module is configured to generate the personalized voice corresponding for the new target speaker based on the speaker embedding and the prosodic features extracted from the new target reference speech without applying the text-to-speech module on new labeled training data associated with the new target speaker” (abstract). See entire document.
Wang et al. (“Accent and Speaker Disentanglement in Many-to-many Voice Conversion”) discloses “we use accent-dependent speech recognizers to obtain bottleneck features for different accented speakers. This aims to wipe out other factors beyond the linguistic information in the BN features for conversion model training. Second, we propose to use adversarial training to better disentangle the speaker and accent information in our encoder-decoder based conversion model. Specifically, we plug an auxiliary speaker classifier to the encoder, trained with an adversarial loss to wipe out speaker information from the encoder output” (abstract). See entire document.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THEODORE JOHN WITHEY whose telephone number is (703)756-1754. The examiner can normally be reached Monday - Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571) 272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THEODORE WITHEY/Examiner, Art Unit 2655 /ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655
Prosecution Insights