DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments with respect to 35 U.S.C. 103 in regards to claims 1 and 11 have been considered, however are not found to be persuasive due to the following reasons. Applicant argues that the newly added limitations are not taught by Zhang nor Rezende, however, Examiner respectfully disagrees because Zhang’s reference encoder/recognition model processes reference audio into latent variables z/style representation. Zhang’s fully connected (FC) layers are MLP-type feed-forward layers. Further, Zhang clearly teaches liner activation by stating “The output, which denotes some embedding of the reference audio, is then passed through two separate fully connect (FC) layer with linear activation function to generate the mean and standard deviation of latent variables z.” See detailed rejection below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-9, 11-14 and 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (“Learning Latent Representations for Style control and transfer in End-to-End Speech Synthesis”; May 12, 2019; pgs. 6945-6949) in view of Rezende et al. (“Variational Interference with Normalizing Flows”; Jun. 1, 2015).
Claims 1 and 11,
Zhang teaches a computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising ([Introduction] [2.2] Zhang describes end-to-end TTS models which generate speech directly from characters and introduces VAE into end-to-end TTS model; Zhang further states that one may use various powerful and complex neural networks for the recognition model):
receiving a reference utterance spoken by a reference speaker ([Introduction] [3.1] Zhang teaches that the latent representation of speaking style can be inferred from a reference audio, which then controls the style of synthesized speech and that style transfer, from reference audio to synthesized speech, is thus achieved; Zhang’s experiments use audiobook recordings dataset read with various storytelling styles by a single English speaker, and at inference we feed audio clips as reference and go though the recognition model);
processing, using a reference encoder comprising multilayer perceptron (MLP), the reference utterance to obtain a variational embedding by ([2.2] Zhang discloses a recognition model or inference network which encodes reference audio into a fixed length short vector of latent representation (or latent variables z which stand for style representation); Zhang further states: here, we only adopt a recurrent reference encoder followed by two fully connected layers; here, Zhang’s reference encoder/recognition model processes reference audio into latent variables z/style representation; Zhang’s fully connected layers are MLP-type feed-forward layers):
predicting, from the reference utterance, using the MLP having liner activation, a mean and standard deviation of latent variables of the reference encoder ([2.2] Zhang states: The output, which denotes some embedding of the reference audio, is then passed through two separate fully connected (FC) layers with linear activation function to generate the mean and standard deviation of latent variables); and
receiving, as input to a text encoder, an input text sequence characterizing a target utterance to be synthesized into expressive speech ([2.2] Zhang states: The input texts are character sequence; Zhang further states: The encoder which deals with character inputs consists of 3 1-D convolutional layers … followed by a bidirectional LSTM layer; Zhang’s model converts combined encoder states to generated target sentence with specific style);
generating, using an attention module, based on an output of the text encoder and the variational embedding, a context vector for each output step of a decoder ([2.2] Zhang states: The output text encoder state in simple added by z and then is consumed by a location-sensitive attention network which converts encoded sequence to a fixed-length context vector for each decoder output step; Zhang further notes that z is passed through an FC layer to make sure the dimension equal to text encoder state before add operation);
generating, using the decoder based the context vector generated for each output step, a sequence of spectrogram frames ([2.2] Zhang states that the acoustic features are mel-frequency spectrograms; Fig. 1 shows the Tacotron2 path including Attention, Decoder, and Mel spectrogram; Zhang also states that the attention module and encoder have the same architecture as Tacotron2 and that the model uses L2-loss of mel spectrogram as reconstruction loss); and
converting, using a synthesizer, the sequence of spectrogram frames into synthesized speech conveying the target utterance ([Introduction] [2.2] [3.3] Zhang states: Then WaveNet vocoder is utilized to reconstruct waveform; Zhang also describes Style transfer, from reference audio to synthesized speech; and explains that synthesized audios share the same input text).
The difference between the prior art and the claimed invention is that Zhang does not explicitly tech deriving, using normalizing flows based on the mean and standard deviation of the latent variables of the reference encoder, the variational embedding.
Rezende teaches deriving, using normalizing flows based on the mean and standard deviation of the latent variables of the reference encoder, the variational embedding ([Abstract] [Algorithm 1] [4.2] Rezende states: distribution constructed though a normalizing flow, whereby a simple initial density is transformed into a more complex one by applying a sequence of invertible transformations; Rezende further states: The density qK(z) obtained by successively transforming a random variable z0 with distribution q0 through a chain of K transformations fk is zK = fK … f1(z0), eq. 6; Rezende also teaches constructing an inference model using deep neural network to build a mapping from the observations x to the parameters of the initial density q0 = N(u, σ) … as well as the parameters of the follow λ; see Algorithm 1; Rezende teaches deriving latent samples via normalizing flows where an inference model maps observations to “q0 = N(u, σ)” and applies a sequence of transforms to obtain zK (algorithm 1)).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Zhang with teachings of Rezende by modifying the learning latent representations for style control and transfer in end-to-end speech synthesis as taught by Zhang to include deriving, using normalizing flows based on the mean and standard deviation of the latent variables of the reference encoder, the variational embedding as taught by Rezende for the benefit of improving the performance and applicability of variational inference (Rezende [Abstract]).
Claims 2 and 12,
Zhang further teaches the computer-implemented method of claim 1, wherein the reference encoder comprises a variational posterior ([2.1] approximate posterior; recognition model as posterior approximation and encoder role).
Claims 3 and 13,
Zhang further teaches the computer-implemented method of claim 1, wherein the synthesized speech comprises prosody characteristics of the reference utterance ([3.3-3.4] evaluate the performance of style transfer; pitch-height, pause time, speaking rate and pitch variation).
Claims 4 and 14,
Zhang further teaches the computer-implemented method of claim 1, wherein the reference utterance corresponds to a different utterance than the target utterance ([3.4] non-parallel transfer, 60 sentences of text and 6 other reference audio clips (uses different text vs reference audio)).
Claims 6 and 16,
Rezende further teaches the computer-implemented method of claim 1, wherein the variational embedding comprises a capacity represented by a number of bits ([Introduction] capacity of the posterior approximation can also result in biases in the MAP estimates of any model parameters (and this is the case e.g., in time-series models)).
Claims 7 and 17,
Zhang further teaches the computer-implemented method of claim 1, wherein the synthesizer comprises a neural vocoder ([2.2] WaveNet vocoder).
Claims 8 and 18,
Zhang further teaches the computer-implemented method of claim 1, wherein the synthesizer comprises a waveform synthesizer ([2.2] WaveNet vocoder is utilized to reconstruct waveform).
Claims 9 and 19,
Zhang further teaches the computer-implemented method of claim 1, wherein the decoder comprises a recurrent neural network ([2.2] complex neural network decoder (the decoder has the same architecture as Tacotron 2)).
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (“Learning Latent Representations for Style control and transfer in End-to-End Speech Synthesis”; May 12, 2019; pgs. 6945-6949) in view of Rezende et al. (“Variational Interference with Normalizing Flows”; Jun. 1, 2015) and further in view of Sotelo et al. (“Char2Wav: End-to-End Speech Synthesis”; ICLR 2017).
Claims 5 and 15,
Zhang and Rezende teach all the limitations in claim 1. The difference between the prior art and the claimed invention is that Zhang nor Rezende do not explicitly teach wherein the input text sequence comprises a sequence of phonemes.
Sotelo teaches wherein the input text sequence comprises a sequence of phonemes ([Abstract] text or phoneme as inputs).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Zhang with teachings of Sotelo by modifying the learning latent representations for style control and transfer in end-to-end speech synthesis as taught by Zhang to include wherein the input text sequence comprises a sequence of phonemes as taught by Sotelo for the benefit of producing audio directly from text (Sotelo [Abstract]).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHREYANS A PATEL whose telephone number is (571)270-0689. The examiner can normally be reached Monday-Friday 8am-5pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SHREYANS A. PATEL
Primary Examiner
Art Unit 2653
/SHREYANS A PATEL/ Examiner, Art Unit 2659
Prosecution Insights