DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim(s) 1-17 is/are pending and has/have been examined.
Examiner Note:
Regarding the analysis of the claims under 101, the claims have been identified as eligible. The combination of the acoustic model output, SSL model feature aggregation across layers, loss construction between representations, backpropagation update, and test-time execution, makes the claim interpretation a specific technical solution rather than conventional model training. As such, the claims are patent eligible.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 2, 4-6, 8-10, 12-14, 16, and 17 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Li et al. (US Patent No. 12,658,174), hereinafter Li.
Regarding claims 1, 9, and 17, Li teaches
(claim 1) A method for training a text-to-speech machine learning model (method (31:13-16)), the method comprising:
(claim 9) A system for training a text-to-speech machine learning model (a system (28:1-2)), comprising:
(claim 9) at least one memory (the system is run on a device that includes a memory (28:1-2,56-67)); and
(claim 9) at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to (the system is run on a device that includes a processor and a memory with executable instructions for the processor (28:1-2,56-67)):
(claim 17) A non-transitory computer readable medium storing thereon computer executable instructions for training a text-to-speech machine learning model (the system is run on a device that includes a processor and a memory with executable instructions for the processor (28:1-2,56-67)), including instructions for:
inputting training text into an acoustic model configured to generate an intermediate representation comprising predicted latent features for a vocoder model that further generates a waveform of speech reciting the training text (a training dataset includes target data representing speech sample audio and input data representing a transcript of the speech, i.e. training text, where the speech generation model processes the input data to generate output data that represents the acoustic features of speech, such as speech tokens that are an intermediate representation between words and audio data and are a projection of latent embedding data, i.e. generate an intermediate representation comprising predicted latent features, and the speech model further includes a decoder that receives the speech token data and generates predicted spectrogram data representing synthesized speech, and a vocoder that converts the predicted spectrogram data to audio waveform data, i.e. a vocoder model that further generates a waveform of speech reciting the training text (4:21-59),(9:5-12),(11:33-35),(12:45-49),(14:1-12),(16:61-66),(19:58-59));
inputting a target waveform into a self-supervising learning (SSL) model configured to generate a vector representation of the target waveform, wherein the target waveform is true speech reciting the training text (the speech tokenizer includes an autoencoder for self-supervised learning, i.e. a self-supervising learning (SSL) model, and may process the target input data that is speech sample audio, i.e. inputting a target waveform…true speech reciting the training text, to produce audio embedding data that is representative vectors that make up the speech tokens of the speech token data, i.e. generate a vector representation of the target waveform (4:21-59),(11:57-12:9),(13:51-14:14),(14:62-64));
extracting and summing SSL features from a plurality of layers of the SSL model (the speech tokenizer includes an audio encoder and quantizer, i.e. from a plurality of layers of the SSL model, where the audio encoder processes the target data to generate latent representations, i.e. SSL features, that are quantized into a finite set of discrete representative vectors to make up the speech tokens of the speech token data, i.e. extracting and summing SSL features (13:51-14:14),(16:61-67));
computing a loss between a sum of the SSL features from the SSL model and the predicted latent features from the acoustic model (the robustness testing component may compare speech token data output by the speech model, i.e. the predicted latent features from the acoustic model, to the speech token data generated from the target data, i.e. a sum of the SSL features from the SSL model, to calculate a loss based on the comparison, i.e. computing a loss between (4:40-59),(9:5-12),(13:15-14:14));
updating, using backpropagation, weights of the acoustic model based on the loss (the reconstruction loss is determined, i.e. based on the loss, and backpropagation is used to update parameters of the speech model, i.e. updating using backpropagation weights of the acoustic model (13:15-14:14),(15:27-65)); and
executing the acoustic model with the updated weights on a test text to generate the intermediate representation (a first portion of the training dataset may be used for training the speech generation models, and a second portion may be used as the evaluation dataset, i.e. test text, where the updated speech generation model may be evaluated, i.e. executing the acoustic model with the updated weights on a test text, to see if the loss from comparing the speech token data increases or decreases, i.e. to generate the intermediate representation (4:21-59),(5:65-6:30),(9:5-12),(14:1-12)).
Regarding claims 2 and 10, Li teaches claims 1 and 9, and further teaches
subsequent to updating the weights of the acoustic model, receiving re-predicted latent features from the acoustic model;
computing another loss between the sum of the SSL features and the re-predicted latent features (a first portion of the training dataset may be used for training the speech generation models, and a second portion may be used as the evaluation dataset, where the updated speech generation model may be evaluated, i.e. subsequent to updating the weights of the acoustic model, to see if the loss from comparing the speech token data, i.e. receiving re-predicted latent features from the acoustic model, for the evaluation data, i.e. computing another loss, which includes a comparison of the output data and the target data, i.e. between the sum of the SSL features and the re-predicted latent features, increases or decreases (4:21-59),(5:7-21),(5:65-6:30),(9:5-12),(13:15-14:14),(14:1-12),(15:27-65)); and
updating, using backpropagation, the weights of the acoustic model until the another loss is less than a threshold loss or a maximum number of iterations has been reached (if the loss increases, the system reverts to a previous version of the model and performs training that includes updating the parameters using backpropagation, i.e. updating using backpropagation the weights of the acoustic model, and when the loss decreases indicating model performance improves, the updated speech generation model is retained, i.e. until the another loss is less than a threshold loss (5:7-24),(5:65-6:30)).
Regarding claims 4 and 12, Li teaches claims 1 and 9, and Li further teaches
the plurality of layers comprises at least one intermediate layer and a final layer of the SSL model (the speech tokenizer includes an audio encoder, i.e. at least one intermediate layer, and quantizer, i.e. a final layer of the SSL model (13:51-14:14),(16:61-67)).
Regarding claims 5 and 13, Li teaches claims 4 and 12, and further teaches
selecting the at least one intermediate layer of the SSL model based on a type of data outputted by the at least one intermediate layer, wherein latent features corresponding to the type of data are to be summed (the speech tokenizer includes an audio encoder, i.e. at least one intermediate layer, and quantizer, where the audio encoder processes the target data to generate latent representations, i.e. latent features, and finetuning the speech model may involve using latent embedding data from a last transformer layer and speech embeddings from the audio encoder, i.e. selecting the at least one intermediate layer of the SSL model, instead of using the speech token data projected from the latent embedding data, i.e. based on a type of data outputted by the at least one intermediate layer (13:51-14:14),(16:61-17:27)).
Regarding claims 6 and 14, Li teaches claims 4 and 12, and further teaches
selecting the at least one intermediate layer of the SSL model based on a position of the at least one intermediate layer, wherein latent features from pre-determined positions in the acoustic model are to be summed (the speech tokenizer includes an audio encoder, i.e. at least one intermediate layer, and quantizer, where the audio encoder processes the target data to generate latent representations, i.e. latent features, and finetuning the speech model may involve using latent embedding data from a last transformer layer and speech embeddings from the audio encoder, i.e. selecting the at least one intermediate layer of the SSL model, which is right before the latent embeddings are projected into speech token data in the linear projection layer of the speech model or the quantizer of the speech tokenizer, i.e. based on a position of the at least one intermediate layer (13:51-14:14),(16:61-17:27)).
Regarding claims 8 and 16, Li teaches claims 1 and 9, and further teaches
the input text is converted into a phoneme sequence (the text encoder may generate text embedding data that represents the content of the input data as phonemes (10:27-39)).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3 and 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li, in view of Belonozhko et al. (“Features of the implementation of real time text to speech systems with data variability”, 6th International Youth Conference on Radio Electronics, Electrical and Power Engineering (REEPE), 09 April 2024), hereinafter Belonozhko.
Regarding claims 3 and 11, Li teaches claims 1 and 9, and further teaches
subsequent to updating the weights of the acoustic model, receiving re-predicted latent features from the acoustic model;
inputting the re-predicted latent features into the vocoder model to receive an output waveform (a first portion of the training dataset may be used for training the speech generation models, and a second portion may be used as the evaluation dataset, where the updated speech generation model may be evaluated, i.e. subsequent to updating the weights of the acoustic model, to see if the loss from comparing the output data and the target data increases or decreases for the evaluation data, where the output data may be an audio waveform generated by the vocoder from the predicted spectrogram data generated from the speech token data, i.e. inputting the re-predicted latent features into the vocoder model to receive an output waveform (4:21-59),(5:7-21),(5:65-6:30),(9:5-12),(11:33-35),(12:45-49),(13:15-14:14),(14:1-12),(15:27-65));
determining a difference between the output waveform and the target waveform (the robustness testing component may update the speech generation models based on a loss calculated using the audio waveform data, i.e. determining a difference between the output waveform and the target waveform (4:21-59),(13:15-50)).
While Li provides calculating a loss between audio waveforms for training a model, Li does not specifically teach updating the vocoder, and thus does not teach
updating, using backpropagation, weights of the vocoder model based on the difference until the difference is less than a threshold difference or a maximum number of iterations has been reached; and
executing the vocoder model with the updated weights on the intermediate representation associated with the test text to generate a test waveform of speech reciting the test text.
Belonozhko, however, teaches updating, using backpropagation, weights of the vocoder model based on the difference until the difference is less than a threshold difference or a maximum number of iterations has been reached (a loss function determined by calculating the difference between the prediction of the audio sample value to the actual audio sample, i.e. based on the difference, and backpropagation is used to update the models weights based on the loss function, i.e. updating using backpropagation weights of the vocoder model based on the difference, where the training process continues until the model achieves the desired accuracy, i.e. until the difference is less than a threshold difference (Sec. IV)); and
executing the vocoder model with the updated weights on the intermediate representation associated with the test text to generate a test waveform of speech reciting the test text (after training, i.e. vocoder model with the updated weights, the model can be used for speech synthesis by generating audio signals based on input text, i.e. executing the vocoder model…associated with the test text to generate a test waveform of speech reciting the test text, where the vocoder model receives a mel spectrogram output by the Tacotron-2 model, i.e. executing the vocoder model with the updated weights on the intermediate representation Figs. 1 and 2,(Sec. IV)).
Where Li teaches the training occurs until the loss decreases indicating model performance improves (5:7-24),(5:65-6:30), and that the process can be performed using an evaluation dataset (4:21-59),(5:65-6:30),(9:5-12),(14:1-12).
Li and Belonozhko are analogous art because they are from a similar field of endeavor in training TTS models. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the calculating a loss between audio waveforms for training a model teachings of Li with the specific training of the vocoder model as taught by Belonozhko. It would have been obvious to combine the references to enable the generation of high-quality audio signals with a high sampling frequency, that contributes to the naturalness of the synthesized speech (Belonozhko (Sec. IV, para 4)).
Claim(s) 7 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Li, in view of Chen et al. (“WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing”, IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 16, NO. 6, OCTOBER 2022), hereinafter Chen.
Regarding claims 7 and 15, Li teaches claims 6 and 14.
While Li provides an audio encoder and quantizer, Li does not specifically teach multiples of three layers, and thus does not teach
every third layer is comprised in the at least one intermediate layer of the SSL model.
Chen, however, teaches every third layer is comprised in the at least one intermediate layer of the SSL model (WavLM has a number of layers varying from 12 to 24, i.e. every third layer, where different layers contain information useful for different tasks, such as the hidden states of the top layer being useful for ASR, and bottom layers are more effective for speaker verification, i.e. comprised in the at least one intermediate layer of the SSL model (Intro, Sec. IIIA, Sec. VA)).
Li and Chen are analogous art because they are from a similar field of endeavor in training speech processing models. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the an audio encoder and quantizer teachings of Li with the specific structure and uses of WavLM as taught by Chen. It would have been obvious to combine the references to boost the performance of speech processing tasks using WavLM (Chen (Intro)).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICOLE A K SCHMIEDER whose telephone number is (571)270-1474. The examiner can normally be reached 8:00 - 5:00 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at (571) 272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICOLE A K SCHMIEDER/Primary Examiner, Art Unit 2659
Prosecution Insights