DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-8, 10-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Song et al. ("AdaVITS: Tiny VITS for Low Computing Resource Speaker Adaptation", 01 June 2022).
Claim 1: Song et al. discloses a speech synthesis method, performed by a computer device, comprising:
obtaining a target phoneme of a target user and a timbre identifier of the target user, the target phoneme being determined based on a target text, and the timbre identifier being configured to identify a timbre of the target user; (Section 2: predict PPG from input text; Section 2.2.1: speaker embedding encodes a speaker’s timbre)
inputting the target phoneme into a first submodel of a speech synthesis model, to obtain a target phonetic posteriorgram of the target phoneme, the target phonetic posteriorgram reflecting features of phonemes in the target phoneme and pronunciation duration features of phonemes in the target phoneme; and (Figure 1a: Text2PPG obtains a phoneme from input text, obtains phonetic posteriorgram from the phoneme, duration predictor)
inputting the target phonetic posteriorgram and the timbre identifier into a second submodel of the speech synthesis model, to obtain target speech corresponding to the target text and the timbre identifier, the second submodel being configured to predict the target phonetic posteriorgram and a predicted intermediate variable of the target speech, and the predicted intermediate variable reflecting a frequency domain feature of the target speech. (Figure 1b: PPG2Wav contains PPG predictor; intermediate variable Z)
Claim 2: Elements of parent claim 1 are disclosed as described above. Song et al. further discloses the method wherein the second submodel comprises a prior encoder and a decoder; (Figure 1b: Prior Encoder and Decoder)
and the inputting the target phonetic posteriorgram and the timbre identifier into a second submodel of the speech synthesis model, to obtain target speech corresponding to the target text and the timbre identifier comprises:
inputting the target phonetic posteriorgram and the timbre identifier into the prior encoder, to obtain the predicted intermediate variable; (Figure 1b: Speaker embedding and PPG are input to PPG encoder)
performing inverse Fourier transform on the predicted intermediate variable through the decoder, to obtain the target speech. (Figure 1b: Intermediate variable ‘Z’ is input to ISTFT Decoder to obtain a Raw Waveform)
Claim 3: Elements of parent claim 2 are disclosed as described above. Song et al. further discloses the method wherein the prior encoder comprises a phonetic posterior encoder; (Figure 1b: Posterior encoder)
and the inputting the target phonetic posteriorgram and the timbre identifier into the prior encoder, to obtain the predicted intermediate variable comprises:
inputting the target phonetic posteriorgram and the timbre identifier into the phonetic posterior encoder, and sampling an average value and a variance of prior distribution of the predicted intermediate variable, to obtain the predicted intermediate variable. (Section 2.2.1: Posterior encoder extracts the mean and variance of the posterior distribution)
Claim 4: Elements of parent claim 2 are disclosed as described above. Song et al. further discloses the method further comprising:
obtaining a fundamental frequency feature corresponding to the target phoneme and a style identifier, wherein the fundamental frequency feature is obtained by performing feature extraction on the target phoneme through the first submodel based on the style identifier, and the style identifier is configured to identify a speech style; (Section 3.2: VISinger Model)
the inputting the target phonetic posteriorgram and the timbre identifier into the prior encoder, to obtain the predicted intermediate variable comprises:
inputting the target phonetic posteriorgram, the timbre identifier, and the fundamental frequency feature into the prior encoder, to obtain the predicted intermediate variable corresponding to the style identifier. (Figure 1b: PPG encoder takes speaker embedding and PPG as input; style information/fundamental frequency would be a speaker embedding)
Claim 5: Elements of parent claim 2 are disclosed as described above. Song et al. further discloses the method wherein the decoder comprises an inverse Fourier transform decoder; and the performing inverse Fourier transform on the predicted intermediate variable through the decoder, to obtain the target speech comprises:
performing inverse Fourier transform on the predicted intermediate variable through the inverse Fourier transform decoder, to obtain the target speech. (Figure 1b: ISTFT Decoder)
Claim 6: Elements of parent claim 5 are disclosed as described above. Song et al. further discloses the method wherein the inverse Fourier transform decoder comprises a plurality of one-dimensional convolutional layers, and the last one-dimensional convolutional layer is connected to an inverse Fourier transform layer. (Figure 2: Decoder architecture depicts Conv1D layers and an ISTFT layer)
Claim 7: Elements of parent claim 1 are disclosed as described above. Song et al. further discloses the method wherein the first submodel comprises a text encoder, a duration regulation device, and a post-processing network; (Figure 1a: Text encoder, length regulator, PostNet)
the inputting the target phoneme into a first submodel of the speech synthesis model, to obtain a target phonetic posteriorgram of the target phoneme comprises:
obtaining predicted pronunciation duration features corresponding to phonemes in the target phoneme; (Figure 1a: Duration predictor)
performing encoding on the target phoneme through the text encoder, to obtain a hidden layer feature of the target phoneme; (Section 2.2.1: phoneme text encoding)
performing frame expansion processing on the hidden layer feature of the target phoneme through the duration regulation device according to the predicted pronunciation duration features corresponding to the phonemes in the target phoneme; (Section 2.1, Figure 1a: Length Regulator Module)
performing convolution processing on the hidden layer feature of the target phoneme obtained after frame expansion through the post-processing network, to obtain the target phonetic posteriorgram of the target phoneme. (Section 2.1, Figure 1a: PostNet module)
Claim 8: Elements of parent claim 7 are disclosed as described above. Song et al. further discloses the method further comprising:
obtaining the style identifier; (Section 2, Figure 1: Speaker embedding)
the performing encoding on the target phoneme through the text encoder, to obtain a hidden layer feature of the target phoneme comprises:
performing encoding on the target phoneme through the text encoder according to the style identifier, to obtain the hidden layer feature of the target phoneme corresponding to the style identifier. (Section 2.2.1: encoding speaker information)
Claim 10: Elements of parent claim 1 are disclosed as described above. Song et al. further discloses the method wherein the speech synthesis model is trained by:
obtaining a sample phoneme of the target user and the timbre identifier of the target user, the sample phoneme being determined based on a sample text corresponding to sample speech of the target user; (Section 2: predict PPG from input text; Section 2.2.1: speaker embedding encodes a speaker’s timbre)
inputting the sample phoneme into the first submodel of the speech synthesis model, to obtain a predicted phonetic posteriorgram of the sample phoneme, the predicted phonetic posteriorgram of the sample phoneme reflecting features of phonemes in the sample phoneme and pronunciation duration features of phonemes in the sample phoneme; (Figure 1a: Text2PPG obtains a phoneme from input text, obtains phonetic posteriorgram from the phoneme, duration predictor)
inputting the predicted phonetic posteriorgram and the timbre identifier into the second submodel of the speech synthesis model, to obtain predicted speech corresponding to the sample text and the timbre identifier, the predicted intermediate variable reflecting a frequency domain feature of the predicted speech; (Figure 1b: PPG2Wav contains PPG predictor; intermediate variable Z)
training the first submodel according to the predicted phonetic posteriorgram; and training the second submodel according to the predicted speech and the predicted intermediate variable. (Section 2.1-2.2: training the models according to the predicted variables)
Claim 11: Elements of parent claim 10 are disclosed as described above. Song et al. further discloses the method wherein the second submodel comprises a prior encoder and a decoder; (Figure 1b: Prior Encoder and Decoder)
and the inputting the predicted phonetic posteriorgram and the timbre identifier into the second submodel of the speech synthesis model, to obtain predicted speech corresponding to the sample text and the timbre identifier comprises:
inputting the predicted phonetic posteriorgram and the timbre identifier into the prior encoder, to obtain the predicted intermediate variable; and (Figure 1b: Speaker embedding and PPG are input to PPG encoder)
performing inverse Fourier transform on the predicted intermediate variable through the decoder, to obtain the predicted speech. (Figure 1b: Intermediate variable ‘Z’ is input to ISTFT Decoder to obtain a Raw Waveform)
Claim 12: Elements of parent claim 11 are disclosed as described above. Song et al. further discloses the method wherein the second submodel further comprises a posterior encoder; (Figure 1b: Posterior encoder)
and the training the second submodel according to the predicted intermediate variable comprises:
obtaining the sample speech; (Section 3.1: speaker speech samples for training)
inputting the sample speech into the posterior encoder, to obtain a real intermediate variable; and (Figure 1b: posterior predictor outputs Z from linear spectrogram)
calculating a relative entropy loss between the predicted intermediate variable and the real intermediate variable, to train the second submodel. (Section 2.2.1: loss calculated from input intermediate variable ‘Z’)
Claim 13: Elements of parent claim 12 are disclosed as described above. Song et al. further discloses the method wherein the posterior encoder comprises a posterior predictor; (Figure 1b: Posterior predictor)
and the inputting the sample speech into the posterior encoder, to obtain a real intermediate variable comprises:
inputting the sample speech into the posterior predictor, and sampling an average value and a variance of posterior distribution of the real intermediate variable, to obtain the real intermediate variable. (Section 2.2.1: Posterior encoder extracts the mean and variance of the posterior distribution)
Claim 14: Elements of parent claim 11 are disclosed as described above. Song et al. further discloses the method wherein the prior encoder comprises a phonetic posterior predictor, and the phonetic posterior predictor is configured to predict, in a process of pre-training the second submodel, a predicted phonetic posteriorgram in the pre-training process according to a predicted intermediate variable in the pre-training process; (Figure 1b: Posterior encoder, intermediate variable Z is inputted to a PPG predictor)
and the method further comprises:
in the pre-training process of the second submodel, calculating a loss function between the predicted phonetic posteriorgram in the pre-training process and a real phonetic posteriorgram in the pre-training process, to train the second submodel. (Figure 1b: PPG Loss)
Claim 15: Elements of parent claim 11 are disclosed as described above. Song et al. further discloses the method wherein the decoder further comprises a discriminator, and the discriminator forms a generative adversarial network with parts of the speech synthesis model other than the discriminator; and the method further comprises: (Figure 1b: MSD and MCD discriminators)
inputting the predicted speech into the discriminator, to obtain a discrimination result of the predicted speech, wherein the discrimination result reflects that the predicted speech is real information or predicted information; and (Figure 1b: Raw Waveform input to discriminators, Real/Fake output)
determining a generative adversarial loss according to the discrimination result and a real source of the predicted speech, to train the generative adversarial network. (Sections 2.2.1-2.2.4, Figure 1b: compute loss according to difference between ground truth and prediction)
Claim 16: Elements of parent claim 10 are disclosed as described above. Song et al. further discloses the method wherein the first submodel comprises a text encoder, a duration regulation device, and a post-processing network; (Figure 1a: Text encoder, length regulator, PostNet)
the inputting the sample phoneme into a first submodel of the speech synthesis model, to obtain a predicted phonetic posteriorgram of the sample phoneme comprises:
obtaining a real pronunciation duration feature corresponding to each phoneme in the sample phoneme; (Figure 1a: Duration predictor)
performing encoding on the sample phoneme through the text encoder, to obtain a hidden layer feature of the sample phoneme; (Section 2.2.1: phoneme text encoding)
performing frame expansion processing on the hidden layer feature of the sample phoneme through the duration regulation device according to the real pronunciation duration feature corresponding to each phoneme in the sample phoneme; (Section 2.1, Figure 1a: Length Regulator Module)
performing convolution processing on the hidden layer feature of the sample phoneme obtained after frame expansion through the post-processing network, to obtain the predicted phonetic posteriorgram of the sample phoneme. (Section 2.1, Figure 1a: PostNet module)
Claim 17: Elements of parent claim 16 are disclosed as described above. Song et al. further discloses the method wherein the first submodel comprises a duration predictor, and the method further comprises: (Figure 1a: duration predictor)
predicting the hidden layer feature of the sample phoneme through the duration predictor, to obtain a predicted pronunciation duration feature corresponding to each phoneme in the sample phoneme; (Figure 1a, Section 3.2: duration prediction module)
calculating a loss function between the predicted pronunciation duration feature and the real pronunciation duration feature, to train the first submodel, wherein after the training is completed, the real pronunciation duration feature configured for being input into the duration regulation device is replaced with the predicted pronunciation duration feature obtained by the duration predictor. (Figure 1a, Section 3.2: duration prediction model training)
Claims 18-20 are analogous to claims 1-2 and are rejected in a similar fashion.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Song et al. ("AdaVITS: Tiny VITS for Low Computing Resource Speaker Adaptation", 01 June 2022) in view of Zhang et al. ("ViSinger: Variational Inference With Adversarial Learning For End-To-End Singing Voice Synthesis", 24 February 2022).
Claim 9: Elements of parent claim 8 are disclosed as described above. Zhang et al. further discloses the method wherein the first submodel further comprises a fundamental frequency predictor; (Figure 1: F0 [fundamental frequency] Predictor) and the method further comprises:
predicting the hidden layer feature of the target phoneme through the fundamental frequency predictor, to obtain the fundamental frequency feature corresponding to the style identifier and the target phoneme, wherein the fundamental frequency feature is configured for being input into the second submodel, to obtain the target speech corresponding to the style identifier. (Figure 1: Output of F0 Predictor is input into Frame Prior Network; Section 2.3.3: F0 [fundamental frequency] is obtained via an F0 predictor consisting of multiple FFT blocks during inference)
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention to have used a fundamental frequency predictor in the method of Song because it is a vital acoustic feature in singing synthesis (see Zhang Abstract, Section 2.2.3). Furthermore, the architecture of AdaVITS-e disclosed by Song et al. is similar to the VISinger model of Zhang (see Song Section 3.2).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See PTO-892.
Zhang et al. (US 20220375453 A1) discloses a speech synthesis method using pronunciation information from text and style information from a speaker.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALVIN ISKENDER whose telephone number is (703)756-4565. The examiner can normally be reached M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, HAI PHAN can be reached at (571) 272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALVIN ISKENDER/ Examiner, Art Unit 2654
/HAI PHAN/ Supervisory Patent Examiner, Art Unit 2654
Prosecution Insights