DETAILED ACTION
This action is in response to the application filed on March 7th, 2024. Claims 1-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-4 and 7-9 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Garman et al. (US Pat. Pub. No. 2021/0256961 A1 hereinafter Garman).
Regarding claim 1, Garman discloses a method comprising: obtaining a dataset of a plurality of sample speech clips (Garman, Fig. 3, 302; [0045]: "For synthesis, the inputs may include: the text to be spoken 302, the language of the text 304, identification of the speaker 306, and the output accent 308 (which may be the same as the language of the text)."); generating a plurality of sequence embeddings based on the plurality of sample speech clips (Garman, Fig. 3, 320; [0033]: "With embedding, each input item may be replaced by a vector of values. Accordingly, phonemes 214 may be embedded in feature vector 220, while prosody 218 may be embedded in feature vector 222."); initializing a plurality of speaker embeddings and a plurality of accent embeddings; updating the plurality of speaker embeddings based on the plurality of sample speech clips; updating the plurality of accent embeddings based on the plurality of sample speech clips (Garman, Fig. 3, 324 and 326; [0052]: "The inputs for the phonemes 316, accents 308, and speakers 306 may be fed into an Embedding layer to generated embeddings 320, 324, 326."); generating a plurality of augmented embeddings based on the plurality of sequence embeddings, the plurality of speaker embeddings, and the plurality of accent embeddings (Garman, Fig. 3, 228 and Output; [0040]: "DRNN 228 may take a sequence of embedded phonemes 220, prosodic values 222, language identifiers 224, and speaker identifiers 226 as input, as described above."; [0041]: "The output of DRNN 228 may be a set of acoustic features 234, as described above. DRNN 228 may be language independent. When trained, DRNN 228 may be a single universal speech model and may encode all the information necessary to produce speech for all of the trained languages and speakers."); and generating a plurality of synthetic speech clips based on the plurality of augmented embeddings (Garman, Fig. 3, 314; [0046]: "Many of the components of system 300 are similar to those shown in FIG. 2. In embodiments, differences may include text conversion block 310 instead of ASR & aligner 212, the output of each frame may be used as input to a later frame after some time delay, and the output goes to a decoder 312 to produce speech.").
Regarding claim 2, the rejection of claim 1 is incorporated. Garman discloses all of the elements of the current invention as stated above. Garman further discloses wherein a sample speech clip of the plurality of sample speech clips is labeled with a text transcript and an accent identifier, and wherein updating the plurality of accent embeddings comprises updating an accent embedding of the plurality of accent embeddings based on the text transcript and the accent identifier (Garman, Fig. 3, 228 and Output; [0040]: "DRNN 228 may take a sequence of embedded phonemes 220, prosodic values 222, language identifiers 224, and speaker identifiers 226 as input, as described above."; Garman, [0058]: "For example, as shown in FIG. 2, phonemes may be embedded 220. In embodiments, the embedding may be performed so that similar phonemes may generate similar embedding vectors. Likewise, the differences between the phonemes may also be reflected in the embedding vectors. Embedding bag layers are similar to embedding layers, but may accept zero or more inputs and may generate one or more outputs. Embedding bag functions may compute, for example, sums or means of several embedding vectors. In embodiments, embedding bag functions may be used with prosodic elements 218, while embedding functions may be used with phonemes 214, languages 208, and speaker elements 210.").
Regarding claim 3, the rejection of claim 1 is incorporated. Garman discloses all of the elements of the current invention as stated above. Garman further discloses wherein a sample speech clip of the plurality of sample speech clips is labeled with a text transcript and a speaker identifier, and wherein updating the plurality of speaker embeddings comprises updating a speaker embedding of the plurality of speaker embeddings based on the text transcript and the speaker identifier (Garman, Fig. 3, 228 and Output; [0040]: "DRNN 228 may take a sequence of embedded phonemes 220, prosodic values 222, language identifiers 224, and speaker identifiers 226 as input, as described above."; Garman, [0058]: "For example, as shown in FIG. 2, phonemes may be embedded 220. In embodiments, the embedding may be performed so that similar phonemes may generate similar embedding vectors. Likewise, the differences between the phonemes may also be reflected in the embedding vectors. Embedding bag layers are similar to embedding layers, but may accept zero or more inputs and may generate one or more outputs. Embedding bag functions may compute, for example, sums or means of several embedding vectors. In embodiments, embedding bag functions may be used with prosodic elements 218, while embedding functions may be used with phonemes 214, languages 208, and speaker elements 210.").
Regarding claim 4, the rejection of claim 1 is incorporated. Garman discloses all of the elements of the current invention as stated above. Garman further discloses wherein generating the plurality of augmented embeddings comprises summing the plurality of sequence embeddings with the plurality of speaker embeddings and the plurality of accent embeddings (Garman, Fig. 3, 228 and Output; [0040]: "DRNN 228 may take a sequence of embedded phonemes 220, prosodic values 222, language identifiers 224, and speaker identifiers 226 as input, as described above."; [0041]: "The output of DRNN 228 may be a set of acoustic features 234, as described above. DRNN 228 may be language independent. When trained, DRNN 228 may be a single universal speech model and may encode all the information necessary to produce speech for all of the trained languages and speakers.").
Regarding claim 7, Garman discloses a computing system comprising processing circuitry and memory for executing a machine learning system (Garman, [0010]: “a method for text-to-speech conversion may be implemented in a computer system comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor”), the machine learning system configured to: obtain a dataset of a plurality of sample speech clips (Garman, Fig. 3, 302; [0045]: "For synthesis, the inputs may include: the text to be spoken 302, the language of the text 304, identification of the speaker 306, and the output accent 308 (which may be the same as the language of the text)."); generate a plurality of sequence embeddings based on the plurality of sample speech clips (Garman, Fig. 3, 320; [0033]: "With embedding, each input item may be replaced by a vector of values. Accordingly, phonemes 214 may be embedded in feature vector 220, while prosody 218 may be embedded in feature vector 222."); initialize a plurality of speaker embeddings and a plurality of accent embeddings; update the plurality of speaker embeddings based on the plurality of sample speech clips; update the plurality of accent embeddings based on the plurality of sample speech clips (Garman, Fig. 3, 324 and 326; [0052]: "The inputs for the phonemes 316, accents 308, and speakers 306 may be fed into an Embedding layer to generated embeddings 320, 324, 326."); generate a plurality of augmented embeddings based on the plurality of sequence embeddings, the plurality of speaker embeddings, and the plurality of accent embeddings (Garman, Fig. 3, 228 and Output; [0040]: "DRNN 228 may take a sequence of embedded phonemes 220, prosodic values 222, language identifiers 224, and speaker identifiers 226 as input, as described above."; [0041]: "The output of DRNN 228 may be a set of acoustic features 234, as described above. DRNN 228 may be language independent. When trained, DRNN 228 may be a single universal speech model and may encode all the information necessary to produce speech for all of the trained languages and speakers."); and generate a plurality of synthetic speech clips based on the plurality of augmented embeddings (Garman, Fig. 3, 314; [0046]: "Many of the components of system 300 are similar to those shown in FIG. 2. In embodiments, differences may include text conversion block 310 instead of ASR & aligner 212, the output of each frame may be used as input to a later frame after some time delay, and the output goes to a decoder 312 to produce speech.").
Regarding claim 8, the rejection of claim 7 is incorporated. Garman discloses all of the elements of the current invention as stated above. Garman further discloses wherein a sample speech clip of the plurality of sample speech clips is labeled with a text transcript and an accent identifier, and wherein to update the plurality of accent embeddings, the machine learning system is configured to: update an accent embedding of the plurality of accent embeddings based on the text transcript and the accent identifier (Garman, Fig. 3, 228 and Output; [0040]: "DRNN 228 may take a sequence of embedded phonemes 220, prosodic values 222, language identifiers 224, and speaker identifiers 226 as input, as described above."; Garman, [0058]: "For example, as shown in FIG. 2, phonemes may be embedded 220. In embodiments, the embedding may be performed so that similar phonemes may generate similar embedding vectors. Likewise, the differences between the phonemes may also be reflected in the embedding vectors. Embedding bag layers are similar to embedding layers, but may accept zero or more inputs and may generate one or more outputs. Embedding bag functions may compute, for example, sums or means of several embedding vectors. In embodiments, embedding bag functions may be used with prosodic elements 218, while embedding functions may be used with phonemes 214, languages 208, and speaker elements 210.").
Regarding claim 9, the rejection of claim 7 is incorporated. Garman discloses all of the elements of the current invention as stated above. Garman further discloses wherein to generate the plurality of augmented embeddings, the machine learning system is configured to: sum the plurality of sequence embeddings with the plurality of speaker embeddings and the plurality of accent embeddings (Garman, Fig. 3, 228 and Output; [0040]: "DRNN 228 may take a sequence of embedded phonemes 220, prosodic values 222, language identifiers 224, and speaker identifiers 226 as input, as described above."; [0041]: "The output of DRNN 228 may be a set of acoustic features 234, as described above. DRNN 228 may be language independent. When trained, DRNN 228 may be a single universal speech model and may encode all the information necessary to produce speech for all of the trained languages and speakers.").
Claim(s) 12-13, 15-16, 17-18, and 20 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Shakeri et al. (US Pat. No. 11,790,884 B1 hereinafter Shakeri).
Regarding claim 12, Shakeri discloses a method comprising: obtaining an audio waveform (Shakeri, Fig. 8, 8.1; Col. 17, lines 51-52: "In step 8.1, input data is inputted into a synthesizer module. The input data represents speech content."); decomposing the audio waveform into first one or more magnitude spectral slices and an original phase (Shakeri, Fig. 8, 8.2; Col. 18, lines 8-11: "In step 8.2, source acoustic features are generated as output of the synthesizer module. The source acoustic features are acoustic features for the speech content in the voice of a source speaker."; Col. 3, lines 43-45: "“Acoustic features” as used in some embodiments described herein may include any suitable acoustic representation of frequency, magnitude and/or phase information."); processing, by an autoencoder trained based on accented speech clips with aligned phonemes, the first one or more magnitude spectral slices to map the first one or more magnitude spectral slices associated with a source accent to second one or more magnitude spectral slices associated with a target accent (Shakeri, Fig. 8, 8.3-8.6; Col. 18, lines 12-14: "In step 8.3, a target speaker embedding and the source acoustic features are inputted into an acoustic feature encoder of the voice convertor."; lines 40-41: "In step 8.4, one or more acoustic feature encodings are generated as output of the acoustic feature encoder."; lines 63-66: "In step 8.5, the one or more acoustic feature encodings are inputted into an acoustic feature decoder of the voice convertor module. In step 8.6, target acoustic features are generated."); and generating a modified audio waveform in part by combining the second one or more magnitude spectral slices and the original phase (Shakeri, Fig. 8, 8.7; Col. 19, lines 4-8: "In step 8.7, speech audio in the voice of the player is generated. This step comprises processing the target acoustic features with one or more modules, the one or more modules comprising a vocoder configured to generate speech audio in the voice of the player.").
Regarding claim 13, the rejection of claim 12 is incorporated. Shakeri discloses all of the elements of the current invention as stated above. Shakeri further discloses receiving an indication of the target accent (Shakeri, Fig. 8, 8.3; Col. 18, lines 12-19: "In step 8.3, a target speaker embedding and the source acoustic features are inputted into an acoustic feature encoder of the voice convertor. The speaker embedding is a learned representation of the voice of a player of the video game. The speaker embedding may be generated from output of a speaker encoder of the voice convertor, the generating comprising inputting player identifier data into the speaker encoder."), wherein processing the first one or more magnitude spectral slices to map the first one or more magnitude spectral slices to the second one or more spectral slices comprises converting, by the autoencoder, based on the indication of the target accent, first spectral characteristics of the first one or more magnitude spectral slices associated with an original accent to second spectral characteristics of the second one or more magnitude spectral slices associated with the target accent (Shakeri, Fig. 8, 8.3-8.7; Col. 18, lines 12-14: "In step 8.3, a target speaker embedding and the source acoustic features are inputted into an acoustic feature encoder of the voice convertor."; lines 40-41: "In step 8.4, one or more acoustic feature encodings are generated as output of the acoustic feature encoder."; lines 63-66: "In step 8.5, the one or more acoustic feature encodings are inputted into an acoustic feature decoder of the voice convertor module. In step 8.6, target acoustic features are generated. The target acoustic features comprise acoustic features for the speech content in the voice of the player. The generating comprises decoding the one or more acoustic feature encodings using the acoustic feature decoder. In step 8.7, speech audio in the voice of the player is generated. This step comprises processing the target acoustic features with one or more modules, the one or more modules comprising a vocoder configured to generate speech audio in the voice of the player.").
Regarding claim 15, the rejection of claim 12 is incorporated. Shakeri discloses all of the elements of the current invention as stated above. Shakeri further discloses outputting the modified audio waveform (Shakeri, Fig. 8, 8.7; Col. 19, lines 4-8: "In step 8.7, speech audio in the voice of the player is generated. This step comprises processing the target acoustic features with one or more modules, the one or more modules comprising a vocoder configured to generate speech audio in the voice of the player.").
Regarding claim 16, the rejection of claim 12 is incorporated. Shakeri discloses all of the elements of the current invention as stated above. Shakeri further discloses wherein the first one or more magnitude spectral slices comprises a two-dimensional representation of blocks of spectral magnitudes corresponding to overlapping frames of the audio waveform (Shakeri, Fig. 8, 8.2; Col. 18, lines 8-11: "In step 8.2, source acoustic features are generated as output of the synthesizer module. The source acoustic features are acoustic features for the speech content in the voice of a source speaker."; Col. 3, lines 43-45: "“Acoustic features” as used in some embodiments described herein may include any suitable acoustic representation of frequency, magnitude and/or phase information.").
Regarding claim 17, Shakeri discloses a computing system comprising processing circuitry and memory for executing a machine learning system (Shakeri, Col. 20, lines 48-54: “Generally speaking, the one or more processors 1002 execute one or more instructions of the operating instructions 1008, which are stored permanently or semi-permanently in the non-volatile memory 1006, using the volatile memory 1004 to temporarily store data generated during execution of said operating instructions 1008.”), the machine learning system configured to: obtain an audio waveform (Shakeri, Fig. 8, 8.1; Col. 17, lines 51-52: "In step 8.1, input data is inputted into a synthesizer module. The input data represents speech content."); decompose the audio waveform into first one or more magnitude spectral slices and an original phase (Shakeri, Fig. 8, 8.2; Col. 18, lines 8-11: "In step 8.2, source acoustic features are generated as output of the synthesizer module. The source acoustic features are acoustic features for the speech content in the voice of a source speaker."; Col. 3, lines 43-45: "“Acoustic features” as used in some embodiments described herein may include any suitable acoustic representation of frequency, magnitude and/or phase information."); process, by an autoencoder trained based on accented speech clips with aligned phonemes, the first one or more magnitude spectral slices to map the first one or more magnitude spectral slices associated with a source accent to second one or more magnitude spectral slices associated with a target accent (Shakeri, Fig. 8, 8.3-8.6; Col. 18, lines 12-14: "In step 8.3, a target speaker embedding and the source acoustic features are inputted into an acoustic feature encoder of the voice convertor."; lines 40-41: "In step 8.4, one or more acoustic feature encodings are generated as output of the acoustic feature encoder."; lines 63-66: "In step 8.5, the one or more acoustic feature encodings are inputted into an acoustic feature decoder of the voice convertor module. In step 8.6, target acoustic features are generated."); and generate a modified audio waveform in part by combining the second one or more magnitude spectral slices and the original phase (Shakeri, Fig. 8, 8.7; Col. 19, lines 4-8: "In step 8.7, speech audio in the voice of the player is generated. This step comprises processing the target acoustic features with one or more modules, the one or more modules comprising a vocoder configured to generate speech audio in the voice of the player.").
Regarding claim 18, the rejection of claim 17 is incorporated. Shakeri discloses all of the elements of the current invention as stated above. Shakeri further discloses wherein to process the first one or more magnitude spectral slices to the second one or more spectral slices, the machine learning system is configured to: convert first spectral characteristics of the first one or more magnitude spectral slices associated with an original accent to second spectral characteristics of the second one or more magnitude spectral slices associated with a target accent (Shakeri, Fig. 8, 8.3-8.7; Col. 18, lines 12-14: "In step 8.3, a target speaker embedding and the source acoustic features are inputted into an acoustic feature encoder of the voice convertor."; lines 40-41: "In step 8.4, one or more acoustic feature encodings are generated as output of the acoustic feature encoder."; lines 63-66: "In step 8.5, the one or more acoustic feature encodings are inputted into an acoustic feature decoder of the voice convertor module. In step 8.6, target acoustic features are generated. The target acoustic features comprise acoustic features for the speech content in the voice of the player. The generating comprises decoding the one or more acoustic feature encodings using the acoustic feature decoder. In step 8.7, speech audio in the voice of the player is generated. This step comprises processing the target acoustic features with one or more modules, the one or more modules comprising a vocoder configured to generate speech audio in the voice of the player.").
Regarding claim 20, the rejection of claim 17 is incorporated. Shakeri discloses all of the elements of the current invention as stated above. Shakeri further discloses wherein the machine learning system is further configured to: output the modified audio waveform (Shakeri, Fig. 8, 8.7; Col. 19, lines 4-8: "In step 8.7, speech audio in the voice of the player is generated. This step comprises processing the target acoustic features with one or more modules, the one or more modules comprising a vocoder configured to generate speech audio in the voice of the player.").
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 5 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garman as applied to claims 1-4 and 7-9 above, and further in view of Finkelstein et al. (US Pat. Pub. No. 2023/0018384 A1 hereinafter Finkelstein).
Regarding claim 5, the rejection of claim 1 is incorporated. Garman discloses all of the elements of the current invention as stated above. However, Garman fails to expressly recite wherein the plurality of synthetic speech clips comprises a first synthetic speech clip associated with a speaker and a first accent and a second synthetic speech clip associated with the speaker and a second accent, and wherein the method further comprises: aligning a first set of frames associated with the first synthetic speech clip with a second set of frames associated with the second synthetic speech clip; and generating an instance of training data based on the alignment of the first set of frames associated with the first synthetic speech clip and the second set of frames associated with the second synthetic speech clip.
Finkelstein teaches wherein the plurality of synthetic speech clips comprises a first synthetic speech clip associated with a speaker and a first accent and a second synthetic speech clip associated with the speaker and a second accent (Finkelstein, [0025]: "Accordingly, a given text input can produce synthesized speech in a given language across various different accents/dialects and/or different speaking styles, as well as produce synthesized speech across different languages."), and wherein the method further comprises: aligning a first set of frames associated with the first synthetic speech clip with a second set of frames associated with the second synthetic speech clip (Finkelstein, [0031]: "examples herein are directed toward the trained voice cloning system 200 generating training synthesized speech representations 202 that clone the voice of a target speaker in a target accent/dialect (e.g., second accent/dialect)."; [0005]: "The operations may further include sampling, from the training synthesized speech representation, a sequence of fixed-length reference frames providing reference prosodic features that represent the prosody captured by the training synthesized speech representation."); and generating an instance of training data based on the alignment of the first set of frames associated with the first synthetic speech clip and the second set of frames associated with the second synthetic speech clip (Finkelstein, [0031]: "examples herein are directed toward the trained voice cloning system 200 generating training synthesized speech representations 202 that clone the voice of a target speaker in a target accent/dialect (e.g., second accent/dialect).").
Garman and Finkelstein are analogous arts because they each belong to the same field of speech processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the parametric speech synthesis system of Garman, to incorporate the teachings of Finkelstein to generate a training data set based on synthesized speech. Generating a synthetic training data set allows the system to create training data that would be challenging to collect naturally (Finkelstein, [0025]). This ensures a large and varied training set is available for training speech processing systems.
Regarding claim 10, the rejection of claim 7 is incorporated. Garman discloses all of the elements of the current invention as stated above. However, Garman fails to expressly recite wherein the plurality of synthetic speech clip comprises a first synthetic speech clip associated with a speaker and a first accent and a second synthetic speech clip associated with the speaker and a second accent, and wherein the machine learning system is further configured to: align a first set of frames associated with the first synthetic speech clip with a second set of frames associated with the second synthetic speech clip; and generate an instance of training data based on the alignment of the first set of frames associated with the first synthetic speech clip and the second set of frames associated with the second synthetic speech clip.
Finkelstein teaches wherein the plurality of synthetic speech clip comprises a first synthetic speech clip associated with a speaker and a first accent and a second synthetic speech clip associated with the speaker and a second accent (Finkelstein, [0025]: "Accordingly, a given text input can produce synthesized speech in a given language across various different accents/dialects and/or different speaking styles, as well as produce synthesized speech across different languages."), and wherein the machine learning system is further configured to: align a first set of frames associated with the first synthetic speech clip with a second set of frames associated with the second synthetic speech clip (Finkelstein, [0031]: "examples herein are directed toward the trained voice cloning system 200 generating training synthesized speech representations 202 that clone the voice of a target speaker in a target accent/dialect (e.g., second accent/dialect)."; [0005]: "The operations may further include sampling, from the training synthesized speech representation, a sequence of fixed-length reference frames providing reference prosodic features that represent the prosody captured by the training synthesized speech representation."); and generate an instance of training data based on the alignment of the first set of frames associated with the first synthetic speech clip and the second set of frames associated with the second synthetic speech clip (Finkelstein, [0031]: "examples herein are directed toward the trained voice cloning system 200 generating training synthesized speech representations 202 that clone the voice of a target speaker in a target accent/dialect (e.g., second accent/dialect).").
Garman and Finkelstein are analogous arts because they each belong to the same field of speech processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the parametric speech synthesis system of Garman, to incorporate the teachings of Finkelstein to generate a training data set based on synthesized speech. Generating a synthetic training data set allows the system to create training data that would be challenging to collect naturally (Finkelstein, [0025]). This ensures a large and varied training set is available for training speech processing systems.
Claim(s) 6 and 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garman as applied to claims 1-4 and 7-9 above, and further in view of Shakeri.
Regarding claim 6, the rejection of claim 1 is incorporated. Garman discloses all of the elements of the current invention as stated above. However, Garman fails to expressly recite providing the plurality of synthetic speech clips to an autoencoder model; and training the autoencoder model to modify speech of an audio waveform based on the plurality of synthetic speech clips.
Shakeri teaches providing the plurality of synthetic speech clips to an autoencoder model; and training the autoencoder model to modify speech of an audio waveform based on the plurality of synthetic speech clips (Shakeri, Col. 15, lines 26-28: "As shown in FIG. 6, acoustic feature encoder 605 and acoustic feature decoder 606 are trained on a voice conversion task using one or more training examples 601.").
Garman and Shakeri are analogous arts because they each belong to the same field of speech processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the parametric speech synthesis system of Garman, to incorporate the teachings of Shakeri to use the training data set to train a voice conversion autoencoder model. Training the autoencoder allows the system to properly convert a voice into another target voice (Shakeri, Col. 15, lines 28-34). This ensures the system can successfully convert voices for the user.
Regarding claim 11, the rejection of claim 7 is incorporated. Garman discloses all of the elements of the current invention as stated above. However, Garman fails to expressly recite wherein the machine learning system is further configured to: provide the plurality of synthetic speech clips to an autoencoder model; and train the autoencoder model to modify speech of an audio waveform based on the plurality of synthetic speech clips.
Shakeri teaches wherein the machine learning system is further configured to: provide the plurality of synthetic speech clips to an autoencoder model; and train the autoencoder model to modify speech of an audio waveform based on the plurality of synthetic speech clips (Shakeri, Col. 15, lines 26-28: "As shown in FIG. 6, acoustic feature encoder 605 and acoustic feature decoder 606 are trained on a voice conversion task using one or more training examples 601.").
Garman and Shakeri are analogous arts because they each belong to the same field of speech processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the parametric speech synthesis system of Garman, to incorporate the teachings of Shakeri to use the training data set to train a voice conversion autoencoder model. Training the autoencoder allows the system to properly convert a voice into another target voice (Shakeri, Col. 15, lines 28-34). This ensures the system can successfully convert voices for the user.
Claim(s) 14 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shakeri as applied to claims 12-13 and 15-16 above, and further in view of Garman and Finkelstein.
Regarding claim 14, the rejection of claim 12 is incorporated. Shakeri discloses all of the elements of the current invention as stated above. Shakeri further discloses training the autoencoder based on the instance of training data (Shakeri, Col. 15, lines 26-28: "As shown in FIG. 6, acoustic feature encoder 605 and acoustic feature decoder 606 are trained on a voice conversion task using one or more training examples 601."). However, Shakeri fails to expressly recite wherein training the autoencoder based on accented speech clips with aligned phonemes comprises: obtaining a dataset of a plurality of sample speech clips; generating a plurality of sequence embeddings based on the plurality of sample speech clips; initializing a plurality of speaker embeddings and a plurality of accent embeddings; updating the plurality of speaker embeddings based on the plurality of sample speech clips; updating the plurality of accent embeddings based on the plurality of sample speech clips; generating a plurality of augmented embeddings based on the plurality of sequence embeddings, the plurality of speaker embeddings, and the plurality of accent embeddings; generating a plurality of synthetic speech clips based on the plurality of augmented embeddings, wherein the plurality of synthetic speech clips comprises a first synthetic speech clip associated with a speaker and a first accent and a second synthetic speech clip associated with the speaker and a second accent; aligning a first set of frames associated with the first synthetic speech clip with a second set of frames associated with the second synthetic speech clip; generating an instance of training data based on the alignment of the first set of frames associated with the first synthetic speech clip and the second set of frames associated with the second synthetic speech clip.
Garman teaches wherein training the autoencoder based on accented speech clips with aligned phonemes comprises: obtaining a dataset of a plurality of sample speech clips (Garman, Fig. 3, 302; [0045]: "For synthesis, the inputs may include: the text to be spoken 302, the language of the text 304, identification of the speaker 306, and the output accent 308 (which may be the same as the language of the text)."); generating a plurality of sequence embeddings based on the plurality of sample speech clips (Garman, Fig. 3, 320; [0033]: "With embedding, each input item may be replaced by a vector of values. Accordingly, phonemes 214 may be embedded in feature vector 220, while prosody 218 may be embedded in feature vector 222."); initializing a plurality of speaker embeddings and a plurality of accent embeddings; updating the plurality of speaker embeddings based on the plurality of sample speech clips; updating the plurality of accent embeddings based on the plurality of sample speech clips (Garman, Fig. 3, 324 and 326; [0052]: "The inputs for the phonemes 316, accents 308, and speakers 306 may be fed into an Embedding layer to generated embeddings 320, 324, 326."); generating a plurality of augmented embeddings based on the plurality of sequence embeddings, the plurality of speaker embeddings, and the plurality of accent embeddings (Garman, Fig. 3, 228 and Output; [0040]: "DRNN 228 may take a sequence of embedded phonemes 220, prosodic values 222, language identifiers 224, and speaker identifiers 226 as input, as described above."; [0041]: "The output of DRNN 228 may be a set of acoustic features 234, as described above. DRNN 228 may be language independent. When trained, DRNN 228 may be a single universal speech model and may encode all the information necessary to produce speech for all of the trained languages and speakers."); generating a plurality of synthetic speech clips based on the plurality of augmented embeddings, wherein the plurality of synthetic speech clips comprises a first synthetic speech clip associated with a speaker and a first accent and a second synthetic speech clip associated with the speaker and a second accent (Garman, Fig. 3, 314; [0046]: "Many of the components of system 300 are similar to those shown in FIG. 2. In embodiments, differences may include text conversion block 310 instead of ASR & aligner 212, the output of each frame may be used as input to a later frame after some time delay, and the output goes to a decoder 312 to produce speech.").
Shakeri and Garman are analogous arts because they each belong to the same field of speech processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the Speech generation system of Shakeri, to incorporate the teachings of Garman to generate synthetic speech with differing accents. This allows the system to generate speech in the speaker’s voice with any given accent (Garman, [0022]), thus providing the system with varied speech samples for later use. However, Shakeri, in view of Garman, fails to expressly recite aligning a first set of frames associated with the first synthetic speech clip with a second set of frames associated with the second synthetic speech clip; generating an instance of training data based on the alignment of the first set of frames associated with the first synthetic speech clip and the second set of frames associated with the second synthetic speech clip.
Finkelstein teaches aligning a first set of frames associated with the first synthetic speech clip with a second set of frames associated with the second synthetic speech clip (Finkelstein, [0031]: "examples herein are directed toward the trained voice cloning system 200 generating training synthesized speech representations 202 that clone the voice of a target speaker in a target accent/dialect (e.g., second accent/dialect)."; [0005]: "The operations may further include sampling, from the training synthesized speech representation, a sequence of fixed-length reference frames providing reference prosodic features that represent the prosody captured by the training synthesized speech representation."); generating an instance of training data based on the alignment of the first set of frames associated with the first synthetic speech clip and the second set of frames associated with the second synthetic speech clip (Finkelstein, [0031]: "examples herein are directed toward the trained voice cloning system 200 generating training synthesized speech representations 202 that clone the voice of a target speaker in a target accent/dialect (e.g., second accent/dialect).").
Shakeri, Garman, and Finkelstein are analogous arts because they each belong to the same field of speech processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the Speech generation system of Shakeri, as modified by the parametric speech synthesis system of Garman, to incorporate the teachings of Finkelstein to generate a training data set based on synthesized speech. Generating a synthetic training data set allows the system to create training data that would be challenging to collect naturally (Finkelstein, [0025]). This ensures a large and varied training set is available for training speech processing systems.
Regarding claim 19, the rejection of claim 17 is incorporated. Shakeri discloses all of the elements of the current invention as stated above. Shakeri further discloses train the autoencoder based on the instance of training data (Shakeri, Col. 15, lines 26-28: "As shown in FIG. 6, acoustic feature encoder 605 and acoustic feature decoder 606 are trained on a voice conversion task using one or more training examples 601."). However, Shakeri fails to expressly recite wherein to train the autoencoder based on accented speech clips with aligned phonemes, the machine learning model is configured to: obtain a dataset of a plurality of sample speech clips; generate a plurality of sequence embeddings based on the plurality of sample speech clips; initialize a plurality of speaker embeddings and a plurality of accent embeddings; update the plurality of speaker embeddings based on the plurality of sample speech clips; update the plurality of accent embeddings based on the plurality of sample speech clips; generate a plurality of augmented embeddings based on the plurality of sequence embeddings, the plurality of speaker embeddings, and the plurality of accent embeddings; generate a plurality of synthetic speech clips based on the plurality of augmented embeddings, wherein the plurality of synthetic speech clips comprises a first synthetic speech clip associated with a speaker and a first accent and a second synthetic speech clip associated with the speaker and a second accent; align a first set of frames associated with the first synthetic speech clip with a second set of frames associated with the second synthetic speech clip; generate an instance of training data based on the alignment of the first set of frames associated with the first synthetic speech clip and the second set of frames associated with the second synthetic speech clip.
Garman teaches wherein to train the autoencoder based on accented speech clips with aligned phonemes, the machine learning model is configured to: obtain a dataset of a plurality of sample speech clips (Garman, Fig. 3, 302; [0045]: "For synthesis, the inputs may include: the text to be spoken 302, the language of the text 304, identification of the speaker 306, and the output accent 308 (which may be the same as the language of the text)."); generate a plurality of sequence embeddings based on the plurality of sample speech clips (Garman, Fig. 3, 320; [0033]: "With embedding, each input item may be replaced by a vector of values. Accordingly, phonemes 214 may be embedded in feature vector 220, while prosody 218 may be embedded in feature vector 222."); initialize a plurality of speaker embeddings and a plurality of accent embeddings; update the plurality of speaker embeddings based on the plurality of sample speech clips; update the plurality of accent embeddings based on the plurality of sample speech clips (Garman, Fig. 3, 324 and 326; [0052]: "The inputs for the phonemes 316, accents 308, and speakers 306 may be fed into an Embedding layer to generated embeddings 320, 324, 326."); generate a plurality of augmented embeddings based on the plurality of sequence embeddings, the plurality of speaker embeddings, and the plurality of accent embeddings (Garman, Fig. 3, 228 and Output; [0040]: "DRNN 228 may take a sequence of embedded phonemes 220, prosodic values 222, language identifiers 224, and speaker identifiers 226 as input, as described above."; [0041]: "The output of DRNN 228 may be a set of acoustic features 234, as described above. DRNN 228 may be language independent. When trained, DRNN 228 may be a single universal speech model and may encode all the information necessary to produce speech for all of the trained languages and speakers."); generate a plurality of synthetic speech clips based on the plurality of augmented embeddings, wherein the plurality of synthetic speech clips comprises a first synthetic speech clip associated with a speaker and a first accent and a second synthetic speech clip associated with the speaker and a second accent (Garman, Fig. 3, 314; [0046]: "Many of the components of system 300 are similar to those shown in FIG. 2. In embodiments, differences may include text conversion block 310 instead of ASR & aligner 212, the output of each frame may be used as input to a later frame after some time delay, and the output goes to a decoder 312 to produce speech.").
Shakeri and Garman are analogous arts because they each belong to the same field of speech processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the Speech generation system of Shakeri, to incorporate the teachings of Garman to generate synthetic speech with differing accents. This allows the system to generate speech in the speaker’s voice with any given accent (Garman, [0022]), thus providing the system with varied speech samples for later use. However, Shakeri, in view of Garman, fails to expressly recite align a first set of frames associated with the first synthetic speech clip with a second set of frames associated with the second synthetic speech clip; generate an instance of training data based on the alignment of the first set of frames associated with the first synthetic speech clip and the second set of frames associated with the second synthetic speech clip.
Finkelstein teaches align a first set of frames associated with the first synthetic speech clip with a second set of frames associated with the second synthetic speech clip (Finkelstein, [0031]: "examples herein are directed toward the trained voice cloning system 200 generating training synthesized speech representations 202 that clone the voice of a target speaker in a target accent/dialect (e.g., second accent/dialect)."; [0005]: "The operations may further include sampling, from the training synthesized speech representation, a sequence of fixed-length reference frames providing reference prosodic features that represent the prosody captured by the training synthesized speech representation."); generate an instance of training data based on the alignment of the first set of frames associated with the first synthetic speech clip and the second set of frames associated with the second synthetic speech clip (Finkelstein, [0031]: "examples herein are directed toward the trained voice cloning system 200 generating training synthesized speech representations 202 that clone the voice of a target speaker in a target accent/dialect (e.g., second accent/dialect).").
Shakeri, Garman, and Finkelstein are analogous arts because they each belong to the same field of speech processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the Speech generation system of Shakeri, as modified by the parametric speech synthesis system of Garman, to incorporate the teachings of Finkelstein to generate a training data set based on synthesized speech. Generating a synthetic training data set allows the system to create training data that would be challenging to collect naturally (Finkelstein, [0025]). This ensures a large and varied training set is available for training speech processing systems.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Lubin et al. (US Pat. Pub. No. 2024/0355346 A1) discloses a system for voice modification.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TYLER J BECKER whose telephone number is (703)756-1271. The examiner can normally be reached M-Th, 7:15am-5:45pm PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TYLER BECKER/ Examiner, Art Unit 2657
/DANIEL C WASHBURN/ Supervisory Patent Examiner, Art Unit 2657