DETAILED ACTION
This communication is in response to the Application filed on 10/28/2024. Claims 1-4, 6-8, 10-11, 13-15, 17-19, 21-24, and 26 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 13, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/30/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority
Applicant claims the benefit of CIP Application No. 17/731,474, filed April 28, 2022. Claims 1-4, 6-8, 10-11, 13-15, 17-19, 21-24, and 26 have been afforded the benefit of this filing date.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(P) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(P). A request for reconsideration while not provided for in 37 CFR 1.113(P) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-4, 6-8, 10-11, 13-15, 17-19, 21-24, and 26 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-4, 6-7, 9-11, 13-15, 17-21, and 23 of US Patent Publication US 11848005 B2 in view of US Patent Publication US 11367456 B2 (Xie et al.).
Claims not mentioned below map as the following where the instant application is termed as (I)
and the US Patent Publication as (P). Claim 2 (I): Claim 2 (P), Claim 3 (I): Claim 3 (P), Claim 4
(I): Claim 1 (P), Claim 6 (I): Claim 1 (P), Claim 7 (I): Claim 1 (P), Claim 8 (I): Claim 4 (P), Claim 10 (I): Claim 6 (P), Claim 11 (I): Claim 7 (P), Claim 13 (I): Claim 9 (P), Claim 15 (I): Claim 11 (P), Claim 17 (I): Claim 13 (P), Claim 18 (I): Claim 14 (P), Claim 19 (I): Claim 15 (P), Claim 21 (I): Claim 17 (P), Claim 22 (I): Claim 18 (P), Claim 23 (I): Claim 19 (P), Claim 24 (I): Claim 20 (P).
Instant Application: 18860751
CIssued Patent: US 11848005
Claim 1: A system for training a speech-to-speech (S2S) machine learning (ML) model for adapting at least one voice attribute of speech, comprising: at least one processor executing a code for: creating an S2S training dataset of a plurality of S2S records, wherein an S2S record comprises: a first audio content comprising speech having at least one first voice attribute, and a ground truth label of a second audio content comprising speech having at least one second voice attribute, wherein the first audio content and the second audio content have the same lexical content
comprising a same sequence of words in a same language,
and are time-synchronized according to segment-level start and end timestamps
of lexical content of the first audio content synchronized with segment-level start and end timestamps of the same lexical content of the second audio content,
wherein the segment-level start and end timestamps are selected from a group consisting of: word-level start and end timestamps, phrase-level start and end timestamps, sentence-level start and end timestamps, and syllable-level start and end timestamps,
wherein duration of phones of the second audio content are controlled in response to segment-level durations defined by the segment-level start and end time stamps
which are synchronized with the segment-level start and end time stamps of the first audio content;
and training the S2S ML model using the S2S training dataset, wherein the S2S ML model is fed an input of a source audio content with at least one source voice attribute and generates an outcome of the source audio content with at least one target voice attribute.
Claim 1: A computer-implemented method of training a speech-to-speech (S2S) machine learning (ML) model for adapting at least one voice attribute of speech, comprising: creating an S2S training dataset of a plurality of S2S records, wherein an S2S record comprises a first audio content comprising speech having at least one first voice attribute, and a ground truth label of a second audio content comprising speech having at least one second voice attribute, wherein the first audio content and the second audio content have the same lexical content
(See Xie et al. Below)
and are time-synchronized according to segment-level start and end timestamps,
(See Xie et al. Below)
(See claim 21)
wherein duration of phones of the second audio content are controlled in response to segment-level durations defined by the segment-level start and end time stamps;
(See Xie et al. Below)
creating a text-to-speech (TTS) training dataset of a plurality of TTS records, wherein a TTS record comprises a text transcription of a third audio content comprising speech having the at least one-second voice attribute, and a ground truth indication of the third audio content comprising speech having the at least one second voice attribute; training a TTS ML model on the TTS training dataset, wherein the S2S record of the S2S training dataset is created by:feeding the first audio content comprising speech having the at least one first voice attribute into the TTS ML model, wherein the first audio content is of the S2S record, wherein feeding the first audio content into the TTS model further comprises: segmenting the first audio content into segments comprising a plurality of single utterances, wherein the S2S record is for a single utterance; feeding the first audio content into a speech recognition alignment component that in response to an input of the first audio content and text transcript of the first audio content, generates an outcome of the segment-level start and end time stamps, and feeding the segments with corresponding ones of the segment-level start and end time stamps into the TTS ML model for obtaining the second audio content as the outcome of the TTS ML model, wherein the text transcript is the same for the first audio content and the second audio content; and obtaining the second audio content comprising speech having the at least one second voice attribute as the outcome of the TTS ML model, wherein the second audio content is of the S2S record; wherein the TTS ML model comprises: a voice attribute encoder that at least one of: dynamically generates the second audio content having the at least one second voice attribute in response to an input of the third audio content comprising speech having the at least one second voice attribute, statically generates the second audio content having the at least one second voice attribute in response to a selection of the at least one second voice attribute from a plurality of candidate voice attributes, and statistically generates the second audio content having the at least one second voice attribute, an encoder-decoder that generates a spectrogram in response to an input of the segments and corresponding ones of the segment-level start and end time stamps, a duration based modeling predictor and controller that controls durations of phones of the second audio content in response to an input of the segment durations defined by the segment-level start and end time stamps, and a vocoder that generates time-domain speech from the spectrogram;
and training the S2S ML model using the S2S training dataset, wherein the S2S ML model is fed an input of a source audio content with at least one source voice attribute and generates an outcome of the source audio content with at least one target voice attribute.
Claim 21: The computer-implemented method of claim 1, wherein the segment-level start and end timestamps are selected from a group consisting of: word-level start and end timestamps, phrase-level start and end timestamps, sentence-level start and end timestamps, and syllable-level start and end timestamps.
Claim 14: A system of near real-time adaptation of at least one voice attribute, comprising: at least one processor executing a code for: feeding a source audio content comprising speech having at least one source voice attribute into a S2S ML model trained
on a training dataset of a plurality of S2S records, wherein an S2S record comprises: a first audio content comprising speech having at least one first voice attribute, and a ground truth label of a second audio content comprising speech having at least one second voice attribute, wherein the first audio content and the second audio content have the same lexical content
comprising a same sequence of words in a same language,
and are time-synchronized according to segment-level start and end timestamps
of lexical content of the first audio content synchronized with segment-level start and end timestamps of the same lexical content of the second audio content,
wherein the segment-level start and end timestamps are selected from a group consisting of: word-level start and end timestamps, phrase-level start and end timestamps, sentence-level start and end timestamps, and syllable-level start and end timestamps,
wherein duration of phones of the second audio content are controlled in response to segment-level durations defined by the segment-level start and end time stamps
which are synchronized with the segment-level start and end time stamps of the first audio content;
and obtaining a target audio content as an outcome of the S2S ML model for playing on a speaker, wherein the target audio content comprises at least one target voice attribute of the source audio content, and the target audio content and the source audio content have the same lexical content and are time- synchronized according to segment-level start and end timestamps.
Claim 10: A computer implemented method of near real- time adaptation of at least one voice attribute, comprising: feeding the source audio content comprising speech having at least one source voice attribute into the S2S ML model trained as in claim 1;
Claim 1: creating an S2S training dataset of a plurality of S2S records, wherein an S2S record comprises a first audio content comprising speech having at least one first voice attribute, and a ground truth label of a second audio content comprising speech having at least one second voice attribute, wherein the first audio content and the second audio content have the same lexical content
(See Xie et al. Below)
Claim 1: and are time-synchronized according to segment-level start and end timestamps,
(See Xie et al. Below)
Claim 21: The computer-implemented method of claim 1, wherein the segment-level start and end timestamps are selected from a group consisting of: word-level start and end timestamps, phrase-level start and end timestamps, sentence-level start and end timestamps, and syllable-level start and end timestamps.
Claim 1: wherein duration of phones of the second audio content are controlled in response to segment-level durations defined by the segment-level start and end time stamps;
(See Xie et al. Below)
Claim 10: and obtaining a target audio content as the outcome of the S2S ML model for playing on a speaker, wherein the target audio content comprises at least one target voice attribute of the source audio content, and the target audio content and the source audio content have the same lexical content and are time-synchronized according to the segment-level start and end timestamps.
Claim 26 has an identical mapping to Claim 1 with the exception of being a method claim rather than a system claim
Claims 1, 21, and Xie et al.
Regarding claims 1, 14, and 26, the Patent Publication does not explicitly teach comprising a same sequence of words in a same language, and are time-synchronized according to segment-level start and end timestamps of lexical content of the first audio content synchronized with segment-level start and end timestamps of the same lexical content of the second audio content, which are synchronized with the segment-level start and end time stamps of the first audio content;
In Xie et al. a voice conversion method is described in which the first and second audio content have the same sequence of words in the same language (Col. 3, Lines 1-19). Segments of the audio are synchronized using segment-level timestamps (Col. 3, Lines 20-40). Also, the output timestamps are synchronized with the one from the input (Col. 4, Lines 60-64).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed inventions to have modified the voice conversion model as taught by the US Patent Publication to use timestamp synchronization for words in the audio to allow real-time conversion of the source voice data at the same rate/cadence of the source (Xie et al. Col. 1, Lines 33-45).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 13, 14, 24 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Publication US 8751239 B2 (Tian et al.) in view of US Patent Publication US 11367456 B2 (Xie et al.) and US Patent Publication US 5617507 A (Lee et al.).
Regarding Claims 1 and 26, Tian et al. teaches A system for training a speech-to-speech (S2S) machine learning (ML) model for adapting at least one voice attribute of speech, comprising: at least one processor executing a code for:
(The conversion device 66 may be capable of transforming source speech 80 into target speech 82. In this regard, the conversion device 66 may be employed to build transformation models which may essentially include one or more trained Gaussian mixture models (GMMs) for transforming the source speech 80 into the target speech 82. In order to produce the transformation models, GMMs may be trained using training source speech data 84 and training target speech data 86 to determine corresponding conversion functions with respect to corresponding intermediate synthetic speech as described below.) (Col. 9, Lines 39-49).
(In one embodiment, the model trainer 78 and/or the TTS device 76 may be controlled by or otherwise embodied as the processing element 74 (e.g., the controller 20 of the mobile terminal 10 or a processor of a server, computer or other device).) (Col. 9, Lines 59-63)
Claim 26 lists the alternative limitation A computer-implemented method of training a speech-to-speech (S2S) machine learning (ML) model for adapting at least one voice attribute of speech, comprising:
(The conversion device 66 may be capable of transforming source speech 80 into target speech 82. In this regard, the conversion device 66 may be employed to build transformation models which may essentially include one or more trained Gaussian mixture models (GMMs) for transforming the source speech 80 into the target speech 82.) (Col. 9, Lines 39-49).
(In one embodiment, the model trainer 78 and/or the TTS device 76 may be controlled by or otherwise embodied as the processing element 74 (e.g., the controller 20 of the mobile terminal 10 or a processor of a server, computer or other device).) (Col. 9, Lines 59-63)
creating an S2S training dataset of a plurality of S2S records,
(In an exemplary embodiment, the training data may be representative of, for example, audio corresponding to a given number of utterances spoken by a source voice and a potentially different given number of potentially different utterances spoken by a target voice which may be stored, for example, in a database, or gathered from free speech recorded from the source or target, respectively.) (Col. 14, Lines 1-8).
Tian et al. creates an S2S training set beginning with utterances gathered from source voices.
wherein an S2S record comprises: a first audio content comprising speech having at least one first voice attribute,
(Training of a GMM model (e.g., the second conversion model 72) for conversion between the synthetic speaker Z and target speaker Y is described below. Similarly, the GMM model for synthetic to target conversion can also be trained as N.sub.ZY(v, .mu..sub.l, .SIGMA..sub.l). The conversion function that converts synthetic feature z.sub.t to target feature y.sub.t is given by Equation (3).) (Col. 13, Lines 13-19).
Voice attributes referred to as features are identified in the conversion of source voice to target voice where a synthetic voice is made matching a feature of the target voice.
and a ground truth label of a second audio content comprising speech having at least one second voice attribute,
(In this regard, as shown in FIG. 3, the model trainer 78 may be configured to receive training source speech data 84 from a source speaker and training target speech data 86 from a target speaker. The training source speech data 84 and the training target speech data 86 may each be, for example, samples of spoken syllables, words, phrases, or sentences from the source speaker and target speaker, respectively.) (Col. 11, Lines 1-7).
(Training of a GMM model (e.g., the second conversion model 72) for conversion between the synthetic speaker Z and target speaker Y is described below. Similarly, the GMM model for synthetic to target conversion can also be trained as N.sub.ZY(v, .mu..sub.l, .SIGMA..sub.l). The conversion function that converts synthetic feature z.sub.t to target feature y.sub.t is given by Equation (3).) (Col. 13, Lines 13-19).
Training set including ground truth target features are used to train the model.
wherein the first audio content and the second audio content have the same lexical content comprising a same sequence of words in a same language, (XIE ET AL. ALSO TEACHES)
(Once in receipt of the training source text 88 and the training target text 90, the TTS device 76 may be configured to produce parallel training source synthetic speech 92 corresponding to the training source text 88 and parallel training target synthetic speech 94 corresponding to the training target text 90. The training source synthetic speech 92 may then be provided to the first conversion model 70 and the training target synthetic speech 94 may be provided to the second conversion model 72 (e.g., under the control of the model trainer 78 or the processing element 74).) (Col. 11, Lines 50-59).
Parallel speech shows that the lexical content remains the same from the source speech to target speech. This implies that the words are in the same sequence/language but because it is not specifically stated Xie et al. address it further below.
and training the S2S ML model using the S2S training dataset,
(Likewise, having received the parallel training target speech data 86 and the training target synthetic speech 94, the second conversion model may determine a conversion function for transforming synthetic speech into target speech as a result of the training of the second conversion model. As a result, once both the first and second conversion models 70 and 72 have been trained, which may be done offline if desired; the first and second conversion models 70 and 72 may be concatenated together as indicated in FIG. 3 in order to provide a mechanism for text-independent source to target voice conversion.) (Col. 11, Line 64 to Col. 12, Line 8).
The training data is used to train an S2S model
wherein the S2S ML model is fed an input of a source audio content with at least one source voice attribute and generates an outcome of the source audio content with at least one target voice attribute.
(As a result, once both the first and second conversion models 70 and 72 have been trained, which may be done offline if desired; the first and second conversion models 70 and 72 may be concatenated together as indicated in FIG. 3 in order to provide a mechanism for text-independent source to target voice conversion. In this regard, the first conversion model 70 may convert source speech into intermediate synthetic speech that may then be converted to corresponding target speech by the second conversion model. Thus, embodiments of the present invention may provide a source-to-TTS voice conversion stage concatenated with a TTS-to-target voice conversion stage in which the text (and therefore the corresponding training speech) corpuses need not be parallel.) (Col. 12, Lines 2-17).
Once the model is trained it converts source speech to target speech using matching features in the speech.
Tian et al. does not explicitly teach: wherein the first audio content and the second audio content have the same lexical content comprising a same sequence of words in a same language, and are time-synchronized according to segment-level start and end timestamps of lexical content of the first audio content synchronized with segment-level start and end timestamps of the same lexical content of the second audio content, wherein the segment-level start and end timestamps are selected from a group consisting of: word-level start and end timestamps, phrase-level start and end timestamps, sentence-level start and end timestamps, and syllable-level start and end timestamps, wherein duration of phones of the second audio content are controlled in response to segment-level durations defined by the segment-level start and end time stamps which are synchronized with the segment-level start and end time stamps of the first audio content;
However, Xie et al. teaches wherein the first audio content and the second audio content have the same lexical content comprising a same sequence of words in a same language,
(The to-be-converted voice data is partitioned in the order of the time that the data in the to-be-converted voice data is obtained. It can be understandable that, in another embodiment, partition extraction can be performed on the to-be-converted voice data in a sequential manner according to the preset unique partition duration to obtain the plurality of to-be-converted partition voices, where the partition voice only includes the unique partition voice.) (Col. 3, Lines 1-19).
(The voice conversion refers to mapping each acoustic feature parameter of a source voice to each acoustic feature parameter of a target voice while the expressed content does not change after the voice conversion.) (Col. 4, Lines 21-24).
Xie et al. partitions an input voice, converts it to a target voice, and outputs it in the same without changing the expressed content.
and are time-synchronized according to segment-level start and end timestamps of lexical content of the first audio content synchronized with segment-level start and end timestamps of the same lexical content of the second audio content,
(The to-be-converted voice data is partitioned in the order of the time that the data in the to-be-converted voice data is obtained. It can be understandable that, in another embodiment, partition extraction can be performed on the to-be-converted voice data in a sequential manner according to the preset unique partition duration to obtain the plurality of to-be-converted partition voices, where the partition voice only includes the unique partition voice. In which, a start time of the unique partition voice can be used as the partition mark of the to-be-converted partition voice, or the to-be-converted partition voice can be numbered in the order of generation (by partitioning the to-be-converted voice) and the number is used as the partition mark of the to-be-converted partition voice.) (Col. 3, Lines 1-29).
(The preset unique partition duration is the preset duration of the voice data that each to-be-converted partition voice has individually, and the user can set the preset unique partition duration according to actual needs. For example, the preset unique partition duration can be 0 ms, 100 ms, 200 ms, 300 ms, 400 ms, 500 ms, 600 ms, 800 ms, or 1000 ms.) (Col. 3, Lines 35-40).
The source voice and target voice are partitioned in time-synchronized intervals where consistent start times and durations are used to recreate the target voice in the same timing. A start time is a start timestamp and a set duration creates an end timestamp by adding that duration to the start timestamp.
which are synchronized with the segment-level start and end time stamps of the first audio content;
(In which, the restored partition voice is played sequentially according to the sequence of the partition marks carried by the restored partition voices. In another embodiment, the restored partition voice can be stored according to the partition mark carried by the restored partition voice.) (Col. 4, Lines 60-64).
The output is synchronized to the partitions of the input voice.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the voice conversion method as taught by Tian et al. to time-synchronize segments of the input voice as taught by Xie et al. This would have been an obvious improvement to allow real-time conversion of the source voice data at the same rate/cadence of the source (Xie et al. Col. 1, Lines 33-45).
Tian et al. in view of Xie et al. does not explicitly teach: wherein the segment-level start and end timestamps are selected from a group consisting of: word-level start and end timestamps, phrase-level start and end timestamps, sentence-level start and end timestamps, and syllable-level start and end timestamps, wherein duration of phones of the second audio content are controlled in response to segment-level durations defined by the segment-level start and end time stamps
However, Lee et al. teaches wherein the segment-level start and end timestamps are selected from a group consisting of: word-level start and end timestamps, phrase-level start and end timestamps, sentence-level start and end timestamps, and syllable-level start and end timestamps,
(The preparation of the speech segment for storage in the speech segment storage block (5) is as follows. A synthesis unit is first selected. Such synthesis units include phoneme, allophone, diphone, syllable, demisyllable, CVC, VCV, CV, VC unit (here, "C" stands for a consonant, "V" stands for a vowel phoneme, respectively) or combinations thereof. The synthesis units which are most widely used in the current speech synthesis method are the diphones and the demisyllables.) (Col. 8, Lines 21-29).
(In the speech segment storage block shown in FIG. 2, each voiced speech segment decomposed into as many wavelets as the number of pitch pulses according to the method shown in FIG. 4 is stored in the format as shown in FIG. 6A, which is referred to as the speech segment information. In a header field which is a fore part of the speech segment information, boundary time points B1, B2, . . . , BL which are important time points in the speech segment and pitch pulse positions P1, P2, . . . , PM of each pitch pulse signal used in synthesis of each wavelet is stored, in which the number of samples corresponding to each time point is recorded taking the first sample position of the first pitch pulse signal e1(n) as 0.) (Col. 15, Line 64 to Col. 16, Line 9).
Lee et al. teaches a system of voice conversion which identifies timestamps for phonemes, allophones, diphones, syllables, demi-syllables, and combinations of them which would create words, phrases, and sentences.
wherein duration of phones of the second audio content are controlled in response to segment-level durations defined by the segment-level start and end time stamps
(In FIG. 8A is illustrated the correlation between the original speech segment and the speech segment to be synthesized. The original boundary time points B1, B2, etc., indicated by dotted lines, the boundary time points B'1, B'2, etc. of the synthesized sound and the correlation between them indicated by the dashed lines are included in the time warping information received from the duration control subblock.) (Col. 21, Lines 1-8).
(In FIG. 8A, for example, because the first subsegment and the last subsegment of the original speech segment should be respectively compressed to 2/3 times and be expanded to 2 times, the correlation thereof appears as the lines of the slope of 2/3 and 2 in the time warping function of FIG. 8B, respectively. The second subsegment does not vary in its duration so as to appear as a line of slope of 1 in the time warping function.) (Col. 21, Lines 21-29)
These timestamps control the output sound durations using a time warping method. This can be visualized in Fig. 8
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the voice conversion method as taught by Tian et al. to identify timestamps for sounds as taught by Lee et al. This would have been an obvious improvement to ensure the duration of each sound does not vary in the transformed speech (Lee et al. Col. 21, Lines 28-29).
Regarding Claim 13, Tian et al. in view of Xie et al. and Lee et al. teaches the system of claim 1.
Furthermore, Tian et al. teaches wherein the S2S record of the S2S training dataset further includes an indication of a certain at least one second voice attribute selected from a plurality of voice attributes,
(As will be described in further detail below, if the target voice 104 is not already in the database of voices 111, the system 100 processes the voice 104 using a transformation engine 118 and maps it in a multi-dimensional discrete or continuous space 112 that represents encoded voice data. The representation is referred to as “mapping” the voices. When the encoded voice data is mapped, the vector space 112 makes characterizations about the voices and places them relative to one another on that basis. For example, part of the representation may have to do with pitch of the voice, or gender of the speaker.) (Col. 5, Lines 48-63).
Input target voice features are stored within the system in order to be used for converting source voices
and the S2S ML model is further fed the at least one target voice attribute for generating the outcome of the source audio content having the at least one target voice attribute.
(In step 306, the augmented voice profile 144 for the target 104 is taken and applied to the speech segment 103. In other words, the frequency of the speech segment 103 is transformed to reflect the frequency distributions present in the target voice 104. This transforms the speech segment 103 into the target voice 104. … In contrast, during the voice conversion shown in FIG. 1, the system 100 has already been trained on enough voices that the conversion can occur rather smoothly without the need for further training, resulting in real time or near real-time speech-to-speech conversions (although further training is optional).) (Col. 22, Lines 21-47).
Regarding Claim 14, Tian et al. teaches A system of near real-time adaptation of at least one voice attribute, comprising: at least one processor executing a code for:
(The conversion device 66 may be capable of transforming source speech 80 into target speech 82. In this regard, the conversion device 66 may be employed to build transformation models which may essentially include one or more trained Gaussian mixture models (GMMs) for transforming the source speech 80 into the target speech 82. In order to produce the transformation models, GMMs may be trained using training source speech data 84 and training target speech data 86 to determine corresponding conversion functions with respect to corresponding intermediate synthetic speech as described below.) (Col. 9, Lines 39-49).
(In one embodiment, the model trainer 78 and/or the TTS device 76 may be controlled by or otherwise embodied as the processing element 74 (e.g., the controller 20 of the mobile terminal 10 or a processor of a server, computer or other device).) (Col. 9, Lines 59-63)
feeding a source audio content comprising speech having at least one source voice attribute into a S2S ML model trained
(as illustrated in FIG. 4 may include processing source speech at a first voice conversion model trained with respect to conversion of training source speech to synthetic speech corresponding to the training source speech at operation 110. An output of the first voice conversion model may then be communicated to a second voice conversion model trained with respect to conversion to training target speech from synthetic speech corresponding to the training target speech at operation 120.) (Col. 14, Lines 57-65)
Source audio is provided to a speech conversion model.
on a training dataset of a plurality of S2S records,
(In an exemplary embodiment, the training data may be representative of, for example, audio corresponding to a given number of utterances spoken by a source voice and a potentially different given number of potentially different utterances spoken by a target voice which may be stored, for example, in a database, or gathered from free speech recorded from the source or target, respectively.) (Col. 14, Lines 1-8).
Tian et al. creates an S2S training set beginning with utterances gathered from source voices.
wherein an S2S record comprises: a first audio content comprising speech having at least one first voice attribute,
(Training of a GMM model (e.g., the second conversion model 72) for conversion between the synthetic speaker Z and target speaker Y is described below. Similarly, the GMM model for synthetic to target conversion can also be trained as N.sub.ZY(v, .mu..sub.l, .SIGMA..sub.l). The conversion function that converts synthetic feature z.sub.t to target feature y.sub.t is given by Equation (3).) (Col. 13, Lines 13-19).
Voice attributes referred to as features are identified in the conversion of source voice to target voice where a synthetic voice is made matching a feature of the target voice.
and a ground truth label of a second audio content comprising speech having at least one second voice attribute,
(In this regard, as shown in FIG. 3, the model trainer 78 may be configured to receive training source speech data 84 from a source speaker and training target speech data 86 from a target speaker. The training source speech data 84 and the training target speech data 86 may each be, for example, samples of spoken syllables, words, phrases, or sentences from the source speaker and target speaker, respectively.) (Col. 11, Lines 1-7).
(Training of a GMM model (e.g., the second conversion model 72) for conversion between the synthetic speaker Z and target speaker Y is described below. Similarly, the GMM model for synthetic to target conversion can also be trained as N.sub.ZY(v, .mu..sub.l, .SIGMA..sub.l). The conversion function that converts synthetic feature z.sub.t to target feature y.sub.t is given by Equation (3).) (Col. 13, Lines 13-19).
Training set including ground truth target features are used to train the model.
wherein the first audio content and the second audio content have the same lexical content comprising a same sequence of words in a same language, (XIE ET AL. ALSO TEACHES)
(Once in receipt of the training source text 88 and the training target text 90, the TTS device 76 may be configured to produce parallel training source synthetic speech 92 corresponding to the training source text 88 and parallel training target synthetic speech 94 corresponding to the training target text 90. The training source synthetic speech 92 may then be provided to the first conversion model 70 and the training target synthetic speech 94 may be provided to the second conversion model 72 (e.g., under the control of the model trainer 78 or the processing element 74).) (Col. 11, Lines 50-59).
Parallel speech shows that the lexical content remains the same from the source speech to target speech. This implies that the words are in the same sequence/language but because it is not specifically stated Xie et al. address it further below.
and obtaining a target audio content as an outcome of the S2S ML model for playing on a speaker,
(as illustrated in FIG. 4 may include processing source speech at a first voice conversion model trained with respect to conversion of training source speech to synthetic speech corresponding to the training source speech at operation 110. An output of the first voice conversion model may then be communicated to a second voice conversion model trained with respect to conversion to training target speech from synthetic speech corresponding to the training target speech at operation 120.) (Col. 14, Lines 57-65)
(The synthetic speech may then be communicated to an output device such as an audio mixer for appropriate mixing prior to delivery to another output device such as a speaker, or as in this case, a voice conversion model.) (Col. 10, Lines 60-63)
The speech conversion model creates an output to a speaker.
wherein the target audio content comprises at least one target voice attribute of the source audio content,
(Thus, in an exemplary voice conversion application, x and z may correspond to similar features from a source speaker X and synthetic speaker Z, respectively. For example, x and z may correspond to a line spectral frequency (LSF) extracted from the given short segment of the speeches of the source and synthetic speaker, respectively.) (Col. 12, Lines 36-42).
(The conversion function that converts synthetic feature z.sub.t to target feature y.sub.t is given by Equation (3).) (Col. 13, Lines 17-18).
The target audio contains a feature that is converted from a source audio feature shared with the synthetic audio.
Tian et al. does not explicitly teach: wherein the first audio content and the second audio content have the same lexical content comprising a same sequence of words in a same language, and are time-synchronized according to segment-level start and end timestamps of lexical content of the first audio content synchronized with segment-level start and end timestamps of the same lexical content of the second audio content, wherein the segment-level start and end timestamps are selected from a group consisting of: word-level start and end timestamps, phrase-level start and end timestamps, sentence-level start and end timestamps, and syllable-level start and end timestamps, wherein duration of phones of the second audio content are controlled in response to segment-level durations defined by the segment-level start and end time stamps which are synchronized with the segment-level start and end time stamps of the first audio content;
However, Xie et al. teaches wherein the first audio content and the second audio content have the same lexical content comprising a same sequence of words in a same language,
(The to-be-converted voice data is partitioned in the order of the time that the data in the to-be-converted voice data is obtained. It can be understandable that, in another embodiment, partition extraction can be performed on the to-be-converted voice data in a sequential manner according to the preset unique partition duration to obtain the plurality of to-be-converted partition voices, where the partition voice only includes the unique partition voice.) (Col. 3, Lines 1-19).
(The voice conversion refers to mapping each acoustic feature parameter of a source voice to each acoustic feature parameter of a target voice while the expressed content does not change after the voice conversion.) (Col. 4, Lines 21-24).
Xie et al. partitions an input voice, converts it to a target voice, and outputs it in the same without changing the expressed content.
and are time-synchronized according to segment-level start and end timestamps of lexical content of the first audio content synchronized with segment-level start and end timestamps of the same lexical content of the second audio content,
(The to-be-converted voice data is partitioned in the order of the time that the data in the to-be-converted voice data is obtained. It can be understandable that, in another embodiment, partition extraction can be performed on the to-be-converted voice data in a sequential manner according to the preset unique partition duration to obtain the plurality of to-be-converted partition voices, where the partition voice only includes the unique partition voice. In which, a start time of the unique partition voice can be used as the partition mark of the to-be-converted partition voice, or the to-be-converted partition voice can be numbered in the order of generation (by partitioning the to-be-converted voice) and the number is used as the partition mark of the to-be-converted partition voice.) (Col. 3, Lines 1-29).
(The preset unique partition duration is the preset duration of the voice data that each to-be-converted partition voice has individually, and the user can set the preset unique partition duration according to actual needs. For example, the preset unique partition duration can be 0 ms, 100 ms, 200 ms, 300 ms, 400 ms, 500 ms, 600 ms, 800 ms, or 1000 ms.) (Col. 3, Lines 35-40).
The source voice and target voice are partitioned in time-synchronized intervals where consistent start times and durations are used to recreate the target voice in the same timing. A start time is a start timestamp and a set duration creates an end timestamp by adding that duration to the start timestamp.
which are synchronized with the segment-level start and end time stamps of the first audio content;
(In which, the restored partition voice is played sequentially according to the sequence of the partition marks carried by the restored partition voices. In another embodiment, the restored partition voice can be stored according to the partition mark carried by the restored partition voice.) (Col. 4, Lines 60-64).
The output is synchronized to the partitions of the input voice.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the voice conversion method as taught by Tian et al. to time-synchronize segments of the input voice as taught by Xie et al. This would have been an obvious improvement to allow real-time conversion of the source voice data at the same rate/cadence of the source (Xie et al. Col. 1, Lines 33-45).
Tian et al. in view of Xie et al. does not explicitly teach: wherein the segment-level start and end timestamps are selected from a group consisting of: word-level start and end timestamps, phrase-level start and end timestamps, sentence-level start and end timestamps, and syllable-level start and end timestamps, wherein duration of phones of the second audio content are controlled in response to segment-level durations defined by the segment-level start and end time stamps
However, Lee et al. teaches wherein the segment-level start and end timestamps are selected from a group consisting of: word-level start and end timestamps, phrase-level start and end timestamps, sentence-level start and end timestamps, and syllable-level start and end timestamps,
(The preparation of the speech segment for storage in the speech segment storage block (5) is as follows. A synthesis unit is first selected. Such synthesis units include phoneme, allophone, diphone, syllable, demisyllable, CVC, VCV, CV, VC unit (here, "C" stands for a consonant, "V" stands for a vowel phoneme, respectively) or combinations thereof. The synthesis units which are most widely used in the current speech synthesis method are the diphones and the demisyllables.) (Col. 8, Lines 21-29).
(In the speech segment storage block shown in FIG. 2, each voiced speech segment decomposed into as many wavelets as the number of pitch pulses according to the method shown in FIG. 4 is stored in the format as shown in FIG. 6A, which is referred to as the speech segment information. In a header field which is a fore part of the speech segment information, boundary time points B1, B2, . . . , BL which are important time points in the speech segment and pitch pulse positions P1, P2, . . . , PM of each pitch pulse signal used in synthesis of each wavelet is stored, in which the number of samples corresponding to each time point is recorded taking the first sample position of the first pitch pulse signal e1(n) as 0.) (Col. 15, Line 64 to Col. 16, Line 9).
Lee et al. teaches a system of voice conversion which identifies timestamps for phonemes, allophones, diphones, syllables, demi-syllables, and combinations of them which would create words, phrases, and sentences.
wherein duration of phones of the second audio content are controlled in response to segment-level durations defined by the segment-level start and end time stamps
(In FIG. 8A is illustrated the correlation between the original speech segment and the speech segment to be synthesized. The original boundary time points B1, B2, etc., indicated by dotted lines, the boundary time points B'1, B'2, etc. of the synthesized sound and the correlation between them indicated by the dashed lines are included in the time warping information received from the duration control subblock.) (Col. 21, Lines 1-8).
(In FIG. 8A, for example, because the first subsegment and the last subsegment of the original speech segment should be respectively compressed to 2/3 times and be expanded to 2 times, the correlation thereof appears as the lines of the slope of 2/3 and 2 in the time warping function of FIG. 8B, respectively. The second subsegment does not vary in its duration so as to appear as a line of slope of 1 in the time warping function.) (Col. 21, Lines 21-29)
These timestamps control the output sound durations using a time warping method. This can be visualized in Fig. 8
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the voice conversion method as taught by Tian et al. to identify timestamps for sounds as taught by Lee et al. This would have been an obvious improvement to ensure the duration of each sound does not vary in the transformed speech (Lee et al. Col. 21, Lines 28-29).
Regarding Claim 24, Tian et al. in view of Xie et al. and Lee et al. teaches the system of claim 1.
Furthermore, Xie et al. teaches wherein the second audio content is generated from the first audio content using segment-level start and end timestamps of the first audio content.
(partitioning the to-be-converted voice data in an order of data obtaining time as a plurality of to-be-converted partition voices, where the to-be-converted partition voice data carries a partition mark. … The to-be-converted voice data is partitioned in the order of the time that the data in the to-be-converted voice data is obtained. It can be understandable that, in another embodiment, partition extraction can be performed on the to-be-converted voice data in a sequential manner according to the preset unique partition duration to obtain the plurality of to-be-converted partition voices, where the partition voice only includes the unique partition voice.) (Col. 3, Lines 1-19).
(In which, the restored partition voice is played sequentially according to the sequence of the partition marks carried by the restored partition voices. In another embodiment, the restored partition voice can be stored according to the partition mark carried by the restored partition voice.) (Col. 4, Lines 60-64).
Xie et al. generates a voice conversion output using start and end timestamps from the input
Claims 2, 3, 8, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Publication US 8751239 B2 (Tian et al.) in view of US Patent Publication US 11367456 B2 (Xie et al.) and US Patent Publication US 5617507 A (Lee et al.) and US Patent Publication US 10614826 B2 (Huffman et al.).
Regarding Claim 2, Tian et al. in view of Xie et al. and Lee et al. teaches the system of claim 1.
Tian et al. in view of Xie et al. and Lee et al. does not explicitly teach: wherein the S2S ML model generate the outcome of the source audio content with at least one target voice attribute as time aligned and having the same duration of the input source audio content with at least one source voice attribute, and each word and other non-verbal expressions of the outcome of the source audio content is spoken at the same time as the input source audio content.
However, Huffman et al. teaches wherein the S2S ML model generate the outcome of the source audio content with at least one target voice attribute as time aligned and having the same duration of the input source audio content with at least one source voice attribute, and each word and other non-verbal expressions of the outcome of the source audio content is spoken at the same time as the input source audio content.
(The transformation of voices is also referred to as timbre conversion. Throughout the application, “voice” and “timbre” are used interchangeably. The timbre of the voices allows listeners to distinguish and identify particular voices that are otherwise speaking the same words at the same pitch, accent, amplitude, and cadence. Timbre is a physiological property resulting from the set of frequency components a speaker makes for a particular sound. In illustrative embodiments, the timbre of the speech segment 103 is converted to the timbre of the target voice 104, while maintaining the original cadence, rhythm, and accent/pronunciation of the source voice 102.) (Col. 4, Lines 45-56).
(The system 100 does not have to make choices about all of the other characteristics like cadence, accent, etc. because these characteristics are provided by the input speech. Thus, the input speech (e.g., speech segment 103) is specifically transformed into a different timbre, while maintaining the other speech characteristics.) (Col. 10, Lines 24-30).
Huffman et al. teaches a voice conversion model which reproduces the timbre (cadence, accent, etc.) of the input speech in the output/target speech. This method maintains the duration of voice attributes for both verbal (accent) and non-verbal(amplitude) expressions. Amplitude represents a non-verbal audio as it is purely a measure of loudness.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the voice conversion model as taught by Tian et al. in view of Xie et al. and Lee et al. to maintain the timbre of voices as taught by Huffman et al. This would have been an obvious improvement the timbre of the voices allows listeners to identify voices that are otherwise speaking the same words at the same pitch, accent, amplitude, and cadence (Huffman et al. Col. 4, Lines 45-56).
Regarding Claim 3, Tian et al. in view of Xie et al. and Lee et al. teaches the system of claim 1.
Tian et al. in view of Xie et al. and Lee et al. does not explicitly teach: wherein at least one voice attribute comprises a non-verbal aspect of speech, selected from a group comprising: pronunciation, accent, voice timbre, voice identity, emotion, expressiveness, gender, age, and body type.
However, Huffman et al. teaches wherein at least one voice attribute comprises a non-verbal aspect of speech, selected from a group comprising: pronunciation, accent, voice timbre, voice identity, emotion, expressiveness, gender, age, and body type.
(The timbre of the voices allows listeners to distinguish and identify particular voices that are otherwise speaking the same words at the same pitch, accent, amplitude, and cadence. Timbre is a physiological property resulting from the set of frequency components a speaker makes for a particular sound. In illustrative embodiments, the timbre of the speech segment 103 is converted to the timbre of the target voice 104, while maintaining the original cadence, rhythm, and accent/pronunciation of the source voice 102.) (Col. 4, Lines 45-56).
Huffman maintains pitch, accent, timbre, and cadence when performing voice conversion.
Same motivation applies as in claim 2 for Huffman
Regarding Claim 8, Tian et al. in view of Xie et al. and Lee et al. teaches the system of claim 1.
Tian et al. in view of Xie et al. and Lee et al. does not explicitly teach: wherein at least one voice attribute comprises a non-verbal aspect of speech, selected from a group comprising: pronunciation, accent, voice timbre, voice identity, emotion, expressiveness, gender, age, and body type.
However, Huffman et al. teaches wherein verbal content and at least one non-verbal content of the first audio content is preserved in the second audio content and the S2S ML model is trained for preserving verbal content and at least one non- verbal content of the source audio content.
(The timbre of the voices allows listeners to distinguish and identify particular voices that are otherwise speaking the same words at the same pitch, accent, amplitude, and cadence. Timbre is a physiological property resulting from the set of frequency components a speaker makes for a particular sound. In illustrative embodiments, the timbre of the speech segment 103 is converted to the timbre of the target voice 104, while maintaining the original cadence, rhythm, and accent/pronunciation of the source voice 102.) (Col. 4, Lines 45-56).
(The system 100 does not have to make choices about all of the other characteristics like cadence, accent, etc. because these characteristics are provided by the input speech. Thus, the input speech (e.g., speech segment 103) is specifically transformed into a different timbre, while maintaining the other speech characteristics.) (Col. 10, Lines 24-30).
(However, it should be understood that data relating to the target voice 104 itself may implicitly relate to a plurality of voices, as the characteristics of other voices are already understood in some capacity via the discriminator's 142 learned parameters at the time it maps or refines the target voice 104. Furthermore, as the target voice 104 is refined through training or the addition of more voices to the vector space 112, the target voice 104 further provides data relative to a plurality of voices.) (Col. 18, Lines 1-8).
Verbal (accent) and non-verbal (amplitude) content is used to train the speech conversion model as additional target voices are added to the vector space.
Same motivation applies as in claim 2 for Huffman
Regarding Claim 15, Tian et al. in view of Xie et al. and Lee et al. teaches the system of claim 14.
Tian et al. in view of Xie et al. and Lee et al. does not explicitly teach: wherein the processor is further configured for: feeding the source audio content into a feature extractor that generates a first type of a first intermediate acoustic representation of the source audio content.
Furthermore, Huffman et al. teaches wherein the processor is further configured for: feeding the source audio content into a feature extractor that generates a first type of a first intermediate acoustic representation of the source audio content.
(Returning to FIG. 3, the process proceeds to step 306, which extracts frequency distributions from the analytical audio segments 124. The frequency distribution of any particular analytical audio segment 124 is different for every voice. This is why different speakers' timbres are distinguishable. … After the extraction is complete, the analytical audio segments 124 frequency data is obtained. The result is a set of frequency strengths at various points in time, which in illustrative embodiments are arranged as an image with frequency on the vertical axis and time on the horizontal axis (a spectrogram).) (Col. 10, Liens 31-57).
(The process begins at step 1002, which provides speech data that represents the speech segment 103 to the system 100. For example, the speech segment 103, which inherently contains speech data that represents the speech segment 103, may be provided to the input 108. Alternatively, the generator 140 can provide data that represents a speech segment (e.g., from a text input). Thus, the speech data that represents the speech segment 103 may be in the form of audio as a waveform, a spectrogram, vocoder parameters, or other data that encodes the prosody and phone content of the speech segment 103. Furthermore, the speech data may be the output of some intermediate of the neural network 116.) (Col. 21, Lines 56-67).
An intermediate representation of the features is made by extracting them into a spectrogram.
Same motivation applies as in claim 2 for Huffman
Claims 4 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Publication US 8751239 B2 (Tian et al.) in view of US Patent Publication US 11367456 B2 (Xie et al.) and US Patent Publication US 5617507 A (Lee et al.), US Patent Publication US 20220068257 A1 (Biadsy et al.), and "ETEH: Unified Attention-Based End-to-End ASR and KWS Architecture" (Cheng et al.).
Regarding Claim 4, Tian et al. in view of Xie et al. and Lee et al. teaches the system of claim 1.
Furthermore, Tian et al. teaches the S2S record of the S2S training dataset is created by: feeding the first audio content comprising speech having the at least one first voice attribute into the TTS ML model, wherein the first audio content is of the S2S record;
(In an exemplary embodiment, the conversion device 66 may include a first conversion model 70 and a second conversion model 72 and the training device 66 may include a model trainer 78 and a text-to-speech (TTS) device 76.) (Col. 9, Lines 50-53).
(In this regard, as shown in FIG. 3, the model trainer 78 may be configured to receive training source speech data 84 from a source speaker and training target speech data 86 from a target speaker. The training source speech data 84 and the training target speech data 86 may each be, for example, samples of spoken syllables, words, phrases, or sentences from the source speaker and target speaker, respectively. In an exemplary embodiment, the training source speech data 84 and the training target speech data 86 may comprise non-parallel corpuses of training speech. In other words, the syllables, words, phrases, or sentences from the source speaker need not match with the syllables, words, phrases, or sentences from the target speaker.) (Col. 11, Lines 1-27).
Tian et al. teaches this method of creating S2S records which uses a TTS model to create synthetic speech.
and obtaining the second audio content comprising speech having at least one second voice attribute as an outcome of the TTS ML model, wherein the second audio content is of the S2S record;
(Once in receipt of the training source text 88 and the training target text 90, the TTS device 76 may be configured to produce parallel training source synthetic speech 92 corresponding to the training source text 88 and parallel training target synthetic speech 94 corresponding to the training target text 90. The training source synthetic speech 92 may then be provided to the first conversion model 70 and the training target synthetic speech 94 may be provided to the second conversion model 72 (e.g., under the control of the model trainer 78 or the processing element 74).) (Col. 11, Lines 1-13).
(Training of a GMM model (e.g., the second conversion model 72) for conversion between the synthetic speaker Z and target speaker Y is described below. Similarly, the GMM model for synthetic to target conversion can also be trained as N.sub.ZY(v, .mu..sub.l, .SIGMA..sub.l). The conversion function that converts synthetic feature z.sub.t to target feature y.sub.t is given by Equation (3).) (Col. 13, Lines 13-19).
Tian et al. in view of Xie et al. and Lee et al. does not explicitly teach: wherein the processor is further configured for: creating a text-to-speech (TTS) training dataset of a plurality of TTS records, wherein a TTS record comprises a text transcription of a third audio content comprising speech having at least one-second voice attribute, and a ground truth indication of the third audio content comprising speech having at least one second voice attribute; and segmenting the first audio content into a plurality of single utterances, wherein the S2S record is for a single utterance.
However, Biadsy et al. teaches wherein the processor is further configured for: creating a text-to-speech (TTS) training dataset of a plurality of TTS records,
(Referring to FIG. 2C, the adaption stage 200c of the training process 200 includes using the set of spoken training utterances 305 collected during the personalized seed data collection stage 200a of FIG. 2A to adapt both a text-to-speech (ITS) model 210 and a reference S2S conversion model 301 to synthesize speech in the voice of the target speaker 104 and that captures the atypical speech (e g. ALS speech) associated with the target speaker 104. The adaption stage 200c may occur before, after, or concurrently with the data generation stage 200b of FIG. 2B. The TS model 210 may be pre-trained on input text to generate synthesized canonical fluent speech in the voices of one or more predefined speakers. As such, ground-truth speech samples used to train the TTS model 210 may be obtained from speakers with typical speech.) (para 54-55)
Biadsy et al. builds a dataset for a TTS model.
wherein a TTS record comprises a text transcription of a third audio content comprising speech having at least one-second voice attribute,
(The TS model 210 may be pre-trained on input text to generate synthesized canonical fluent speech in the voices of one or more predefined speakers. As such, ground-truth speech samples used to train the TTS model 210 may be obtained from speakers with typical speech.) (Paragraph 55)
Biadsy et al. builds a transcription of audio content comprising speech.
and a ground truth indication of the third audio content comprising speech having at least one second voice attribute;
(The TS model 210 may be pre-trained on input text to generate synthesized canonical fluent speech in the voices of one or more predefined speakers. As such, ground-truth speech samples used to train the TTS model 210 may be obtained from speakers with typical speech.) (Paragraph 55)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the voice conversion model as taught by Tian et al. in view of Xie et al. and Lee et al. to create a TTS training data set as taught by Biadsy et al. This would have been an obvious improvement to improve the capability of the TTS model that Tian et al. is also using (Biadsy et al. Paragraph 32).
Tian et al. in view of Xie et al., Lee et al., and Biadsy et al. does not explicitly teach: and segmenting the first audio content into a plurality of single utterances, wherein the S2S record is for a single utterance.
However, Cheng et al. teaches and segmenting the first audio content into a plurality of single utterances, wherein the S2S record is for a single utterance.
(The forced alignment procedure consists in finding HMM state sequence that best matches the sequence of the observed parameter vectors, given a parametric representation of the input utterance corresponding to the acoustic-phonetic units, i.e. phonemes, contained in the speech utterance. To accomplish this, Viterbi algorithm is per formed, so that a marker is produced for every transition between the final state of an HMM and the initial state of the succeeding one. With a well trained GMM-HMM model, these markers are expected to be located close to the transitions between the acoustic-phonetic units.) (Section 2D, Paragraph 1).
Cheng et al. segments an input utterance according to single units of the utterance. This is done for ASR models which is equivalent to this scenario where a source audio is being provided to a TTS model and has associated text created for it.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the voice conversion model as taught by Tian et al. in view of Xie et al., Lee et al., and Biadsy et al. to segment training utterances as taught by Cheng et al. This would have been an obvious improvement as segmentation and can result in higher confidence scores for ASR decoding which in this case would improve the accuracy of the data in the TTS dataset (Cheng et al. Section 1, Paragraph 1).
Regarding Claim 6, Tian et al. in view of Xie et al., Lee et al., Biadsy et al., and Cheng et al. teaches the system of claim 4.
Furthermore, Tian et al. teaches wherein feeding the first audio content into the TTS model further comprises: feeding the first audio content into a (speech recognition alignment component) (Cheng et al.) that in response to an input of the first audio content and text transcript of the first audio content,
(In an exemplary embodiment, the model trainer 78 may also be configured to provide, to the TTS device 76, training source text 88 corresponding to the training source speech data 84 and training target text 90 corresponding to the training target speech. In this regard, for example, the model trainer 78 could receive or generate text information corresponding to the training source text 88 and/or the training target text 90.) (Col. 11, Lines 14-22).
Tian et al. takes an input and text transcript for audio content.
Cheng et al. teaches feeding the first audio content into a (speech recognition alignment component)
(GMM-HMM forced alignment is a widely used method to generate FTA between a speech utterance and its text transcription. … he forced alignment procedure consists in finding HMM state sequence that best matches the sequence of the observed parameter vectors, given a parametric representation of the input utterance corresponding to the acoustic-phonetic units, i.e. phonemes, contained in the speech utterance. ) (Section 2D, Paragraph 1).
Speech recognition alignment is performed on the input audio.
Cheng et al. teaches generates an outcome of segment-level start and end time stamps,
(GMM-HMM forced alignment is a widely used method to generate FTA between a speech utterance and its text transcription. … The forced alignment procedure consists in finding HMM state sequence that best matches the sequence of the observed parameter vectors, given a parametric representation of the input utterance corresponding to the acoustic-phonetic units, i.e. phonemes, contained in the speech utterance. … the forced alignment process generates phoneme-level FTA (PFTA), i.e. the phonemes’ start and end time markers:) (Section 2D, Paragraphs 1-2)
Cheng et al. teaches using speech recognition alignment component and segmenting segment timestamps.
Tian et al. further teaches and feeding the segments with corresponding segment-level start and end time stamps into the TTS model for obtaining the second audio content as the outcome of the TTS,
(If the training source text 88 and/or the training target text 90 are generated at the model trainer 78, the generation thereof may be accomplished by a speech recognition device configured to receive the training source speech data 84 and the training target speech data 86, recognize text within the training source speech data 84 and the training target speech data 86, respectively, and generate the respective training source text 88 and the training target text 90 based on the recognized text. The speech recognition may be accomplished by any known method.)
(Once in receipt of the training source text 88 and the training target text 90, the TTS device 76 may be configured to produce parallel training source synthetic speech 92 corresponding to the training source text 88 and parallel training target synthetic speech 94 corresponding to the training target text 90.) (Col. 11, Lines 50-55)
Training text is fed to a model to produce speech
wherein the text transcript is the same for the first audio content and the second audio content.
(In an exemplary embodiment, the model trainer 78 may also be configured to provide, to the TTS device 76, training source text 88 corresponding to the training source speech data 84 and training target text 90 corresponding to the training target speech. … As an alternative, the model trainer 78 could communicate the training source speech data 84 and the training target speech data 86 to the TTS device 76 or to another device, and the TTS device 76 or other device may include functionality for converting speech to text or receiving the corresponding training source text 88 and training target text 90.) (Col. 11, Lines 14-55)
(Once in receipt of the training source text 88 and the training target text 90, the TTS device 76 may be configured to produce parallel training source synthetic speech 92 corresponding to the training source text 88 and parallel training target synthetic speech 94 corresponding to the training target text 90.) (Col. 11, Lines 50-55)
The transcript in the source speech and synthetic speech is the same content used to produce the target speech.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over US Patent Publication US 8751239 B2 (Tian et al.) in view of US Patent Publication US 11367456 B2 (Xie et al.) and US Patent Publication US 5617507 A (Lee et al.) and "Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis" (Jia et al.).
Regarding Claim 10, Tian et al. in view of Xie et al. and Lee et al. teaches the system of claim 1.
Tian et al. in view of Xie et al. and Lee et al. does not explicitly teach: wherein
However, Jia et al. teaches wherein the processor is further configured for: feeding the first audio content into a feature extractor that generates a first type of a first intermediate acoustic representation of the first audio content;
(Our system is composed of three independently trained neural networks, illustrated in Figure 1: (1) a recurrent speaker encoder, based on [22], which computes a fixed dimensional vector from a speech signal, (2) a sequence-to-sequence synthesizer, based on [15], which predicts a mel spectrogram from a sequence of grapheme or phoneme inputs, conditioned on the speaker embedding vector, and (3) an autoregressive WaveNet [19] vocoder, which converts the spectrogram into time domain waveforms. ) (Section 2, Paragraph 1).
Jia et al. performs voice conversion on user input. As can be seen in Fig. 1 graphemes and phonemes of the input are extracted and put into the encoder. A mel spectrogram is formed representing a first intermediate representation.
feeding the first type of the first intermediate representation into an analyzer component that generates a second type of the first intermediate acoustic representation of the first audio content;
(We follow [22], which proposed a highly scalable and accurate neural network framework for speaker
verification. The network maps a sequence of log-mel spectrogram frames computed from a speech
utterance of arbitrary length, to a fixed-dimensional embedding vector, known as d-vector [20, 9].) (Section 2.1, Paragraph 2).
The mel spectrogram frames are used in a neural network which represents a second type of the first intermediate representation.
at least one of: (i) feeding the second audio content into a voice attribute encoder that generates a second intermediate acoustic representation of the at least one second voice attribute of the second audio content, and (ii) obtaining the second intermediate acoustic representation of the second audio by indicating the at least one second voice attribute of the second audio content;
(The synthesizer is trained on pairs of text transcript and target audio. At the input, we map the text to
a sequence of phonemes, which leads to faster convergence and improved pronunciation of rare words
and proper nouns.) (Section 2.2 Paragraph 3).
As seen in Fig. 1, the phoneme/graphene (first audio content) is fed into an encoder that outputs an embedding vector (second representation)
combining the second type of the first intermediate acoustic representation of the first audio with the second intermediate acoustic representation of the second audio to obtain a combination,
(We extend the recurrent sequence-to-sequence with attention Tacotron 2 architecture [15] to support multiple speakers following a scheme similar to [8]. An embedding vector for the target speaker is concatenated with the synthesizer encoder output at each time step. In contrast to [8], we find that simply passing embeddings to the attention layer, as in Figure 1, converges across different speakers.) (Section 2.2, Paragraph 1).
From Fig. 1, the output from the speaker encoder is concatenated with the synthesizer encoder output.
and feeding the combination into a synthesizer component that generates a third intermediate acoustic representation representing the first audio content having the at least one second voice attribute;
(The network maps a sequence of log-mel spectrogram frames computed from a speech utterance of arbitrary length, to a fixed-dimensional embedding vector, known as d-vector [20, 9].) (Section 2.1, Paragraph 2).
Again in Fig. 1, the output of the concatenation is input into attention and decoder blocks to output a mel spectrogram for the speaker embeddings.
wherein the S2S record includes at least one of: (i) the combination and the ground truth label comprises the third intermediate acoustic representation; and (ii) the second type of the first intermediate acoustic representation of the first audio content, and the ground truth comprises the third intermediate acoustic representation.
(The mel spectrogram predicted by the synthesizer network captures all of the relevant detail needed for high quality synthesis of a variety of voices, allowing a multi speaker vocoder to be constructed by simply training on data from many speakers.) (Section 2.3, Paragraph 1).
A multi speaker is created using training data from the created mel spectrograms (third intermediate representation)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the voice conversion model as taught by Tian et al. in view of Xie et al. and Lee et al. to create a to encode the audio data into a vector form as taught by Jia et al. This would have been an obvious improvement to generate a variety of speakers from the spectrogram (Jia et al. Section 1, Paragraph 1).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over US Patent Publication US 8751239 B2 (Tian et al.) in view of US Patent Publication US 11367456 B2 (Xie et al.), US Patent Publication US 5617507 A (Lee et al.), "Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis" (Jia et al.), and US Patent Publication US 10614826 B2 (Huffman et al.).
Regarding Claim 11, Tian et al. in view of Xie et al., Lee et al., and Jia et al. teaches the system of claim 1.
Furthermore, Jia et al. teaches wherein the third intermediate acoustic representation is designed for being fed into a vocoder that generates and outcome for playing on a speaker
(Our system is composed of three independently trained neural networks, illustrated in Figure 1: (1) a recurrent speaker encoder, based on [22], which computes a fixed dimensional vector from a speech signal, (2) a sequence-to-sequence synthesizer, based on [15], which predicts a mel spectrogram from a sequence of grapheme or phoneme inputs, conditioned on the speaker embedding vector, and (3) an autoregressive WaveNet [19] vocoder, which converts the spectrogram into time domain waveforms.) (Section 2, Paragraph 1).
Jia et al. creates a third intermediate representation in the form of a mel spectrogram which is then fed into a vocoder.
Tian et al. in view of Xie et al., Lee et al., and Jia et al. does not explicitly teach: with the at least one second voice attribute while preserving lexical content and non-adapted non-verbal attributes of the first audio content
However, Huffman et al. teaches with the at least one second voice attribute while preserving lexical content and non-adapted non-verbal attributes of the first audio content.
(As an example, Arnold Schwarzenegger may use the system 100 to convert his speech segment 103 (e.g., “I'll be back”) into the voice/timbre of James Earl Jones. In this example, Arnold's voice is the source voice 102 and James' voice is the target voice 104. Arnold may provide a speech sample 105 of James' voice to the system 100, which uses the speech sample 105 to transform his speech segment (as described further below). The system 100 takes the speech segment 103, transforms it into James' voice 104, and outputs the transformed speech segment 106 in the target voice 104. Accordingly, the speech segment 103 “I'll be back” is output in James' voice 104.) (Col. 4, Line 45 to Col. 5, Line 9).
(In contrast, illustrative embodiments transform speech, rather than synthesize it, using speech-to-speech conversion (also referred to as voice-to-voice conversion). The system 100 does not have to make choices about all of the other characteristics like cadence, accent, etc. because these characteristics are provided by the input speech. Thus, the input speech (e.g., speech segment 103) is specifically transformed into a different timbre, while maintaining the other speech characteristics.) (Col. 10, Lines 24-30).
(The transformation of voices is also referred to as timbre conversion. Throughout the application, “voice” and “timbre” are used interchangeably. The timbre of the voices allows listeners to distinguish and identify particular voices that are otherwise speaking the same words at the same pitch, accent, amplitude, and cadence.) (Col. 4, Lines 45-56).
Speech is converted to new non-verbal attributes while maintaining lexical content. This can be seen in the “I’ll be back” example. Amplitude of the audio represents a non-verbal attribute
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the voice conversion model as taught by Tian et al. in view of Xie et al. and Lee et al. to preserve non-verbal attributes as taught by Huffman et al. This would have been an obvious improvement as the timbre of the voices allows listeners to identify voices that are otherwise speaking the same words at the same pitch, accent, amplitude, and cadence (Huffman et al. Col. 4, Lines 45-56).
Claim 17 and 18 is rejected under 35 U.S.C. 103 as being unpatentable over US Patent Publication US 8751239 B2 (Tian et al.) in view of US Patent Publication US 11367456 B2 (Xie et al.), US Patent Publication US 5617507 A (Lee et al.), US Patent Publication US 10614826 B2 (Huffman et al.), and US Patent Publication US 20220068257 A1 (Biadsy et al.).
Regarding Claim 17, Tian et al. in view of Xie et al., Lee et al., and Huffman et al. teaches the system of claim 15.
Tian et al. in view of Xie et al., Lee et al., and Huffman et al. does not explicitly teach: wherein the processor is further configured for: feeding the first type of the first intermediate representation into an analyzer component that generates a second type of the first intermediate acoustic representation of the source audio content.
However, Biadsy et al. teaches wherein the processor is further configured for: feeding the first type of the first intermediate representation into an analyzer component that generates a second type of the first intermediate acoustic representation of the source audio content.
(FIG. 1A illustrates a speech conversion model 300, 300a configured to convert input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 ... An associated speech conversion model 300 of the speech conversion system 100a includes a speech-to-speech (S2S) conversion model 300a configured to convert the input audio data 102 (e.g., input spectrogram) … The S2S conversion model 300a includes a spectrogram encoder 310 configured to encode the input audio data 102 into a hidden feature representation (e.g., a series of vectors) and a spectrogram decoder 320 configured to decode the hidden representation into the output audio data 106 corresponding to the synthesized canonical fluent speech representation.) (Paragraph 34).
Biadsy et al feeds an input to an encoder to produce a spectrogram which is later decoded into output audio data.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the voice conversion model as taught by Tian et al. in view of Xie et al., Lee et al., and Huffman et al. to utilize multiple intermediate representations as taught by Biadsy et al. This would have been an obvious improvement to improve the quality of the output sound over devices such as telephones (Biadsy et al. Paragraph 37).
Regarding Claim 18, Tian et al. in view of Xie et al., Lee et al., Huffman et al., and Biadsy et al. teaches the system of claim 17.
Biadsy et al. further teaches wherein the second type of the first intermediate acoustic representation comprises a temporal sequence of learned neural network representations of spoken content of the source audio content.
(The S2S conversion model 300a includes a spectrogram encoder 310 configured to encode the input audio data 102 into a hidden feature representation (e.g., a series of vectors) and a spectrogram decoder 320 configured to decode the hidden representation into the output audio data 106 corresponding to the synthesized canonical fluent speech representation. For instance, as the spectrogram encoder 310 receives the input audio data 102 of the utterance 108, the spectrogram encoder 310 may process five frames of audio and convert those five frames of audio to ten vectors. The vectors are not a transcription of the frames of audio data 102, but rather a mathematical representation of the frames of the audio data 102. In turn, the spectrogram decoder 320 may generate the output audio data 106 corresponding to the synthesized canonical fluent speech representation based on the vectors received from the spectrogram encoder 310.) (Paragraph 34).
(The decoder 214 may a neural network configured to receive, as input, the fixed-length context vectors 213 generated by the encoder neural network 212 and generate, as output for each fixed-length context vector 213, a corresponding frame of a mel-frequency spectrogram. A mel-frequency spectrogram is a frequency-domain representation of sound.) (Paragraph 59).
The encoder produces audio frame vectors in a spectrogram representing a temporal sequence.
Claims 19 and 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over US Patent Publication US 8751239 B2 (Tian et al.) in view of US Patent Publication US 11367456 B2 (Xie et al.), US Patent Publication US 5617507 A (Lee et al.), US Patent Publication US 10614826 B2 (Huffman et al.), US Patent Publication US 20220068257 A1 (Biadsy et al.) and "Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis" (Jia et al.).
Regarding Claim 19, Tian et al. in view of Xie et al., Lee et al., Huffman et al., and Biadsy et al. teaches the system of claim 17.
Furthermore, Tian et al. teaches wherein the processor is further configured for: at least one of: (i) obtaining a sample audio content having the at least one target voice attribute,
(The training source speech data 84 and the training target speech data 86 may each be, for example, samples of spoken syllables, words, phrases, or sentences from the source speaker and target speaker, respectively.) (Col. 10, Lines 60-63).
Sample audio with target attributes is obtained for training.
Tian et al. in view of Xie et al., Lee et al., Huffman et al. and Biadsy et al. does not explicitly teach: feeding the sample audio content into a voice attribute encoder that generates a second intermediate acoustic representation of the at least one target voice attribute of the sample audio content, and (ii) obtaining the second intermediate acoustic representation of the at least one target voice attribute according to an indication of the at least one target voice attribute.
However, Jia et al. teaches feeding the sample audio content into a voice attribute encoder that generates a second intermediate acoustic representation of the at least one target voice attribute of the sample audio content,
(The speaker encoder is used to condition the synthesis network on a reference speech signal from the desired target speaker. Critical to good generalization is the use of a representation which captures the characteristics of different speakers, and the ability to identify these characteristics using only a short adaptation signal, independent of its phonetic content and background noise. These requirements are satisfied using a speaker-discriminative model trained on a text-independent speaker verification task) (Section 2.1, Paragraph 1).
Fig. 1 shows the voice attribute being put into an encoder to form a second intermediate representation.
and (ii) obtaining the second intermediate acoustic representation of the at least one target voice attribute according to an indication of the at least one target voice attribute.
(We follow [22], which proposed a highly scalable and accurate neural network framework for speaker verification. The network maps a sequence of log-mel spectrogram frames computed from a speech utterance of arbitrary length, to a fixed-dimensional embedding vector, known as d-vector [20, 9]. The network is trained to optimize a generalized end-to-end speaker verification loss, so that embeddings of utterances from the same speaker have high cosine similarity, while those of utterances from different speakers are far apart in the embedding space. The training dataset consists of speech audio examples segmented into 1.6 seconds and associated speaker identity labels; no transcripts are used.) (Section 2.1, Paragraph 2).
The encoder embeds speaker identity labels.
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the voice conversion model as taught by Tian et al. in view of Xie et al., Lee et al., Huffman et al., and Biadsy et al. to encode the audio data into a vector form as taught by Jia et al. This would have been an obvious improvement to generate a variety of speakers from the spectrogram (Jia et al. Section 1, Paragraph 1).
Regarding Claim 21, Tian et al. in view of Xie et al., Lee et al., Huffman et al., Biadsy et al., and Jia et al. teaches the system of claim 19.
Furthermore, Jia et al. teaches wherein the indication of the at least one target voice attribute is selected from a plurality of candidate voice attributes that are learned during training.
(The network is trained to optimize a generalized end-to-end speaker verification loss, so that embeddings of utterances from the same speaker have high cosine similarity, while those of utterances from different speakers are far apart in the embedding space. The training dataset consists of speech audio examples segmented into 1.6 seconds and associated speaker identity labels; no transcripts are used.) (Section 2.1, Paragraph 2).
Voice attributes are pulled from a collection of different speakers.
Regarding Claim 22, Tian et al. in view of Xie et al., Lee et al., Huffman et al., Biadsy et al., and Jia et al. teaches the system of claim 19.
Furthermore, Jia et al. teaches wherein the processor is further configured for: combining the second type of the first intermediate acoustic representation of the first audio with the second intermediate acoustic representation of the at least one target voice attribute to obtain a combination,
(An embedding vector for the target speaker is concatenated with the synthesizer encoder output at each time step. In contrast to [8], we find that simply passing embeddings to the attention layer, as in Figure 1, converges across different speakers. … At the input, we map the text to a sequence of phonemes, which leads to faster convergence and improved pronunciation of rare words and proper nouns. The network is trained in a transfer learning configuration, using a pretrained speaker encoder (whose parameters are frozen) to extract a speaker embedding from the target audio, i.e. the speaker reference signal is the same as the target speech during training.) (Sections 2.2, Paragraphs 1-3).
The embedded audio is concatenated and converges with a target speaker embedding. See Fig. 1.
and feeding the combination into a synthesizer component that generates a third intermediate acoustic representation representing the source audio content having the at least one target voice attribute.
(At the input, we map the text to a sequence of phonemes, which leads to faster convergence and improved pronunciation of rare words and proper nouns. The network is trained in a transfer learning configuration, using a pretrained speaker encoder (whose parameters are frozen) to extract a speaker embedding from the target audio, i.e. the speaker reference signal is the same as the target speech during training.) (Sections 2.2, Paragraphs 1-3).
The output of the concatenation is decoded into a spectrogram representing the third intermediate representation. See Fig. 1
Regarding Claim 23, Tian et al. in view of Xie et al., Lee et al., Huffman et al., Biadsy et al., and Jia et al. teaches the system of claim 22.
Furthermore, Jia et al. teaches wherein the processor is further configured for feeding the third intermediate acoustic representation into a vocoder component that generates an outcome
(Our system is composed of three independently trained neural networks, illustrated in Figure 1: (1) a recurrent speaker encoder, based on [22], which computes a fixed dimensional vector from a speech signal, (2) a sequence-to-sequence synthesizer, based on [15], which predicts a mel spectrogram from a sequence of grapheme or phoneme inputs, conditioned on the speaker embedding vector, and (3) an autoregressive WaveNet [19] vocoder, which converts the spectrogram into time domain waveforms.) (Section 2, Paragraph 1).
As can be seen in Fig. 1, the output of the attention decoder (third representation) is input into a vocoder.
Tian et al. teaches having the at least one target voice attribute for playing on a speaker
(The synthetic speech may then be communicated to an output device such as an audio mixer for appropriate mixing prior to delivery to another output device such as a speaker, or as in this case, a voice conversion model.) (Col. 10, Lines 60-63)
Output of voice conversion played through speaker
Huffman et al. teaches while preserving lexical content and non-adapted non-verbal attributes of the source audio content.
(The timbre of the voices allows listeners to distinguish and identify particular voices that are otherwise speaking the same words at the same pitch, accent, amplitude, and cadence. Timbre is a physiological property resulting from the set of frequency components a speaker makes for a particular sound. In illustrative embodiments, 102.) (Col. 4, Lines 45-56) the timbre of the speech segment 103 is converted to the timbre of the target voice 104, while maintaining the original cadence, rhythm, and accent/pronunciation of the source voice
(In contrast, illustrative embodiments transform speech, rather than synthesize it, using speech-to-speech conversion (also referred to as voice-to-voice conversion). The system 100 does not have to make choices about all of the other characteristics like cadence, accent, etc. because these characteristics are provided by the input speech. Thus, the input speech (e.g., speech segment 103) is specifically transformed into a different timbre, while maintaining the other speech characteristics.) (Col. 10, Lines 24-30).
Huffman et al. maintains lexical content and non-verbal attributes in voice conversion.
Examiner’s Suggestions
I believe the simplest approach to overcoming the current prior art is to amend the independent claims to more closely match claim 7 of the Continuation in Part Application 17/731,474 (Claim 1 of US Patent Publication US 11848005 B2). Doing so would narrow the scope of the claims to something that we already know has been granted an allowance in the past. Obviously, this is not a guarantee, and any amendments are subject to further search and consideration.
Claim 7: A computer-implemented method of training a speech-to-speech (S2S) machine learning (ML) model for adapting at least one voice attribute of speech, comprising: creating an S2S training dataset of a plurality of S2S records, wherein an S2S record comprises a first audio content comprising speech having at least one first voice attribute, and a ground truth label of a second audio content comprising speech having at least one second voice attribute, wherein the first audio content and the second audio content have the same lexical content and are time-synchronized according to segment-level start and end timestamps, wherein duration of phones of the second audio content are controlled in response to segment-level durations defined by the segment-level start and end time stamps; creating a text-to-speech (TTS) training dataset of a plurality of TTS records, wherein a TTS record comprises a text transcription of a third audio content comprising speech having the at least one-second voice attribute, and a ground truth indication of the third audio content comprising speech having the at least one second voice attribute; training a TTS ML model on the TTS training dataset, wherein the S2S record of the S2S training dataset is created by: feeding the first audio content comprising speech having the at least one first voice attribute into the TTS ML model, wherein the first audio content is of the S2S record, wherein feeding the first audio content into the TTS model further comprises: segmenting the first audio content into segments comprising a plurality of single utterances, wherein the S2S record is for a single utterance; feeding the first audio content into a speech recognition alignment component that in response to an input of the first audio content and text transcript of the first audio content, generates an outcome of the segment-level start and end time stamps, and feeding the segments with corresponding ones of the segment-level start and end time stamps into the TTS ML model for obtaining the second audio content as the outcome of the TTS ML model, wherein the text transcript is the same for the first audio content and the second audio content; and obtaining the second audio content comprising speech having the at least one second voice attribute as the outcome of the TTS ML model, wherein the second audio content is of the S2S record; wherein the TTS ML model comprises: a voice attribute encoder that at least one of: dynamically generates the second audio content having the at least one second voice attribute in response to an input of the third audio content comprising speech having the at least one second voice attribute, statically generates the second audio content having the at least one second voice attribute in response to a selection of the at least one second voice attribute from a plurality of candidate voice attributes, and statistically generates the second audio content having the at least one second voice attribute, an encoder-decoder that generates a spectrogram in response to an input of the segments and corresponding ones of the segment-level start and end time stamps, a duration based modeling predictor and controller that controls durations of phones of the second audio content in response to an input of the segment durations defined by the segment-level start and end time stamps, and a vocoder that generates time-domain speech from the spectrogram; and training the S2S ML model using the S2S training dataset, wherein the S2S ML model is fed an input of a source audio content with at least one source voice attribute and generates an outcome of the source audio content with at least one target voice attribute.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS DANIEL LOWEN whose telephone number is (571)272-5828. The examiner can normally be reached Mon-Fri 8:00am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras D Shah can be reached at (571) 270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICHOLAS D LOWEN/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
05/30/2026
Prosecution Insights