DETAILED ACTION
This Office Action is responsive to amendments and arguments filed on August 17th, 2026. Claims 1, 5-7, 16 and 20 are amended, claims 4 and 19 are cancelled and claims 21-22 are added. Claims 1-3, 5-18 and 20-22 are pending and have been examined; hence, this action is made FINAL.
Any previous objections/rejections not mentioned in this Office Action have been withdrawn by the Examiner.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on October 17th, 2024 and February 26th, 2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Response to Arguments and Amendments
Regarding rejections made under 35 U.S.C. 101, Applicant argues that the "claims thus recite a specific combination of steps for generating synthetic audio signals to generate a set of training data, and the training a speech processing model on the set of training data to improve its downstream performance," (page 12 of Remarks).
Applicant's argument is persuasive, in that training a speech processing model using the claimed dataset in order to improve performance recites more than a judicial exception like gathering and organizing data. Accordingly, the rejections under 35 U.S.C. 101 are withdrawn.
Regarding rejections made under 35 U.S.C. 103, Applicant argues that "the cited portions of Lajszczak do not disclose or suggest generating 'a respective synthetic audio signal corresponding to the source audio signal and the speaker prompt embedding, wherein the respective synthetic audio signal represents the input speech represented by the source audio signal, spoken by the respective second speaker characterized by the speaker prompt embedding instead of by the respective first speaker that was speaking in the source audio signal,'" and "Lajszczak describes that the generated speech continuation represents 'consistent acoustic features, such as the style of speaking, voice characteristics, etc.' of the prompt. Thus, Lajszczak describes keeping the speaker identity constant so that the generated speech continuation sounds like the same speaker that spoke the prompt, rather than the synthetic audio signal representing 'the input speech represented by the source audio signal, spoken by the respective second speaker characterized by the speaker prompt embedding instead of by the respective first speaker that was speaking in the source audio signal,' as recited in claim 1," (page 13 of Remarks).
Applicant's argument is not persuasive in view of Lajszczak figure 7 and method 700, which teach a method for augmenting a voice dataset with synthesized audio samples under different speaker characters. The rejections under 35 U.S.C. 103 are maintained, with attention drawn to these sections of Lajszczak.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 5-18 and 20-22 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent 12,254,864 to Lajszczak et al. (hereinafter, "Lajszczak") in view of U.S. Patent 11,790,884 to Shakeri et al. (hereinafter, "Shakeri").
Regarding claims 1, 16 and 20, Lajszczak teaches a method, system and computer-readable medium for augmenting a target voice dataset comprising: receiving a plurality of source audio signals that each represent input speech spoken by a respective first speaker (column 20, line 16, "The method 700 may include receiving first audio data representing first speech (stage 710). The audio data may be prerecorded or received by a microphone of the device. An acoustic front end of the device may digitize the audio data and convert it into a frequency content format such as Mel-spectrograms.");generating, for each source audio signal, a respective semantic representation of the source audio signal (column 20, line 22, "The method 700 may include generating first encoded data representing semantic information and feature data of the first speech (stage 720). A neural network encoder, such as the speech unit encoder 130, may process the first audio data to generate encoded speech unit data. The encoded speech unit data may include one or more arrays, vector, matrices, or other data structures that represent an encoding of semantic information and acoustic information (e.g., feature data) of the audio data.");obtaining, for each of a plurality of second speakers, a respective speaker prompt embedding characterizing speech of the respective second speaker (column 20, line 56, "The method 700 may include receiving second audio data representing second speech (stage 740) and generating second encoded data representing semantic information and feature data of the second speech (stage 750). The second audio data may be from a target voice dataset to be augmented to create a larger dataset for training a TTS model.");generating, for each source audio signal, one or more synthetic audio signals, comprising:for each of one or more respective speaker prompt embeddings selected from the respective speaker prompt embeddings, providing an input comprising (i) the respective semantic representation of the source audio signal and (ii) the respective speaker prompt embedding to an audio generation model to generate a respective synthetic audio signal corresponding to the source audio signal and the speaker prompt embedding, wherein the respective synthetic audio signal represents the input speech represented by the source audio signal, spoken by the respective second speaker characterized by the speaker prompt embedding instead of by the respective first speaker that was speaking in the source audio signal (column 20, line 62, "An encoder may process the second audio data in a manner similar to that of the stage 720. The method 700 may include processing the second encoded data using the computer model to generate third encoded data (stage 760). The computer model may process a sequence of one or more encoded speech data units to predict one or more subsequent encoded speech data units. The predicted encoded data may represent a predicted continuation of the second speech," and column 21, line 7, "The method 700 may include generating third audio data representing first synthesized speech having voice characteristics similar to those of the second speech (stage 770).");generating a set of training data for training a speech processing model, wherein the set of training data comprises a plurality of paired training examples, each paired training example comprising(i) a respective source audio signal,(ii) a respective synthetic audio signal generated from the respective source audio signal, and(iii) a respective speaker prompt for the respective second speaker that is speaking in the respective synthetic audio signal (column 21, line 10, "The decoder may process the third encoded data to generate audio data. The audio data may be used to, for example, augment the target voice dataset. The augmented voice dataset may be used to train a TTS model, such as a TTS model 480, as described with reference to FIG. 4. A device such as the device 610 and/or a system such as the system 620 may use the trained TTS model to reproduce the target voice when, for example, narrating a book, delivering news, outputting a message, etc.," and column 3, line 40, "The trained encoder may be used to generate encoded speech units based on a human speech prompt of a target voice. The ASLM may receive the encoded speech units corresponding to the prompt, and generate encoded speech units corresponding to a predicted continuation of the speech. The trained decoder may process the continuation encoded speech units (and, in some cases, the prompt encoded speech units as well) and generate audio data. The audio data may represent an augmented dataset that can be used to train a TTS model."); andtraining the speech processing model on the set of training data (column 10, line 60, "The TTS model 480 thus trained may, in conjunction with a vocoder, reproduce the target voice as illustrated in FIG. 5 and described below.").
While Lajszczak is directed to generating an augmented voice dataset wherein each of the claimed elements is present, however, explicit reference to “respective” dataset elements isn’t disclosed. Shakeri may be referenced to teach audio training sets at column 13, line 48, "The speaker encoder 506 is trained using training set 501 comprising training examples 502-1, 502-2, 502-3. Each training example 502 comprises speech audio 504 and a speaker label 503 corresponding to the speaker identity of the respective speech audio 504. For example, if speech audio 504-1 and speech audio 504-2 were provided by the same speaker, then speaker labels 503-1 and 503-2 are identical. Speaker label 503-3 is different to that of speaker labels 503-1, 503-2 if speech audio 504-3 was provided by a different speaker to that of speech audio 504-1, 504-2. Speaker labels 503 may be represented by one hot vectors such that a different one-hot vector indicates each speaker in the training set 501. In general, the training set 501 comprises a plurality of examples of speech audio 504 for each speaker."
Lajszczak and Shakeri are considered analogous because they are each concerned with generating speech audio. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the audio generation method of Lajszczak with the respective labeling of Shakeri for the purpose of improving training organization. Given that all the claimed elements were known in the prior art, one skilled in the art could have combined the elements by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Regarding claims 2 and 17, Lajszczak teaches a method and system further comprising: generating, for each source audio signal, a transcript of the speech of the source audio signal (column 21, line 31, "In some cases, the ASR component 650 may generate transcripts of recorded speech in one or more of the target speech dataset 115, the speech dataset 215, and/or the training dataset 315 in the event that the dataset does not include a transcript and/or was not generated based the speaker(s) reading from a provided transcript.").
Regarding claims 3 and 18, Lajszczak teaches a method and system wherein the input further comprises (iii) the transcript of the speech of the source audio signal (column 9, line 58, "In some implementations, the ASLM 140 may be pre-trained using text data 315, and then fine-tuned using encoded speech unit data generated by the speech unit encoder 130 from speech in the training corpus (e.g., the training dataset 315). The text data 315 may represent annotations (e.g., a machine and/or human transcription) in the form of text, tokens, phonemes, etc.").
Regarding claims 5 and 21, Lajszczak teaches a method and system wherein the respective speaker prompt for the respective second speaker comprises the respective speaker prompt embedding from which the respective synthetic audio signal was generated (column 20, line 56, "The method 700 may include receiving second audio data representing second speech (stage 740) and generating second encoded data representing semantic information and feature data of the second speech (stage 750). The second audio data may be from a target voice dataset to be augmented to create a larger dataset for training a TTS model.").
Regarding claims 6 and 22, Lajszczak teaches a method and system wherein the respective speaker prompt for the respective second speaker comprises a speaker prompt audio signal represented by the respective speaker prompt embedding from which the respective synthetic audio signal was generated (column 20, line 56, "The method 700 may include receiving second audio data representing second speech (stage 740) and generating second encoded data representing semantic information and feature data of the second speech (stage 750). The second audio data may be from a target voice dataset to be augmented to create a larger dataset for training a TTS model.").
Regarding claim 7, Lajszczak teaches the method of claim 1, wherein obtaining, for each of a plurality of second speakers, a respective speaker prompt embedding characterizing speech of the respective second speaker comprises:
receiving, for each of the plurality of second speakers, a respective speaker prompt audio signal for the second speaker (column 20, line 56, "The method 700 may include receiving second audio data representing second speech (stage 740) and generating second encoded data representing semantic information and feature data of the second speech (stage 750). The second audio data may be from a target voice dataset to be augmented to create a larger dataset for training a TTS model."); and
generating each of the respective speaker prompt embeddings from the respective speaker prompt audio signals (column 20, line 56, "The method 700 may include receiving second audio data representing second speech (stage 740) and generating second encoded data representing semantic information and feature data of the second speech (stage 750). The second audio data may be from a target voice dataset to be augmented to create a larger dataset for training a TTS model.").
Regarding claim 8, Lajszczak teaches the method of claim 7, wherein generating each of the respective speaker prompt embeddings from the respective speaker prompt audio signals comprises providing each respective speaker prompt audio signal as input to an encoder to generate the respective speaker prompt embedding (column 3, line 40, "The trained encoder may be used to generate encoded speech units based on a human speech prompt of a target voice.").
Regarding claim 9, Lajszczak teaches the method of claim 8, wherein the encoder comprises an encoder neural network of a neural audio codec (column 3, line 10, " In some implementations, a neural network encoder can be trained to receive and process audio data to generate and output encoded speech unit data.").
Regarding claim 10, Lajszczak teaches the method of claim 1, wherein generating, for each source audio signal, a respective semantic representation of the source audio signal comprises providing each source audio signal as input to a semantic tokenizer to generate the respective semantic representation (column 6, line 61, "For example, the speech unit encoder 130 may be trained to extract certain information from audio data and encode it into encoded speech units, where the extracted/encoded information represents features relevant to the task to be performed; in this case, semantic and/or acoustic information about speech samples," and column 9, line 58, "In some implementations, the ASLM 140 may be pre-trained using text data 315, and then fine-tuned using encoded speech unit data generated by the speech unit encoder 130 from speech in the training corpus (e.g., the training dataset 315). The text data 315 may represent annotations (e.g., a machine and/or human transcription) in the form of text, tokens, phonemes, etc.").
Regarding claim 11, Lajszczak teaches the method of claim 1, wherein the audio generation model is configured to generate the respective synthetic audio signal by processing an encoded representation derived from the input using a token decoder neural network to generate a sequence of output tokens representing the respective synthetic audio signal (column 7, line 9, "The speech unit decoder 150 may process encoded speech units (e.g., received from the speech unit encoder 130 and/or the ASLM 140) and generate audio data representing speech corresponding to the encoded speech units… The speech unit decoder 150 may be a neural network model such as a CNN and/or RNN," and column 20, line 22, "The method 700 may include generating first encoded data representing semantic information and feature data of the first speech (stage 720). A neural network encoder, such as the speech unit encoder 130, may process the first audio data to generate encoded speech unit data. The encoded speech unit data may include one or more arrays, vector, matrices, or other data structures that represent an encoding of semantic information and acoustic information (e.g., feature data) of the audio data. The encoder may be trained in conjunction with a neural network decoder, such as the speech unit decoder 150, using techniques described with reference to FIG. 2. The encoder and decoder may be trained to encode and decode semantic information (e.g., representing phonemes and/or words) and features of the audio data (e.g., relevant to determining and/or reproducing features such as prosody, timbre, speaker identity, speaking style, emotion, etc.).").
Regarding claim 12, Lajszczak teaches the method of claim 1, wherein the audio generation model is configured to generate the respective synthetic audio signal by processing a masked representation of the respective synthetic audio signal derived from at least the input using a neural network to generate a sequence of output tokens representing the respective synthetic audio signal (column 20, line 39, "The method 700 may include training a computer model to predict a subsequent portion of the first encoded data based on a preceding portion of the first encoded data (stage 730). The encoded speech unit data generated during the stage 720 may be used to train a computer model to predict encoded speech units based on previously received encoded speech units. The computer model may be the ASLM 140, which may be trained according to the techniques described with reference to FIG. 3. For example, partial sequences of encoded speech units may be input to the computer model, which can be trained to correctly predict masked encoded speech units. The predicted encoded speech units may represent speech that is consistent both semantically and acoustically with the preceding speech, in that the predicted speech may represent natural language reasonably likely to follow the preceding speech, and have voice characteristics similar to the preceding speech.").
Regarding claim 13, Lajszczak teaches the method of claim 1, wherein the speech processing model is configured to generate an output audio signal by processing an encoded representation derived from an input source audio signal and an input speaker prompt for a speaker using a token decoder neural network to generate a sequence of output tokens representing the output audio signal (column 12, line 26, "The output of the preprocessing component 520 may be a symbolic linguistic representation, which may include a sequence of phonetic units. In some implementations, the sequence of phonetic units may be annotated with prosodic characteristics. In some implementations, prosody may be applied in part or wholly by a TTS model 480. This symbolic linguistic representation may be sent to the TTS model 480 for conversion into audio data (e.g., in the form of Mel-spectrograms or other frequency content data format).").
Regarding claim 14, Lajszczak does not explicitly teach a method “wherein the speech processing model is configured to generate an output audio signal by:obtaining a stream of input source audio tokens for an input source audio signal up to a current time step;obtaining a stream of input speaker audio tokens for an input speaker prompt for a speaker up to the current time step; andprocessing an encoded representation derived from at least some of the input source audio tokens up to the current time step and at least some of the input speaker audio tokens up to the current time step using a token decoder neural network to predict a stream of audio output tokens representing at least part of the output audio signal.”
Shakeri teaches a method for speech generation wherein the speech processing model is configured to generate an output audio signal by:
obtaining a stream of input source audio tokens for an input source audio signal up to a current time step (column 8, line 18, "The synthesizer 202 may comprise a speech content encoder to generate one or more speech content encodings for the input data. The input data comprises an indication of an utterance for each of a plurality of input time steps.");
obtaining a stream of input speaker audio tokens for an input speaker prompt for a speaker up to the current time step (column 8, line 43, "In some embodiments, the synthesizer 202 may further comprise a speech style encoder to generate a speech style encoding from speech style features. The speech style encoder may comprise a feedforward network comprising one or more fully connected layers. Each fully connected layer receives an input and applies a learned linear transformation to the input. The fully connected layer may further apply a non-linear transformation to generate an output for the layer. The input of a first fully connected layer comprises the speech style features, and the input to subsequent fully connected layers comprises the output of a previous fully connected layer."); and
processing an encoded representation derived from at least some of the input source audio tokens up to the current time step and at least some of the input speaker audio tokens up to the current time step using a token decoder neural network to predict a stream of audio output tokens representing at least part of the output audio signal (column 8, line 55, "Speech content encodings and speech style encodings may be combined to generate one or more combined encodings. A combining operation may comprise any binary operation resulting in a single encoding. For example, the combination may be performed by an addition, an averaging, a dot product, or a Hadamard product. Source acoustic features may be generated from processing the combined output. The generating may comprise decoding the one or more combined encodings by a decoder of the synthesizer to generate source acoustic features. The decoder may comprise one or more recurrent layers. The decoder may further comprise an attention mechanism. For each output time step of a plurality of output time steps, the combined encoding for each input time step may be received. The attention mechanism may generate an attention weight for each combined encoding. The attention mechanism may generate a context vector for the output time step by averaging each combined encoding using the respective attention weight. The decoder may process the context vector of the output time step to generate source acoustic features for the output time step.").
Lajszczak and Shakeri are considered analogous because they are each concerned with speech generation. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified data processing of Lajszczak with the time steps of Shakeri for the purpose of improving real-time processing performance. Given that all the claimed elements were known in the prior art, one skilled in the art could have combined the elements by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Regarding claim 15, Lajszczak teaches the method of claim 1, wherein the speech processing model is configured to generate an output audio signal by processing a masked representation of the output audio signal derived from an input source audio signal and an input speaker prompt for a speaker using a neural network to generate a sequence of output tokens representing the output audio signal (column 7, line 34, "While the speech unit decoder 150 may appear analogous to a TTS model in that it receives a form of data representing speech and generates audio data, the speech unit decoder 150 may be distinguished from TTS techniques because TTS typically receives word token or phoneme data (and, in some cases, conditioning data related to desired voice characteristics) and generates synthesized speech, whereas the speech unit decoder 150 receives encoded speech unit data that contains a much richer representation of speech and which may not map one-to-one to particular phonemes and/or words.").
While the decoder used by Lajszczak is taught to be distinct from a system the uses tokens to generate output audio, the conventional TTS system disclosed in Lajszczak reads on the claim limitation.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
U.S. Patent Application Publication 2022/0051654 to Finkelstein et al.
U.S. Patent Application Publication 2023/0018384 to Finkelstein et al.
U.S. Patent Application Publication 2024/0233713 to Agostinelli et al.
U.S. Patent Application Publication 2024/0304175 to Erdmann et al.
U.S. Patent Application Publication 2024/0304180 to Singh et al.
U.S. Patent Application Publication 2024/0404505 to Nguyen et al.
U.S. Patent Application Publication 2025/0006175 to Coppo et al.
WIPO Publication 2022/133630 to Zhao et al.
China Invention Application 112530413 to Xue.
"Non-parallel training in voice conversion using an adaptive restricted Boltzmann machine” by Nakashika et al.
"The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods” by Lorenzo-Trueba et al.
"Transfer learning from speech synthesis to voice conversion with non-parallel training data” by Zhang et al.
"W2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training” by Chung et al.
"Accent Conversion using Pre-trained Model and Synthesized Data from Voice Conversion” by Nguyen et al.
"Overview of voice conversion methods based on deep learning” by Walczyna and Piotrowski.
"SoundStorm: Efficient parallel audio generation” by Borsos et al.
"AudioLM: a language modeling approach to audio generation” by Borsos et al.
"Contrastive Learning from Synthetic Audio Doppelgangers” by Cherep and Singh.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SEAN T SMITH whose telephone number is (571)272-6643. The examiner can normally be reached Monday - Friday 8:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, PIERRE-LOUIS DESIR can be reached at (571) 272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SEAN THOMAS SMITH/Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659
Prosecution Insights