DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
All objections/rejections not mentioned in this Office Action have been withdrawn by the Examiner.
Status of the Claims
Prior to entry of the amendment(s) and/or consideration of the argument(s), the status of the claims is as follows.
Claim(s) 1-28 is/are pending.
Claim(s) 1, 8-10, 15, and 22-24 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Non-Patent Literature to Borsos et al. (Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M. and Zeghidour, N., 2022. AudioLM: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and lang. proc., arXiv:2209.03143v1., hereinafter Borsos) with further evidence from Non-Patent Literature to Gulati et al. (Gulati, A., Qin, J., Chiu, C.C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y. and Pang, R., 2020. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100., hereinafter Gulati).
Claims 2, 5-7, 16, and 19-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Borsos as applied to claims 1 and 15 above, and further in view of Non-Patent Literature to Ao (Ao, J., Wang, R., Zhou, L., Wang, C., Ren, S., Wu, Y., Liu, S., Ko, T., Li, Q., Zhang, Y. and Wei, Z., 2022, May. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 5723-5738)., hereinafter Ao).
Response to Arguments
Applicant’s arguments regarding the prior art rejections under 35 U.S.C. §102/103, see pages 2-3 of the Response to Non-Final Office Action dated 15 January 2026, which was received on 05 March 2026 (hereinafter Response and Office Action, respectively), have been fully considered.
With respect to the rejection(s) of claim(s) 1 and 15 under 35 U.S.C. §102 as being anticipated by Borsos, applicant argues that Borsos fails to disclose “an audio encoder configured to... generate, as output, a corresponding sequence of audio encodings” and “a language model decoder configured to receive, as input, the sequence of audio encodings output from the audio encoder” as recited in claims 1 and 15. Applicant further argues that Borsos fails to disclose “generate, as output, an output sequence of speech features characterizing a continuation of the spoken prompt” as recited in claims 1 and 15. These arguments are not persuasive.
Applicant further cites section III of Borsos as supporting the applicant’s position that the encoding in Borsos is not an audio encoding. Though not considered necessary for the rejection, FIG. 1 provides a visual explanation of section III, as cited by the applicant in the response and forming the basis for the arguments. Thus, relevant portions of section III and FIG. 1 are incorporated into the arguments and explanations below, to offer further clarity regarding the cited sections and how they relate to the rejection.
All of the above arguments appear to turn on the generation of audio encodings. Therefore, we initially focus on the audio encodings and the audio encoders themselves, as described in Borsos. Applicant does not define the phrase audio encoder in the specification. As such, the broadest reasonable interpretation of “audio encoder” is a functional component configured to receive an audio signal as input and transform it into a compressed, structured, and/or lower-dimensional representation (the audio encoding), such as, for the purpose of downstream processing or transmission.
As previously indicated, Borsos teaches “an audio encoder configured to... generate, as output, a corresponding sequence of audio encodings.” As provided in the Office Action, Borsos discloses “w2v-BERT and SoundStream” which generate “the semantic tokens” and “the acoustic tokens” respectively, and “For speech continuation, we use prompts of 3 seconds” which are received as input by “w2v-BERT and SoundStream” {the audio encoder}. (Borsos, pg. 6, col. 1, lines 34-42; FIG. 2). SoundStream and w2v-BERT each receive the input waveforms for extracting the corresponding acoustic and semantic tokens, where the input waveform, the “prompt of 3 seconds” for “speech continuation,” is a sequence of speech features which characterizes the spoken prompt. Borsos further describes, “For generating the prompts, we truncate samples to the desired prompt length” thus describing the generating being based on a sequence of inputs, and then, using the sequence of inputs, w2v-BERT and Sound Stream “extract the corresponding w2v-BERT and SoundStream tokens.” As further shown in FIG. 1, the acoustic and semantic tokens exit the respective model prior to decoding, thus the output tokens remain encoded.
Applicant’s argument regarding the discussion of the AudioLM framework has been fully considered, but it is not persuasive. First and foremost, SoundStream is expressly described in Borsos as a “neural audio codec” which “adopts a convolutional encoder to map the input waveform to a sequence of embeddings” and the “RVQ discretizes these embeddings into codebook symbols, allowing the decoder to map “this discrete representation to real-valued embeddings.” This discrete representation, as generated from the audio waveform is, by definition, an audio encoding. As such, applicant’s arguments related to the model framework are insufficient to rebut the clear and unequivocal language in Borsos.
Further, the distinction that applicant is trying to make regarding the production of “discrete tokens (based on codebook indices from a finite vocabulary),” as not being “audio encodings” is unclear. In digital signal processing, an encoder’s job is to map continuous, real-world analog signals into a digital representation, so that it can be transmitted, stored, or processed efficiently. A finite vocabulary (also referred to as an “alphabet” or a “codebook”) is merely a set of rules by which the encoded digital representation described in Borsos is both encoded and decoded. The finite vocabulary is not a disqualifier. It is the exact mechanism by which the audio encoder described in Borsos functions. As claims 1 and 15 recite an audio encoder which “receive, as input, a sequence of speech features characterizing a spoken prompt; and generate, as output, a corresponding sequence of audio encodings,” the use of “codebook indices from a finite vocabulary” as part of that audio encoding does not affect the rejection in light of Borsos.
As such, and respectfully, examiner agrees with the applicant’s underlying argument regarding the finite vocabulary, but argument does not support applicant’s conclusion. As visualized in FIG. 1, Borsos does, in fact, describe an encoder block, which is a convolutional neural network which analyzes the audio over time and maps it to a lower-dimensional latent space. Borsos then performs Residual Vector Quantization (RVQ), which results in mapping continuous vectors to discrete codes using a “finite codebook” (or vocabulary). However, this fact merely emphasizes that SoundStream and w2v-BERT, in the context of AudioFM, is/are an audio encoder. Both models ingest the raw audio waveform, in parallel, to produce a joint sequence of discrete semantic and acoustic representations (encodings).
An audio encoder which requires operation in the absence of a vocabulary/codebook is not recited in the limitations as currently presented and is not required by the broadest reasonable interpretation of claim 1 and 15. However, as the Response only argues what Borsos is not (i.e., Borsos teaches X. X is not an audio encoding), rather than how the “audio encodings” are different, it is unclear what such limitations might look like. Therefore, applicant is invited to amend the claims, during normal prosecution and in light of specification support, such that the claims recite the desired limitations and such limitations can be duly considered.
As all of the argued distinctions revolve around the semantic and acoustic tokens not being an audio encoding, and the status as an audio encoding has been addressed here, it is believed that all arguments have been addressed. As such, the above arguments are not persuasive and the rejection is maintained over said arguments. In light of the arguments indicated by the applicant, the rejections of claims 1 and 15 are maintained as previously presented, with further clarification in light of the arguments presented and the discussion above.
Applicant further argues that the rejection(s) of dependent claims 2-14 and 16-28 should be withdrawn for at least the same reasons as independent claims 1 and 15. Applicant’s arguments are not persuasive for the reasons described above with relation to independent claims 1 and 15.
The Applicant has not provided any further statement and therefore, the Examiner directs the Applicant to the below rationale.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 8-10, 15, and 22-24 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Non-Patent Literature to Borsos et al. (Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M. and Zeghidour, N., 2022. AudioLM: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and lang. proc., arXiv:2209.03143v1., hereinafter Borsos) with further evidence from Non-Patent Literature to Gulati et al. (Gulati, A., Qin, J., Chiu, C.C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y. and Pang, R., 2020. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100., hereinafter Gulati).
Regarding claim 1, Borsos discloses A spoken language model comprising (The systems and methods described with reference to “AudioLM, a framework that enables high-quality audio generation with long-term coherent structure”; Borsos, ¶ pg. 1, col. 2, lines 6-9): an audio encoder (The combination of “w2v-BERT and SoundStream” which generate the “the semantic tokens” and “the acoustic tokens” respectively, is the audio encoder. As correctly indicated by the applicant, the general architecture of the combination of SoundStream and w2v-Bert corresponds to a “tokenizer model” labeled as “enc(x)”, which “maps x into a sequence y =enc(x), y = (y_1,..., y_T0) of discrete tokens from a finite vocabulary,” which is the process of encoding, where “SoundStream adopts a convolutional encoder to map the input waveform” to “acoustic tokens,” and w2v-BERT “map[s] the input audio waveform to a rich set of linguistic features” as the “semantic tokens,”; Borsos, ¶ pg. 3, col. 1, line 8 - col. 2, line 40; pg. 6, col. 1, lines 34-42; FIGS. 1 and 2) configured to: receive, as input, a sequence of speech features characterizing a spoken prompt (SoundStream and w2v-BERT each receive the input waveform for extracting the corresponding acoustic and semantic tokens, where the input waveform, the “prompt of 3 seconds” for “speech continuation,” is a sequence of speech features which characterizes the spoken prompt.; Borsos, ¶ pg. 6, col. 1, lines 34-42; FIGS. 1 and 2); and generate, as output, a corresponding sequence of audio encodings (“For generating the prompts, we truncate samples to the desired prompt length” thus describing generating, based on a sequence of inputs, and then, using the sequence of inputs, w2v-BERT and Sound Stream “extract the corresponding w2v-BERT and SoundStream tokens.” As further shown in FIG. 1, the tokens exit the respective model prior to decoding, thus they remain encoded.; Borsos, ¶ pg. 6, col. 1, lines 34-42; FIGS. 1 and 2); and a language model decoder (The course acoustic model (second stage) and the fine acoustic model (third state), which may be separate of “merged into a single stage”, and further, as indicated by the applicant, are the “decoder-only Transformer language model that operates on the discrete tokens” being the semantic tokens and the acoustic tokens; Borsos, ¶ pg. 3, col. 1, line 8 - 29; pg. 5, col. 2, line 1-11, FIG. 2) configured to: receive, as input, the sequence of audio encodings output from the audio encoder (“In the second stage, we concatenate the entire semantic token sequence (z≤ts, zˆ>ts) along with the coarse acoustic tokens of the prompt y≤Q0≤ta and feed it as conditioning to the coarse acoustic model,”; Borsos, ¶ pg. 5, col. 1, line 35 - col. 2, lines 11; FIG. 2) without any intermediary cross-attention applied to the sequence of audio encodings between the audio encoder and the language model decoder (Explicitly states that it uses “Decoder-only transformers” which rely on self-attention over a concatenated sequence, because the prompt is already inside its own sequence history (the prefix). As there is no separate encoder stack to look across to, there is no cross attention mechanism.; Borsos, ¶ pg. 5, col. 2, line 33-37; FIG. 2); and generate, as output, an output sequence of speech features (The model predicts SoundStream codebook indices (“Finally, we feed both the prompt and the sampled acoustic tokens to the SoundStream decoder to reconstruct a waveform xˆ.”), which represent the speech features.; Borsos, ¶ pg. 5, col. 2, line 1-11, FIG. 2) characterizing a continuation of the spoken prompt (These correspond to a response to the prompt, as part of the generation of continuations, thus characterizing a continuation of the spoken prompt.; Borsos, ¶ pg. 5, col. 1, line 35 - col. 2, lines 11; FIG. 2).
Regarding claim 8, Borsos discloses wherein the audio encoder comprises a plurality of multi-head attention layers (“w2v-BERT” is a “Conformer-based model [47]” which fundamentally relies on multi-head attention layers. Borsos further cites Gulati (the reference corresponding to footnote [47] above), to explain the conformer based model. Gulati explains at FIG. 1, that the conformer includes N conformer blocks, which comprise “two macaron-like feed-forward layers with halfstep residual connections sandwiching the multi-headed self-attention and convolution modules”; Borsos, ¶ pg. 3, col., lines 26-33, Gulati, ¶ pg. 1, col. 2, FIG. 1).
Regarding claim 9, Borsos discloses wherein each multi-head attention layer comprises a conformer layer comprising: a first feed-forward layer; a self-attention layer; a convolution layer; and a second feed-forward layer (The conformer-based model of Borsos, as further explained by Gulati, “includes N conformer blocks, which comprise ‘two macaron-like feed-forward layers with halfstep residual connections sandwiching the multi-headed self-attention and convolution modules’,” which is depicted, in ascending order, as a feed forward module, a multi-head self-attention module, a convolution module, and another feed forward module.; Borsos, ¶ pg. 3, col., lines 26-33, Gulati, ¶ pg. 1, col. 2, FIG. 1).
Regarding claim 10, Borsos discloses wherein the language model decoder comprises a prefix-language model architecture (Explicitly states that it uses “Decoder-only transformers” which rely on self-attention over a concatenated sequence, because the prompt is already inside its own sequence history (the prefix). As there is no separate encoder stack to look across to, there is no cross attention mechanism.; Borsos, ¶ pg. 5, col. 2, line 33-37; FIG. 2).
Regarding claim 15, Borsos discloses A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising (The systems and methods described with reference to “AudioLM, a framework that enables high-quality audio generation with long-term coherent structure” which are described therein as being computed or processed. As such, the execution as a computer-implemented method using well known data processing hardware (e.g., a computer) is necessarily implied; Borsos, ¶ pg. 1, col. 2, lines 6-9): receiving an input sequence of speech features characterizing a spoken prompt (SoundStream and w2v-BERT each receive the input waveform for extracting the corresponding acoustic and semantic tokens, where the input waveform, the “prompt of 3 seconds” for “speech continuation,” is a sequence of speech features which characterizes the spoken prompt.; Borsos, ¶ pg. 6, col. 1, lines 34-42; FIGS. 1 and 2); generating, using an audio encoder of a spoken language model, a corresponding sequence of audio encodings ((The combination of “w2v-BERT and SoundStream” which generate the “the semantic tokens” and “the acoustic tokens” respectively, is the audio encoder. As correctly indicated by the applicant, the general architecture of the combination of SoundStream and w2v-Bert corresponds to a “tokenizer model” labeled as “enc(x)”, which “maps x into a sequence y =enc(x), y = (y_1,..., y_T0) of discrete tokens from a finite vocabulary,” which is the process of encoding, where “SoundStream adopts a convolutional encoder to map the input waveform” to “acoustic tokens,” and w2v-BERT “map[s] the input audio waveform to a rich set of linguistic features” as the “semantic tokens, and for “generating the prompts, we truncate samples to the desired prompt length” thus describing generating, based on a sequence of inputs, and then, using the sequence of inputs, w2v-BERT and Sound Stream “extract the corresponding w2v-BERT and SoundStream tokens.” As further shown in FIG. 1, the tokens exit the respective model prior to decoding, thus they remain encoded.; Borsos, ¶ pg. 3, col. 1, line 8 - col. 2, line 40; pg. 6, col. 1, lines 34-42; FIGS. 1 and 2); and without applying any intermediary cross-attention to the sequence of audio encodings between the audio encoder and a language model decoder of the spoken language model (Explicitly states that it uses “Decoder-only transformers” which rely on self-attention over a concatenated sequence, because the prompt is already inside its own sequence history (the prefix). As there is no separate encoder stack to look across to, there is no cross attention mechanism.; Borsos, ¶ pg. 5, col. 2, line 33-37; FIG. 2), processing, using the language model decoder, the sequence of audio encodings generated by the audio encoder to generate an output sequence of speech features (The course acoustic model (second stage) and the fine acoustic model (third state), which may be separate of “merged into a single stage”, and further, as indicated by the applicant, are the “decoder-only Transformer language model that operates on the discrete tokens” being the semantic tokens and the acoustic tokens. “In the second stage, we concatenate the entire semantic token sequence (z≤ts, zˆ>ts) along with the coarse acoustic tokens of the prompt y≤Q0≤ta and feed it as conditioning to the coarse acoustic model,” where the course acoustic model and the fine acoustic model are the language model decoder, and the model predicts SoundStream codebook indices (“Finally, we feed both the prompt and the sampled acoustic tokens to the SoundStream decoder to reconstruct a waveform xˆ.”), which represent the speech features.; Borsos, ¶ pg. 3, col. 1, line 8 - 29; pg. 5, col. 2, line 1-11, FIG. 2) characterizing a continuation of the spoken prompt (These correspond to a response to the prompt, as part of the generation of continuations, thus characterizing a continuation of the spoken prompt.; Borsos, ¶ pg. 5, col. 1, line 35 - col. 2, lines 11; FIG. 2).
Regarding claim 22, the rejection of claim 15 is incorporated. Claim 22 is substantially the same as claim 8 and is therefore rejected under the same rationale as above.
Regarding claim 23, the rejection of claim 22 is incorporated. Claim 23 is substantially the same as claim 9 and is therefore rejected under the same rationale as above.
Regarding claim 24, the rejection of claim 15 is incorporated. Claim 24 is substantially the same as claim 10 and is therefore rejected under the same rationale as above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2, 5-7, 16, and 19-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Borsos as applied to claims 1 and 15 above, and further in view of Non-Patent Literature to Ao (Ao, J., Wang, R., Zhou, L., Wang, C., Ren, S., Wu, Y., Liu, S., Ko, T., Li, Q., Zhang, Y. and Wei, Z., 2022, May. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 5723-5738)., hereinafter Ao).
Regarding claim 2, the rejection of claim 1 is incorporated. Borsos disclose all of the elements of the current invention as stated above. However, Borsos fail(s) to expressly recite wherein the language model decoder is further configured to generate, as output, a transcription of the spoken prompt and a text representation of the continuation.
Ao teaches “a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning.” (Ao, ¶ Abstract). Regarding claim 2, Ao teaches wherein the language model decoder is further configured to generate, as output, a transcription of the spoken prompt and a text representation of the continuation (“Taking ASR as an example, the final model consists of the speech-encoder pre-net, encoder-decoder, text-decoder pre-net, and text-decoder post-net,” where “We use shared embeddings as the text-encoder pre-net and text-decoder pre/post nets. The pre-net transforms a token index into an embedding vector,” which when read in combination of the received acoustic and semantic tokens of Borsos, are received for both the spoken prompt and the continuation, as part of the operation of the system, and the “post-net transforms the hidden state into the probability distribution of tokens, normalized by the softmax function.” {a transcription of the spoken prompt and a text representation of the continuation}; Ao, ¶ pg. 3, col. 1, lines 28-33; pg. 4, col. 2, lines 39-50).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the high quality audio generation systems of Borsos to incorporate the teachings of Ao to include wherein the language model decoder is further configured to generate, as output, a transcription of the spoken prompt and a text representation of the continuation. Borsos teaches a “textless” NLP approach to audio generation, which, though capable of generating acoustically coherent speech, it does so without explicit text supervision, which limits the model’s ability to generate speech that precisely adheres to a specific semantic script or ground-truth text. Ao teaches a “unified-modal” framework that jointly models text and speech, and demonstrates that incorporating text data into the speech modeling process improves the model’s understanding of semantic information. A person having ordinary skill in the art would be motivated to incorporate Ao’s text processing components into the high quality acoustic generation of Borsos, to combine the generation of plausible speech with semantically accurate speech grounded in text transcription, as recognized by Ao. (Ao, Abstract).
Regarding claim 5 the rejection of claim 1 is incorporated. Borsos disclose all of the elements of the current invention as stated above. However, Borsos fail(s) to expressly recite further comprising an output acoustic projection layer, wherein: the output sequence of speech features comprises a sequence of output speech embeddings in a domain of the language model decoder; and the output acoustic projection layer is configured to project the sequence of output speech embeddings into an output sequence of Mel-spectrogram frames characterizing the continuation of the spoken prompt.
The relevance of Ao is described above with relation to claim 2. Regarding claim 5, Ao teaches further comprising an output acoustic projection layer, (“The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific (speech/text) pre/post-nets” which includes the “speech post-net”; Ao, ¶ Abstract) wherein: the output sequence of speech features comprises a sequence of output speech embeddings in a domain of the language model decoder (“The speech-decoder post-net... uses a linear layer fed with the decoder output” where the decoder output consists of the transformer’s internal hidden states (embeddings). These exist within the decoder’s internal dimensional space (the “domain” of the decoder) before they are converted into audio features.; Ao, ¶ Abstract, pg. 3, col. 1, lines 7-26); and the output acoustic projection layer is configured to project the sequence of output speech embeddings into an output sequence of Mel-spectrogram frames characterizing the continuation of the spoken prompt (Specifically identifies “log Mel-filterbank” {mel-frequency spectrogram frames} as the target output of the speech post-net, where the post-next linear layer performs the projection. Further, and in the context of speech generation tasks (TTS), these generated spectrograms represent the target speech output (the continuation); Ao, ¶ Abstract, pg. 3, col. 1, lines 7-26, pg. 14, col. 1, line 35 - col. 2, line 15).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the high quality audio generation systems of Borsos to incorporate the teachings of Ao to include further comprising an output acoustic projection layer, wherein: the output sequence of speech features comprises a sequence of output speech embeddings in a domain of the language model decoder; and the output acoustic projection layer is configured to project the sequence of output speech embeddings into an output sequence of mel-spectrogram frames characterizing the continuation of the spoken prompt. Borsos operates on discrete tokens derived from a neural audio codec, which can limit utility and functionality. Ao teaches the use of a “post-net” that projects decoder outputs directly into Mel-spectrogram frames, which are an industry standard format with numerous efficient, off the shelf vocoders. A person having ordinary skill in the art would be motivated to incorporate Ao’s Mel-spectrogram post-net to replace or augment the discrete token output head of Borsos, to increase interoperability with existing standard speech synthesis tools and reduce computing overhead associated with training and maintaining a separate system, as understood in the context of Ao. (Ao, Abstract).
Regarding claim 6 the rejection of claim 5 is incorporated. Borsos disclose all of the elements of the current invention as stated above. However, Borsos fail(s) to expressly recite wherein: a synthesizer is configured to convert the output sequence of mel-spectrogram frames into synthesized speech that conveys the continuation of the spoken prompt; and an audible output device is configured to audibly output the synthesized speech conveying the continuation of the spoken prompt.
The relevance of Ao is described above with relation to claim 2. Regarding claim 6, Ao teaches wherein: a synthesizer is configured to convert the output sequence of mel-spectrogram frames into synthesized speech that conveys the continuation of the spoken prompt (“We utilize the HiFi-GAN (Kong et al., 2020) vocoder to convert the log Mel-filterbank {output sequence of mel-spectrogram frames} to the raw waveform {into synthesized speech that conveys the continuation of the spoken prompt}.”; Ao, ¶ pg. 6, col. 1, lines 13-26); and an audible output device is configured to audibly output the synthesized speech conveying the continuation of the spoken prompt (Further disclosed evaluating the naturalness of the output on native speakers, thus disclosing the use of systems capable of outputting the synthesized speech {an audible output device is configured to audibly output the synthesized speech}; Ao, ¶ pg. 6, col. 1, lines 13-26).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the high quality audio generation systems of Borsos to incorporate the teachings of Ao to include wherein: a synthesizer is configured to convert the output sequence of mel-spectrogram frames into synthesized speech that conveys the continuation of the spoken prompt; and an audible output device is configured to audibly output the synthesized speech conveying the continuation of the spoken prompt. Borsos operates on discrete tokens derived from a neural audio codec, which can limit utility and functionality. Ao teaches the use of a “post-net” that projects decoder outputs directly into Mel-spectrogram frames, which are an industry standard format with numerous efficient, off the shelf vocoders. A person having ordinary skill in the art would be motivated to incorporate Ao’s Mel-spectrogram post-net to replace or augment the discrete token output head of Borsos, to increase interoperability with existing standard speech synthesis tools and reduce computing overhead associated with training and maintaining a separate system, as understood in the context of Ao. (Ao, Abstract).
Regarding claim 7 the rejection of claim 1 is incorporated. Borsos disclose all of the elements of the current invention as stated above. However, Borsos fail(s) to expressly recite wherein the sequence of speech features comprises an input sequence of mel-frequency spectrogram frames.
The relevance of Ao is described above with relation to claim 2. Regarding claim 7, Ao teaches wherein the sequence of speech features comprises an input sequence of mel-frequency spectrogram frames (Discloses “the raw waveform Xs is used as the input, and a sequence of the log Mel-filterbank features Xf extracted from raw audio using librosa tool” where “The speech decoder pre-net is a neural network composed of three fully connected layers with the ReLU activation, fed with the log Mel-filterbank Xf”; Ao, ¶ pg. 2, col. 2, lines 32-47).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the high quality audio generation systems of Borsos to incorporate the teachings of Ao to include wherein the sequence of speech features comprises an input sequence of mel-frequency spectrogram frames. Borsos operates on discrete tokens derived from a neural audio codec, which can limit utility and functionality. Ao teaches the use of a “post-net” that projects decoder outputs directly into Mel-spectrogram frames, which are an industry standard format with numerous efficient, off the shelf vocoders. A person having ordinary skill in the art would be motivated to incorporate Ao’s Mel-spectrogram post-net to replace or augment the discrete token output head of Borsos, to increase interoperability with existing standard speech synthesis tools and reduce computing overhead associated with training and maintaining a separate system, as understood in the context of Ao. (Ao, Abstract).
Regarding claim 16, the rejection of claim 15 is incorporated. Claim 16 is substantially the same as claim 2 and is therefore rejected under the same rationale as above.
Regarding claim 19, the rejection of claim 15 is incorporated. Claim 19 is substantially the same as claim 5 and is therefore rejected under the same rationale as above.
Regarding claim 20, the rejection of claim 19 is incorporated. Claim 20 is substantially the same as claim 6 and is therefore rejected under the same rationale as above.
Regarding claim 21, the rejection of claim 15 is incorporated. Claim 21 is substantially the same as claim 7 and is therefore rejected under the same rationale as above.
Allowable Subject Matter
Claims 3-4, 11-14, 17-18 and 25-28 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement for indicating allowability:
Regarding claim 3, and mutatis mutandis claim 17, the closest prior art of record Borsos teaches the limitations of claim 1 as presented above. However, Borsos does not expressly recite the limitations of claims 2 and claims dependent therefrom.
Ao does teach the limitations of claim 2, as explained in the rejection above.
However, none of the prior art references of record, either alone or in combination, teaches, suggests, or makes obvious the combination of limitations as recited in claims 3 and 17.
More specifically, the limitation of “wherein the language model decoder generates the output sequence of speech features autoregressively based on a concatenation of the transcription of the spoken prompt and the text representation of the continuation” is not taught by the prior art of record. None of the prior art of record disclose the above described concatenation. Borsos is silent regarding said concatenation. Examiner notes that, though Ao does specifically recite concatenation in two locations, neither of these relates to the above described concatenation. The concatenation between the x-vector and the “output of the speech decoder pre-net” relates to concatenation of the output with a representation of the speaker’s voice identity, which is not equivalent to a “transcription of the spoken prompt”. The remaining instances describing concatenation refer to concatenation of the models themselves, not the output. As such, Borsos and Ao fail to teach or suggest the above limitation and claims 3 and 17 is allowable over the cited references. Further, examiner is not aware of a prior art reference which can cure the above described deficiencies.
Regarding claim 4, claim 4 depends from claim 3 and incorporates all limitations therefrom. Therefore, claim 4 is allowable for at least the same reasons as claim 3.
Regarding claim 11, and mutatis mutandis claim 25, the closest prior art of record Borsos teaches all limitations of claim 1. However, Borsos does not specifically teach the recited limitations of claims 11.
Ao does teach wherein a training process jointly trains the audio encoder and the language model decoder by: obtaining a plurality of training utterances (Ao describes a unified modal framework that performs joint pre-training on large dataset of speech (utterances) and text, specifically proposing “ a unified modal SpeechT5 framework that explores the encoder-decoder pre-training… Leveraging large scale unlabeled speech and text data, we pre-train speech T5…” and “we use Librispeech…corpus…which contains 960 hours of speech.”; Ao, ¶ Abstract; pg. 13, col. 1, lines 11-18).
However, none of the prior art references of record, either alone or in combination, teaches, suggests, or makes obvious the combination of limitations as recited in claims 11 and 25.
More specifically, the limitation of “each respective training utterance comprising: audio data segmented into: a first sequence of reference speech features characterizing a corresponding prompt segment of the respective training utterance; and a second sequence of reference speech features characterizing a corresponding continuation segment of the respective training utterance; and a ground-truth transcript of the audio data, the ground-truth transcript segmented into: a first text segment representing a transcription of the corresponding prompt segment of the respective training utterance; and a second text segment representing a transcription of the corresponding continuation segment of the respective training utterance; for each respective training utterance: processing, by the audio encoder, the first sequence of reference speech features to generate a corresponding sequence of training audio encodings; processing, by the language model decoder: the corresponding sequence of training audio encodings to generate a corresponding predicted sequence of speech recognition results; and the first text segment to generate a corresponding predicted text segment; determining a first cross-entropy loss term based on the corresponding predicted sequence of speech recognition results and the first text segment representing the transcription of the corresponding prompt segment of the respective training utterance; and determining a second cross-entropy loss term based on the corresponding predicted text segment and the second text segment representing the transcription of the corresponding continuation segment of the respective training utterance; and training the spoken language model based on the first cross-entropy loss terms and the second cross-entropy loss terms determined for the plurality of training utterances” is not taught by the prior art of record.
Borsos does not teach the use of ground truth transcripts during training, as the disclosure is limited to audio. Ao teaches the use of “large-scale unlabeled speech and text corpus, and can be fine-tuned on labeled data (ASR, TTS), it does not teach the specific data preparation step of segmenting a single training utterance into audio prompt/continuation and text prompt/continuation pairs for the purpose of the specific joint training described therein. As the references fail to teach the initial limitation, and consequently fail to teach all limitations which build from said limitation. Therefore, Borsos and Ao fail to teach all limitations of claims 11 and 25 and claims 11 and 25 are allowable over the cited references. Further, the examiner is not aware of a prior art reference which can cure the above described deficiencies.
Regarding claims 12-14, and mutatis mutandis claims 26-28, the dependent claims depend from claims 11 and 25 and incorporate all limitations therefrom. Therefore, claims 12-14 and 26-28 are allowable for at least the same reasons as claims 11 and 25.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Alexandridis (U.S. Pat. No. 12,586,565) discloses techniques for biasing for entities during automatic speech recognition (ASR) processing, incorporating the use of a conformer encoder without cross attention as part of a speech generation process.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Sean E. Serraguard whose telephone number is (313)446-6627. The examiner can normally be reached 07:00-17:00 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel C. Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Sean E Serraguard/ Primary Examiner, Art Unit 2657
Prosecution Insights