Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
This application is a continuation of co-pending International Application No. PCT/EP2023/057107, filed March 20, 2023 which is incorporated by reference in its entirety, and additionally claims priority from European Application No. 22163062.7, filed March 18, 2022, and from European Application No. 22182048.3, filed June 29, 2022, which are also incorporated by reference in their entirety. Claims 1, 2, 4-22, 25 have been afforded the benefit of the March 18, 2022 filing date.
Information Disclosure Statement
The IDS dated June 5, 2026 and July 28, 2026 have been considered and placed in the application file.
Response to Arguments Regarding Claim Interpretation – 35 USC § 112(f)
In light of the arguments regarding the previous 112(f) interpretation, the examiner agrees that the specification describes the corresponding structure as non-limiting examples; therefore, the claim scope will not be limited to the structure described in the specification and instead it will interpreted according to broadest reasonable interpretation.
Response to Arguments Regarding Claim Rejections - 35 USC § 103
Regarding Claims 1, 2, 4-22, 25, Applicant’s remarks filed on July 21, 2026 have been fully considered. The examiner has reviewed each of the Applicant’s arguments and respectfully responds as follows.
Applicant’s Arguments
As the Office Action acknowledges, Zeghidour does not teach the claimed first processing block, preconditioning learnable layer, conditioning learnable layer, styling element, or recurrent preconditioning layer. (See Office Action at 6-7.) Instead, the Office Action relies on Battenberg to supply those limitations, asserting that incorporating Battenberg's alleged preconditioning, conditioning, and styling architecture into SoundStream would improve temporal modeling, controllability, and quality. Applicant respectfully submits that the proposed combination does not teach or suggest the amended independent claims. In particular, Zeghidour discloses SoundStream as an end-to-end neural audio codec having a fully convolutional encoder, a residual vector quantizer, and a fully convolutional decoder that reconstructs an approximation of the original waveform from quantized embeddings. (See Zeghidour at 1-4.) The amended claims separately recite "a first data provisioner configured to provide, for a given frame, first data derived from an input signal" and "at least one preconditioning learnable layer configured to receive the bitstream, or a processed version thereof' and output "target data representing the audio signal in the given frame." The Office Action's mapping of Zeghidour's residual vector quantizer to the claimed "first data provisioner" does not account for these distinct claimed data paths.
Examiner’s response
Applicant’s argument has been considered but is not persuasive. Applicant is correct that Zeghidour alone does not teach the claimed recurrent preconditioning and styling/conditioning arrangement, and Zeghidour does not specifically state that Battenberg’s arrangement would improve temporal modeling, controllability, or quality. However, the rejection relies on the combination of the references. Zeghidour provides the bitstream based audio codec and decoder, while Battenberg provides the recurrent and conditioning teachings. Zeghidour explains that TTS techniques have been used in neural audio codecs and itself used conditioning in its codec. Therefore, there is support for using Battenberg’s known recurrent and conditioning techniques with Zeghidour’s decoder.
Applicant also argues that the claim requires different data paths- the first data provisioner provides first data derived from the input signal, while the preconditioning layer separately receives the bitstream or processed bitstream and outputs target data. Applicant argues that mapping Zeghidour’s residual vector quantizer to the first data provisioner does not account for these separate paths. However, the rejection does not rely on the residual vector quantizer as providing all of the claimed data. Zeghidour teaches separate processing stages, where the input audio is encoded into a representation, that representation is quantized for the compressed bitstream, and the decoder receives the quantized representation to reconstruct the audio. Battenberg is then relied upon for the additional recurrent/precondition processing. Thus, the rejection relies on the combined processing stages of Zeghidour and Battenberg, rather than treating the output of Zeghidour’s residual vector quantization as all of claimed data.
Applicant’s Arguments
Battenberg is directed to a text-to-speech architecture in which encoder pre-net 114 receives character embeddings derived from one-hot text characters and generates transformed character embeddings. However, Battenberg does not receive "the bitstream, or a processed version thereof," and it does not output "target data representing the audio signal in the given frame." (See Battenberg at 4-5, 8.) Battenberg's GRU- RNN 510 is part of a separate reference encoder 500 that receives reference audio spectrogram slices and outputs a prosody embedding PE 550, not part of encoder pre-net 114 and not part of any preconditioning layer that receives a bitstream or processed bitstream. (See Battenberg at 11- 12.) Battenberg's style token layer 610 likewise does not process the claimed target data output by the preconditioning learnable layer; it receives a prosody embedding PE 550 from deterministic reference encoder 500 and uses it as a query to an attention module that produces a weighted sum of global style tokens for input to the text encoder of a TTS model. (See Battenberg at 13-14.) Thus, Battenberg's text pre-net, reference-audio GRU, and style-token layer are separate mechanisms in a TTS prosody/style-transfer architecture, not a single bitstream-derived conditioning pipeline.
Examiner’s Response
Applicant’s argument has been considered but is not persuasive. Applicant correctly observes that Battenberg’s encoder pre net 114 receives character embeddings and does not itself expressly receive a compressed audio bitstream. However, the rejection does not solely rely upon Battenberg to teach the claimed invention. Zeghidour supplies the compressed bitstream architecture, where the encoder and quantizer transmit a compressed bitstream to the receiver for decoding of the audio signal. Applicant also argues that encoder pre-net 114 does not itself output the claimed target data representing the audio signal in the given frame. The rejection however, does not rely solely upon prenett114 for all of Battenberg’s teachings. Battenberg additionally teaches an audio-based reference encoder 500 that receives reference audio represented as spectrogram slices and processes that using convolutional layers followed by GRU-RNN. The GRU output is processed to produce prosody embedding, which captures the characteristics of the reference audio. Applicant also says that the GRU is located in reference encoder 500 rather than encoder pre-net 114. This is true. The rejection does not require incorporation of Battenburg’s particular component arrangement into Zeghidour. Rather, Battenberg demonstrates that recurrent processing of an audio derived representation to generate conditioning information was known, while Zeghidour supplies the bitstream derived audio decoding pathway into which that known recurrent processing is incorporated.
Additionally, Battenberg’s reference encoder and style-token layer are separate mechanisms but that does not negate their teachings. Battenberg teaches that style-token layer 610 receives prosody embedding from reference encoder, generates combination weights, and produces a style embedding used to condition the TTS model. Thus, Battenberg itself establishes a functional relationship between audio derived recurrent processing and subsequent conditioning, even though functions are distributed among identified components.
Applicant’s Arguments
The cited references also do not teach or suggest the amended styling limitation. The amended claims do not merely require that conditioning information be applied in some general manner. Rather, the claims require that the conditioning feature parameters themselves are the "weight parameters and bias parameters" used to elementwise linearly transform the first data or normalized first data. For example, the specification describes that operation as a denormalization or styling operation in which conditioning parameters y and p are applied through element-by-element multiplication and addition, such as y*c+3. (See Specification at 21-22, 33- 34.) Battenberg's reference embeddings, style tokens, and style embeddings condition a TTS text encoder or speech synthesis model, but the Office Action does not identify where Zeghidour or Battenberg generates frame-specific conditioning feature parameters from the claimed target data and uses those same parameters as the multiplicative and additive coefficients of the claimed transformation.
Examiner’s Response
Applicant’s arguments is not persuasive in view of Zeghidour’s FiLM disclosure. Zeghidour applies conditioning through the elementwise transformation
PNG
media_image1.png
300
621
media_image1.png
Greyscale
Where Y and B are coefficients generated by a linear layer. Thus y provides the multiplicative weight/scale and B provides the additive bias applied directly to the feature. Although Zeghidour does not generate these parameters from the claimed target data, Battenberg is relied upon for deriving conditioning information from audio data. Therefore, the rejection relies on the combination of Battenberg’s audio derived conditioning with Zeghidour’s FiLM transformation.
Applicant’s Arguments
Zeghidour's optional FiLM conditioning also does not supply the missing limitation. Zeghidour computes FiLM coefficients from a two-dimensional one-hot denoising-mode input, not from target data representing the audio signal in the given frame and output by a preconditioning learnable layer receiving the bitstream or a processed version thereof.
Examiner’s Response
Applicant correctly identifies that Zeghidour alone generates y and B from a denoising mode input rather than the claimed target data. However, Zeghidour is relied upon for the known FiLM operation of generating conditioning coefficients and applying those coefficients directly to network features, while Battenberg supplies audio derived prosody/style information used for conditioning. This rejection relies on the combined teachings.
Applicant’s Arguments
Applicant also respectfully submits that the proposed combination would not yield the claimed architecture. The stated rationale of improving temporal modeling, controllability, and quality does not identify evidence that Battenberg's style-token architecture would improve those characteristics within Zeghidour's codec-decoder framework or remain compatible with SoundStream's end-to-end codec
architecture.
Examiners Response
Applicant’ argument is not persuasive because Zeghidour itself recognizes that applicability of neural audio generation techniques developed for TTS to neural audio codecs, discussing TTS generative models used as neural codec decoders. Moreover, Zeghidour teaches conditioning its codec to modify audio reconstruction. Thus incorporating known recurrent/conditioning techniques into the neural audio generation pathway would have been a predictable application of known neural audio processing techniques, rather than requiring incorporation of Battenburg’s entire TTS architecture.
Applicant’s Arguments
Applicant notes that Binkowski does not cure these deficiencies from Zeghidour and Battenberg. Starting at page 19, the Office Action relies on Binkowski for multichannel and second-processing-block limitations, including generator blocks, channel changes, up sampling, and final output generation. (See Office Action at 19-27.) Binkowski discloses a feedforward generative neural network for text-to-speech generation that receives conditioning text input or linguistic features and generates an audio output in a single forward pass. (See Binkowski 0003, 0008, 0022-0024, 0047-0050.) Binkowski's generator is conditioned on text inputs, linguistic features, noise, or class identification, not on target data generated by a preconditioning learnable layer receiving "the bitstream, or a processed version thereof," and not on "weight parameters and bias parameters" that are "the conditioning feature parameters" generated by processing that target data
Examiners Response
Binkowski is relied upon for its express disclosure of multichannel intermediate representations, sequential generator blocks receiving preceding block outputs, and processing the final multichannel block outputs, and processing the final multichannel block output to generate single channel audio. Binkowski reduces intermediate channels from 768 to 96 and processes the resulting block output through an output convolutional layer to generate the audio waveform. Binkowski is not relied upon for the bitstream, recurrent layer, or conditioning limitations. Those limitations are addressed by the combination of Zeghidour and Battenberg.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims [1, 2, 4, 5, 6, 7, 8, 21, 22, 25] are rejected under 35 U.S.C. 103 as being unpatentable over Zeghidour et al [“Soundstream: An end-to-end neural audio codec,” ARXIV.ORG, Cornell university library, 201 olin library Cornell university Ithaca, NY 14853, XP091009160, 2021, 12 pages] in view of Battenberg et al [US 11646010 B2] and in further view of Binkowski et al [US 2021/0089909 Al].
Regarding claim 1, Zeghidour et al teaches an audio generator [Abstract "Training leverages
recent advances in text-to-speech and speech enhancement, which combine adversarial and reconstruction losses to allow the generation of high-quality audio content from quantized Embeddings"], configured to generate an audio signal from a bitstream [Page 1, column 2, lines 17- 29 "In this paper we propose Sound Stream, a novel audio codec that can compress speech, music and general audio more efficiently than previous codecs, as illustrated in Figure 1. Sound Stream leverages state-of-the-art solutions in the field of neural audio synthesis, and introduces a new learnable quantization module, to deliver audio at high perceptual quality, while operating at low-to-medium bitrates. Figure 2 illustrates the high level model architecture of the codec. A fully convolutional encoder receives as input a time-domain waveform and produces a sequence of embeddings at a lower sampling rate, which are quantized by a residual vector quantizer. A fully convolutional decoder receives the quantized embeddings and reconstructs an approximation of the original waveform"- where quantized embeddings are the compressed bit stream.], the bitstream representing the audio signal, the audio signal being subdivided in a sequence of frames, the audio generator comprising:
a first data provisioner configured to provide, for a given frame, first data derived from an input signal;
[Page 3, column 2, lines 14- 22 "We consider a single channel recording x ∈ RT, sampled at fs. The Sound Stream model consists of a sequence of three building blocks, as illustrated in Figure 2: • an encoder, which maps x to a sequence of embeddings (see Section III-A), • a residual vector quantizer, which replaces each embedding by the sum of vectors from a set of finite codebooks, thus compressing the representation with a target number of bits (see Section III-C) a decoder, which produces a lossy reconstruction ˆx ∈ RT from quantized embeddings (see Section III-B)"] [Fig. 2: Sound Stream model architecture. A convolutional encoder produces a latent representation of the input audio samples, which is quantized using a variable number nq of residual vector quantizers (RVQ). During training, the model parameters are optimized using a combination of reconstruction and adversarial losses. An optional conditioning input can be used to indicate whether background noise has to be removed from the audio. When deploying the model, the encoder and quantizer on a transmitter client send the compressed bitstream to a receiver client that can then decode the audio signal]; [Page 3 and 4, column 1 and 2, lines 17-19, lines 1-17 "Residual Vector Quantizer: "The goal of the quantizer is to compress the output of the encoder enc(x) to a target bitrate R, expressed in bits/second (bps). In order to train Sound Stream in an end to-end fashion, the quantizer needs to be jointly trained with the encoder and the decoder by backpropagation. The vector quantizer (VQ) proposed in [31], [32] in the context of VQVAEs meets this requirement. This vector quantizer learns a codebook of N vectors to encode each D-dimensional frame of enc(x). The encoded audio enc(x) ∈ RS×D is then mapped to a sequence of one-hot vectors of shape S × N, which can be represented using S log2 N bits” – where the bitstream represents the compressed audio signal and this signal is subdivided into a sequence of frames and the residual vector quantizer is fulfilling the role of a data provisioner].
PNG
media_image1.png
300
621
media_image1.png
Greyscale
“ where an;c
is the feature being transformed,
yn;c
is the multiplicative weight/scale, and bn;c is the additive bias. ];
However, Zeghidour et al does not teach wherein the first data comprise multiple channels; and, a first processing block, configured, for the given frame, to receive the first data and to output first output data in the given frame, wherein the first output data comprise a plurality of channels,
wherein the first processing block comprises: at least one preconditioning learnable layer comprising at least one recurrent learnable layer configured to receive the bitstream, or a processed version thereof, and, for the given frame, output target data representing the audio signal in the given frame, the target data being with multiple channels and multiple samples for the given frame;
at least one conditioning learnable layer configured, for the given frame, to process the target data to obtain conditioning feature parameters for the given frame;
and a styling element, configured to apply the conditioning feature parameters to the first data or normalized first data
But Battenberg et al teaches a first processing block - [Column 1, lines 51-55 “One aspect of the disclosure provides a method for estimating embedding capacity that includes receiving, at a deterministic reference encoder executing on data processing hardware, a reference audio signal and determining, by the data processing hardware, a reference embedding corresponding to the reference audio signal”] , configured, for the given frame, to receive the first data and to output first output data in the given frame,
[Column 5, lines 6-17 “The encoder pre-net neural network 114 is configured to receive a respective embedding of each character in the sequence and process the respective embedding of each character to generate a transformed embedding of the character. For example, the encoder pre-net neural network 114 can apply a set of non-linear transformations to each embedding to generate a transformed embedding. In some cases, the encoder pre-net neural network 114 includes a bottleneck neural network layer with dropout to increase convergence speed and improve generalization capability of the system during training”]; [Column 8, lines 31-37 “ More specifically, each character in the sequence of characters can be represented as a one-hot vector and embedded into a continuous vector. The encoder pre-net neural network 114 receives a respective embedding of each character in the sequence and processes the respective embedding of each character in the sequence to generate a transformed embedding of the character. For example, the encoder pre-net neural network 114 can apply a set of non-linear transformations to each embedding to generate a transformed embedding of the character” where the encoder is doing the preconditioning in the prenet neural network of the embedding (bitstream) and then outputs target data(transformed embedding)];
[Column 4, lines 61-65 “In particular, the seq2seq network 106 processes the sequence of characters using (i) an encoder neural network 112, which includes an encoder pre-net neural network 114
and an encoder CBHG neural network 116, and (ii) an attention-based decoder recurrent neural network 118.” where recurrent learnable layer is mapped to recurrent neural network 118];
at least one conditioning learnable layer configured, for the given frame, to process the target data to obtain conditioning feature parameters for the given frame; [Column 13, lines 52- 61 “During inference, the text encoder of the TTS model 450 may be directly conditioned on a specific token/ style embedding 614 (e.g., Token B) to allow for style control and manipulation without a reference audio signal 412.On the other hand, when a reference audio signal 412 for a target speaker is used whose transcript does not match input text 104 to be synthesized into synthesized speech 480, the style token layer 610 is conditioned upon the reference audio signal 412 represented by the prosodic embedding PE 550 output from the reference encoder 500” where conditioning is done on the target data];
and a styling element, configured to apply the conditioning feature parameters to the first data or normalized first data by
[ Column 13, lines 61-67 “The prosodic embedding PE 550, style embeddings 650, and tokens 615 affect information capacity of the respective embeddings and allow these heuristic-based models 500, 600 to target a specific trade-off between transfer precision (how closely the output resembles the references) and generality (how well an embedding works with arbitrary text”]; [Column 11, lines 40-45 The input text 104 and the reference text of the conditional dependences 416 may include character sequences while the reference and output audio signals 412, 470 correspond to acoustic features that may include mel-frequency spectrograms. The reference encoder 410 includes a neural network” where styling element is used to apply conditioning features to data];
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al with the teaching of Battenberg because incorporating the preconditioning, conditioning and styling architecture into the decoder of the Sound Stream would improve the temporal modeling, controllability and quality of audio generated from a compressed bitstream. This would be a predictable use of known neural network techniques.
Zeghidour et al in view of Battenberg do not teach
wherein the first data comprise multiple channels; and
wherein the first output data comprise a plurality of channels
the target data being with multiple channels and multiple samples for the given frame
wherein the audio generator further comprises a second processing block configured, for the given frame, to receive, as second data, the first output data or data derived from the first output data, and wherein the second processing block is configured to combine the plurality of channels of the second data to obtain the audio signal.
However, Binkowski teaches
wherein the first data comprise multiple channels; and
[0047 “The input to the generator neural network is a vector of linguistic features for each input time step of 400 input time steps that correspond to two seconds of audio (a frequency of200 Hz). Each vector of linguistic features corresponding to a respective input time step includes 567 channels”];
[0048 “The input convolutional neural network layer increases the number of channels for each time step from 567 to 768.” where first data is mapped to feature vector; multiple channels is mapped to 567 channels].
wherein the first output data comprise a plurality of channels,
[0046 “Each subsequent generator block in the sequence of generator blocks receives,
as the block input 202, the block output generated by the previous generator block in the sequence of generator blocks.” where first processing block is the generator block, receive the first data maps to receive block input 202, output first output data maps to block output];
[0049 “The sequence of generator blocks also decreases the number of channels for each time step from 768 to 96.” 96 implies multiple channels];
the target data being with multiple channels and multiple samples for the given frame
[0049 “The sequence of generator blocks also decreases the number of channels for each time step from 768 to 96.” 96 is multiple channels];
[0049 “The next three generator blocks each upsample their respective block inputs by 2x, so that the block output of the fifth generator block has a time dimensionality of 3200 (at a frequency of 1600
Hz). The sixth generator block upsamples its block input by 3x, generating a block output that has a time dimensionality of 9600 (at a frequency of 4800 Hz). The final generator block upsamples its block input by 5x, generating a block output that has a time dimensionality of 48000 (at a frequency
of 24 kHz).”where intermediate block has both multiple channels and multiple samples (48,000 time positions].
wherein the audio generator further comprises a second processing block configured, for the given frame, to receive, as second data, the first output data or data derived from the first output data,
[0046 “Each subsequent generator block in the sequence of generator blocks receives,
as the block input 202, the block output generated by the previous generator block in the sequence of generator blocks.” Where second processing block is the subsequent generator block, second data is block input 202, first output data is block output generated by previous generator block]
and wherein the second processing block is configured to combine the plurality of channels of the second data to obtain the audio signal.
[0049 “The sequence of generator blocks also decreases the number of channels for each time step from 768 to 96.”];
[0050 “The generator neural network includes an output convolutional neural network layer that processes the block output of the final generator block in the sequence of generator blocks and generates the audio output of the generator neural network. The audio output includes 48000
output time steps, and each output time step has a single channel representing the amplitude of the audio wave at the output time step.”];
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg with Binkowski to provide temporal processing of audio representations while progressively processing multichannel intermediate features to generate the reconstructed audio waveform.
Regarding claim 2, Zeghidour does not teach the audio generator of claim 1, configured to
obtain the audio signal from the first output data or a processed version of the first output data.
But Battenberg et al teaches configured to obtain the audio signal from the first output data or a
processed version of the first output data – [Column 13, lines 6-10 “In some examples, the attention module 454 is configured to convert the encoded sequence 453 to a fixed-length context vector 455 for each output step of the decoder 456 to produce the output audio signal 470”]
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al with the teaching of Battenberg because using quantized output representations of the Sound Stream as input to the decoder would thereby generate a an audio signal from the output data of preceding layers to improve compression efficiency and reconstruction quality.
Regarding claim 4, Zeghidour et al does not teach the audio generator of claim 1, wherein the recurrent learnable layer comprises at least one gated recurrent unit, GRU.
But Battenberg et al teaches wherein the recurrent learnable layer comprises at least one gated recurrent unit, GRU – [Column 12, lines 7-14 “A recurrent neural network 510 with a single 128-width Gated Recurrent Unit (GRU-RNN) layer receives the output 506 from the last convolutional layer and outputs a 128-dimensional output 512 applied to a fully connected layer 520 followed by an activation function 530 that outputs the predicted prosody embedding P.sub.E 550. The recurrent neural network 510 may include other types of bidirectional recurrent neural networks”].
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al with the teaching of Battenberg because including gated recurrent unit(GRU) layers into the audio generation system of Zeghidour would improve the modeling of temporal relationships across frames, thereby enhancing the quality and continuity of generated audio. This is predictable use of known sequence modeling techniques in speech processing.
Regarding claim 5, Zeghidour et al does not teach the audio generator of claim 1, wherein the recurrent learnable layer comprises at least one long short term memory, LSTM, recurrent learnable layer.
But Battenberg et al teaches the audio generator of claim 1, wherein the recurrent learnable layer comprises at least one long short term memory, LSTM, recurrent learnable layer – [ Column 10, lines 28- 41 “The input text 104 may include phoneme inputs produced by a text normalization front-end and lexicon since prosody is being addressed, rather than the model's ability to learn pronunciation from graphemes. The decoder 456 may include the decoder recurrent neural network 118 (FIG. 1) and use a reduction factor equal to two (2), thereby producing two spectrogram frames (e.g., output audio signal 470) per timestep. In some examples, two layers of 256-cell long short term memory (LSTM) using zone out with probability equal to 0.1 may replace GRU cells of the decoder 456. In other implementations, the TTS model 450 includes the speech synthesis system disclosed in U.S. application Ser. No. 16/058,640, filed on Aug. 8, 2018, the contents of which are incorporated by reference in their entirety”].
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al with the teaching of Battenberg because incorporating LSTM based recurrent learnable layers into the Sound Stream would improve continuity and long term temporal coherence of generated audio. This is predictable use of known sequence modeling techniques in speech processing.
Regarding claim 6, Zeghidour et al does not teach the audio generator of claim 1, wherein the recurrent learnable layer is configured to generate the output, which is the target data or a predecessor thereof, for a given time instant by keeping into account the output and/or a state of a preceding time instant, wherein the relevance of the output and/or state of a preceding time instant is obtained by training.
But Battenberg et al teaches the audio generator of claim 1, wherein the recurrent learnable layer is configured to generate the output, which is the target data or a predecessor thereof, for a given time instant by keeping into account the output and/or a state of a preceding time instant, wherein the relevance of the output and/or state of a preceding time instant is obtained by training - [Column 1, lines 25-34 “Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. For instance, neural networks may convert input text to output speech. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters”] ; [Column 1, lines 35-48 “Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the internal state of the network from a previous time step in computing an output at a current time step. An example of a recurrent neural network is a long short term (LSTM) neural network that includes one or more LSTM memory blocks. Each LSTM memory block can include one or more cells that each include an input gate, a forget gate, and an output gate that allows the cell to store previous states for the cell, e.g., for use in generating a current activation or to be provided to other components of the LSTM neural network”].
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al with the teaching of Battenberg because incorporating such recurrent learnable layers that utilize prior state information across time steps into the Sound Stream system would improve continuity, temporal coherence and accuracy of generated audio from frame based representations. Leveraging prior time step state is a well-known and predictable technique for enhancing sequence modeling in speech processing.
Regarding claim 7, Zeghidour et al does not teach the audio generator of claim 1, wherein the recurrent learnable layer operates along a series of time steps each comprising at least one state, in such a way that each time step is conditioned by the output and/or state of the preceding time step.
But Battenberg et al teaches the audio generator of claim 1, wherein the recurrent learnable layer operates along a series of time steps each comprising at least one state, in such a way that each time step is conditioned by the output and/or state of the preceding time step - [Column 1, lines 35-41 “Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the internal state of the network from a previous time step in computing an output at a current time step”].
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al with the teaching of Battenberg because incorporating such recurrent learnable layers that operate over a sequence of time steps, where each output is computed using internal state information from a previous time step into the Sound Stream system would improve temporal dependencies across sequential frames. This would ensure that generation of each frame accounts for prior frames, thereby improving temporal continuity, coherence and naturalness of the generated audio.
Regarding claim 8, Zeghidour et al in view of Battenberg et al do not teach the audio generator
of claim 7, further comprising a plurality of feedforward modules, each providing the state and/or output to the immediately subsequent module.
But Binkowski teaches teach the audio generator of claim 7, further comprising a plurality of
feedforward modules, each providing the state and/or output to the immediately subsequent module- [0044 “ FIG. 2 is a diagram of an example generator block 200. The generator block 200 can be a component of a generator neural network, e.g., the generator neural network 110 depicted in FIG. 1, that is configured to process a conditioning text input and to generate an audio output. The
generator neural network can include a sequence of generator blocks. In some implementations, each generator block in the sequence of generator blocks has the same architecture. In some other implementations, one or more generator blocks have a different architecture than the other generator blocks in the sequence of generator blocks “]; [0022 “The training system 100 includes a generative neural network 110, a discriminator neural network system 120, and a parameter updating system 130. The training system 100 is configured to train the generative neural network 110 to receive a conditioning text input 102 and to process the conditioning text input 102 to generate an audio output 112. In some implementations, the generative neural network 110 is a feedforward neural network, i.e., the generative neural network 110 generates the audio output 112 in a single forward pass”].
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg with the teaching of Binkowski because incorporating feedforward modules in the form of generator blocks would improve scalability and quality of audio generation. This is a predictable use of known neural network design patterns.
Regarding claim 21, Zeghidour et al teaches the audio generator according to claim 1, comprising a quantization index converter configured to convert indexes of the bitstream onto codes - [Abstract "Training leverages recent advances in text-to-speech and speech enhancement, which combine adversarial and reconstruction losses to allow the generation of high-quality audio content from quantized Embeddings"];[Page 1, column 2, lines 17- 29 "In this paper we propose Sound Stream, a novel audio codec that can compress speech, music and general audio more efficiently than previous codecs, as illustrated in Figure 1. Sound Stream leverages state-of-the-art solutions in the field of neural audio synthesis, and introduces a new learnable quantization module, to deliver audio at high perceptual quality, while operating at low-to-medium bitrates. Figure 2 illustrates
the high level model architecture of the codec. A fully convolutional encoder receives as input a time-domain waveform and produces a sequence of embeddings at a lower sampling rate, which are quantized by a residual vector quantizer. A fully convolutional decoder receives the quantized embeddings and reconstructs an approximation of the original waveform"]; [Page 3, column 2, lines 14- 22 "We consider a single channel recording x ∈ RT, sampled at fs. The Sound Stream model consists of a sequence of three building blocks, as illustrated in Figure 2: • an encoder, which maps x to a sequence of embeddings (see Section III-A), • a residual vector quantizer, which replaces each embedding by the sum of vectors from a set of finite codebooks, thus compressing the representation with a target number
of bits (see Section III-C) a decoder, which produces a lossy reconstruction ˆx ∈ RT
from quantized embeddings (see Section III-B)"]; Page 3 and 4, column 1 and 2, lines 17-19, lines 1-17 "Residual Vector Quantizer: "The goal of the quantizer is to compress the output of the encoder enc(x) to a target bitrate R, expressed in bits/second (bps). In order to train Sound Stream in an end
to-end fashion, the quantizer needs to be jointly trained with the encoder and the decoder by backpropagation. The vector quantizer (VQ) proposed in [31], [32] in the context of VQ
VAEs meets this requirement. This vector quantizer learns a codebook of N vectors to encode each D-dimensional frame of enc(x). The encoded audio enc(x) ∈ RS×D is then mapped
to a sequence of one-hot vectors of shape S × N, which can be represented using S log2 N bits”].
Regarding claim 22, Zeghidour et al teaches an audio generator, configured to generate an audio
signal from a bitstream [Abstract "Training leverages recent advances in text-to-speech and speech enhancement, which combine adversarial and reconstruction losses to allow the generation of high-quality audio content from quantized Embeddings"], the bitstream representing the audio signal, the bitstream being subdivided into a sequence of indexes, the audio signal being subdivided in a sequence of frames, the audio generator comprising: a quantization index converter configured to convert the indexes of the bitstream onto codes, – [Page 1, column 2, lines 17- 29 "In this paper we propose Sound Stream, a novel audio codec that can compress speech, music and general audio more efficiently than previous codecs, as illustrated in Figure 1. Sound Stream leverages state-of-the-art solutions in the field of neural audio synthesis, and introduces a new learnable quantization module, to deliver audio at high perceptual quality, while operating at low-to-medium bitrates. Figure 2 illustrates the high level model architecture of the codec. A fully convolutional encoder receives as input a time-domain waveform and produces a sequence of embeddings at a lower sampling rate, which are quantized by a residual vector quantizer. A fully convolutional decoder receives the quantized embeddings and reconstructs an approximation of the original waveform"]; [Page 3, column 2, lines 14- 22 "We consider a single channel recording x ∈ RT, sampled at fs. In order to train Sound Stream in an end to-end fashion, the quantizer needs to be jointly trained with the encoder and the decoder by backpropagation. The vector quantizer (VQ) proposed in [31], [32] in the context of VQ VAEs meets this requirement. This vector quantizer learns a codebook of N vectors to encode each D-dimensional frame of enc(x). The encoded audio enc(x) ∈ RS×D is then mapped to a sequence of one-hot vectors of shape S × N, which can be represented using S log2 N bits.
a first data provisioner configured to provide, for a given frame, first data derived from an input signal from an external or internal source or from the bitstream;
[Page 3, column 2, lines 14- 22 "We consider a single channel recording x ∈ RT, sampled at fs. The Sound Stream model consists of a sequence of three building blocks, as illustrated in Figure 2: • an encoder, which maps x to a sequence of embeddings (see Section III-A), • a residual vector quantizer, which replaces each embedding by the sum of vectors from a set of finite codebooks, thus compressing the representation with a target number of bits (see Section III-C) a decoder, which produces a lossy reconstruction ˆx ∈ RT from quantized embeddings (see Section III-B)"] [Fig. 2: Sound Stream model architecture. A convolutional encoder produces a latent representation of the input audio samples, which is quantized using a variable number nq of residual vector quantizers (RVQ). During training, the model parameters are optimized using a combination of reconstruction and adversarial losses. An optional conditioning input can be used to indicate whether background noise has to be removed from the audio. When deploying the model, the encoder and quantizer on a transmitter client send the compressed bitstream to a receiver client that can then decode the audio signal]; [Page 3 and 4, column 1 and 2, lines 17-19, lines 1-17 "Residual Vector Quantizer: "The goal of the quantizer is to compress the output of the encoder enc(x) to a target bitrate R, expressed in bits/second (bps). In order to train Sound Stream in an end to-end fashion, the quantizer needs to be jointly trained with the encoder and the decoder by backpropagation. The vector quantizer (VQ) proposed in [31], [32] in the context of VQVAEs meets this requirement. This vector quantizer learns a codebook of N vectors to encode each D-dimensional frame of enc(x). The encoded audio enc(x) ∈ RS×D is then mapped to a sequence of one-hot vectors of shape S × N, which can be represented using S log2 N bits” – where the bitstream represents the compressed audio signal and this signal is subdivided into a sequence of frames and the residual vector quantizer is fulfilling the role of a data provisioner]
[Page 6, F. Joint compression and enhancement “
PNG
media_image1.png
300
621
media_image1.png
Greyscale
“ where an;c
is the feature being transformed,
yn;c
is the multiplicative weight/scale, and bn;c is the additive bias. ];
However, Zeghidour et al does not teach a first processing block, configured, for the given frame, to receive the first data and to output first output data in the given frame, the first output data comprising a plurality of channels
wherein the first processing block comprises: at least one preconditioning learnable layer configured to receive the bitstream, or a processed version thereof, and, for the given frame, output target data representing the audio signal in the given frame, the target data being with multiple channels;
at least one conditioning learnable layer configured, for the given frame, to process the target data to obtain conditioning feature parameters for the given frame;
and a styling element, configured to apply the conditioning feature parameters to the first data or normalized first data by elementwise linearly transforming the first data or normalized first data through weight parameters and bias parameters, the weight parameters and bias parameters being the conditioning feature parameters; and a second processing block configured to receive, for the given frame, as second data, the first output data or data derived from the first output data, and wherein the second processing block is further configured to combine the plurality of channels of the first output data or of the second output data to obtain the audio signal.
But Battenberg et al teaches a first processing block, configured, for the given frame, to receive the first data and to output first output data in the given frame, data processing hardware, a reference audio signal and determining, by the data processing hardware, a reference embedding corresponding to the reference audio signal”],
wherein the first processing block comprises: at least one preconditioning learnable layer configured to receive the bitstream, or a processed version thereof, and, for the given frame, output target data representing the audio signal in the given frame,
[Column 5, lines 6-17 “The encoder pre-net neural network 114 is configured to receive a respective embedding of each character in the sequence and process the respective embedding of each character to generate a transformed embedding of the character” where the encoder is doing the preconditioning in the prenet neural network of the embedding (bitstream) and then outputs target data(transformed embedding)];
at least one conditioning learnable layer configured, for the given frame, to process the target data to obtain conditioning feature parameters for the given frame;
[Column 13, lines 52- 61 “During inference, the text encoder of the TTS model 450 may be directly conditioned on a specific token/ style embedding 614 (e.g., Token B) to allow for style control and manipulation without a reference audio signal 412.On the other hand, when a reference audio signal 412 for a target speaker is used whose transcript does not match input text 104 to be synthesized into synthesized speech 480, the style token layer 610 is conditioned upon the reference audio signal 412 represented by the prosodic embedding PE 550 output from the reference encoder 500” where conditioning is done on the target data];
and a styling element, configured to apply the conditioning feature parameters to the first data or normalized first data,
[Column 13, lines 52- 61 “During inference, the text encoder of the TTS model 450 may be directly conditioned on a specific token/ style embedding 614 (e.g., Token B) to allow for style control and manipulation without a reference audio signal 412.On the other hand, when a reference audio signal 412 for a target speaker is used whose transcript does not match input text 104 to be synthesized into synthesized speech 480, the style token layer 610 is conditioned upon the reference audio signal 412 represented by the prosodic embedding PE 550 output from the reference encoder 500”]; [Column 13, lines 61-67 “The prosodic embedding PE 550, style embeddings 650, and tokens 615 affect information capacity of the respective embeddings and allow these heuristic-based models 500, 600 to target a specific trade-off between transfer precision (how closely the output resembles the references) and generality (how well an embedding works with arbitrary text” where conditioning is done on the target data]
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al with the teaching of Battenberg because incorporating the preconditioning, conditioning and styling architecture into the decoder of Sound Stream would improve the temporal modeling, controllability and quality of audio generated from a compressed bitstream. This would be a predictable use of known neural network techniques.
Zeghidour et al in view of Battenberg do not teach
the first data comprising a plurality of channels; and
the first output data comprising a plurality of channels,
the target data being with multiple channels
and a second processing block configured, for the given frame, to receive, as second data, the first output data or data derived from the first output data, and wherein the second processing block is configured to combine the plurality of channels of the second data to obtain the audio signal.
However, Binkowski teaches
the first data comprising a plurality of channels; and
[0047 “The input to the generator neural network is a vector of linguistic features for each input time step of 400 input time steps that correspond to two seconds of audio (a frequency of200 Hz). Each vector of linguistic features corresponding to a respective input time step includes 567 channels”];
[0048 “The input convolutional neural network layer increases the number of channels for each time step from 567 to 768.” where first data is mapped to feature vector; multiple channels is mapped to 567 channels].
the first output data comprising a plurality of channels,
[0046 “Each subsequent generator block in the sequence of generator blocks receives,
as the block input 202, the block output generated by the previous generator block in the sequence of generator blocks.” where first processing block is the generator block, receive the first data maps to receive block input 202, output first output data maps to block output];
[0049 “The sequence of generator blocks also decreases the number of channels for each time step from 768 to 96.” 96 implies multiple channels];
the target data being with multiple channels
[0049 “The sequence of generator blocks also decreases the number of channels for each time step from 768 to 96.” 96 is multiple channels];
[0049 “The next three generator blocks each upsample their respective block inputs by 2x, so that the block output of the fifth generator block has a time dimensionality of 3200 (at a frequency of 1600
Hz). The sixth generator block upsamples its block input by 3x, generating a block output that has a time dimensionality of 9600 (at a frequency of 4800 Hz). The final generator block upsamples its block input by 5x, generating a block output that has a time dimensionality of 48000 (at a frequency
of 24 kHz).”where intermediate block has both multiple channels and multiple samples (48,000 time positions].
and a second processing block configured, for the given frame, to receive, as second data, the first output data or data derived from the first output data,
[0046 “Each subsequent generator block in the sequence of generator blocks receives,
as the block input 202, the block output generated by the previous generator block in the sequence of generator blocks.” Where second processing block is the subsequent generator block, second data is block input 202, first output data is block output generated by previous generator block]
and wherein the second processing block is configured to combine the plurality of channels of the second data to obtain the audio signal.
[0049 “The sequence of generator blocks also decreases the number of channels for each time step from 768 to 96.”];
[0050 “The generator neural network includes an output convolutional neural network layer that processes the block output of the final generator block in the sequence of generator blocks and generates the audio output of the generator neural network. The audio output includes 48000
output time steps, and each output time step has a single channel representing the amplitude of the audio wave at the output time step.”];
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg with Binkowski to provide temporal processing of audio representations while progressively processing multichannel intermediate features to generate the reconstructed audio waveform.
Regarding claim 25, a method for generating an audio signal from a bitstream, the bitstream
representing the audio signal, the audio signal being subdivided in a sequence of frames, the method comprising: providing, for a given frame, first data derived from an input signal, the first data comprising multiple channels.
through a first processing block, receiving the first data and outputting first output data in the given frame, the first output data comprising a plurality of channels,
wherein the first processing block comprises: at least one preconditioning learnable layer receiving the bitstream, or a processed version thereof, and, for the given frame, output target data representing the audio signal in the given frame;
at least one conditioning learnable layer processing, e.g. for the given frame, the target data to obtain conditioning feature parameters for the given frame;
a styling element, applying the conditioning feature parameters to the first data or normalized first data by elementwise linearly transforming the first data or normalized first data through weight parameters and bias parameters, the weight parameters and bias parameters being the conditioning feature parameters;
wherein the at least one preconditioning learnable layer comprises at least one recurrent learnable layers wherein the method further comprises:
receiving through a second processing block, as second data, the first output data or data derived from the first output data, wherein the second processing block combines the plurality of channels of the second data to obtain the audio signal, and
obtaining the audio signal from the first output data or a processed version of the first output data.
Claim 25 is rejected for the same reasons as claim 1.
Claims [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 ] are rejected under 35 U.S.C. 103 as being unpatentable over Zeghidour et al [“Soundstream: An end-to-end neural audio codec,” ARXIV.ORG, Cornell university library, 201 olin library Cornell university Ithaca, NY 14853, XP091009160, 2021, 12 pages] in view of Battenberg et al [US 2022/0101826 Al] and in further view of Binkowski et al [US 2021/0089909 Al]and in further view of Chung et al [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, “ARXIV.ORG].
Regarding Claim 9, Zeghidour et al in view of Battenberg et al do in view of Binkowski et al do
not teach the audio generator of claim 1, wherein the recurrent learnable layer is configured to generate a state and/or output ht for a particular t-th state or module by: weighting a candidate state and/or output through an update gate vector zt , to generate a first weighted addend; and weighting the state and/or output ht-1 of the preceding time step through a vector which is complementary to 1 with the update gate vector zt, to generate a second weighted addend; and adding the first addend with the second addend.
However, Chung et al teaches the audio generator of claim 1, wherein the recurrent learnable
layer is configured to generate a state and/or output ht for a particular t-th state or module by:
weighting a candidate state and/or output through an update gate vector zt , to generate a first weighted addend; and weighting the state and/or output ht-1 of the preceding time step through a vector which is complementary to 1 with the update gate vector zt, to generate a second weighted addend; and adding the first addend with the second addend – [Page 4, , Section 3.2 Gated Recurrent Unit:
PNG
media_image2.png
489
1350
media_image2.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg in view of Binkowski et al with the teaching of Chung because applying this well-known technique of Chung to the GRU layers used in Battenberg would improve stability, temporal continuity, and context retention in generated audio.
Regarding Claim 10, Zeghidour et al in view of Battenberg et al in view of Binkowski et al do not teach the audio generator of claim 7, wherein the recurrent learnable layer is configured to generate a state and/or output ht by: through reciprocally complementary weighting vectors, adding a weighted version of a candidate state and/or output with a weighted version of the state and/or output ht -1 of the preceding time step.
However, Chung teaches the audio generator of claim 7, wherein the recurrent learnable layer is configured to generate a state and/or output ht by: through reciprocally complementary weighting vectors, adding a weighted version of a candidate state and/or output with a weighted version of the state and/or output ht -1 of the preceding time step - [Page 4, , Section 3.2 Gated Recurrent Unit:
PNG
media_image3.png
953
1099
media_image3.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg in view of Binkowski et al with the teaching of Chung because Chung discloses a GRU update equation in which a gate vector is applied to one component and value is applied to another component, such that the two values are complementary and weighted components are combined to generate an output. This would improve stability, temporal continuity and context retention in generated audio.
Regarding Claim 11, Zeghidour et al in view of Battenberg et al in view of Binkowski et al do not
teach the audio generator of claim 9, wherein the recurrent learnable layer is configured to generate the candidate state and/or output by at least applying a weight parameter W, obtained by training, to: an element-wise product between a reset gate vector rt and the state and/or output ht-1 of the preceding time step, concatenated with the input xt for the current time instant.
But Chung teaches recurrent learnable layer is configured to generate the candidate state and/or output by at least applying a weight parameter W, obtained by training, to: an element-wise product between a reset gate vector rt and the state and/or output ht-1 of the preceding time step, concatenated with the input xt for the current time instant – [Page 4, Section 3.2 Gated Recurrent Unit “The candidate activation
PNG
media_image4.png
32
28
media_image4.png
Greyscale
is computed similarly to that of the traditional recurrent unit (see Eq. (2)) and as in [Bahdanau et al., 2014], ˜
PNG
media_image5.png
58
382
media_image5.png
Greyscale
, where rt is a set of reset gates and is an element-wise multiplication. When off
PNG
media_image6.png
49
190
media_image6.png
Greyscale
, the reset gate effectively makes the unit act as if it is reading the first symbol of an input sequence, allowing it to forget the previously computed state. The reset gate
PNG
media_image7.png
52
41
media_image7.png
Greyscale
is computed similarly to the update gate:
PNG
media_image8.png
67
403
media_image8.png
Greyscale
. See Fig. 1 (b) for the graphical illustration of the GRU”].
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg in view of Binkowski et al with the teaching of Chung because applying this GRU reset-gate mechanism with the audio generator of Zeghidour would enable selective retention and thereby improve stability, adaptability, and temporal accuracy of generated audio.
Regarding claim 12, Zeghidour et al in view of Battenberg et al in view of Binkowski et al do not
teach the audio generator of claim 11, the audio generator of claim 11, further configured to apply an activation function after having applied the weight parameter W.
However, Chung teaches further configured to apply an activation function after having applied the weight parameter W – tanh is the nonlinear activation function shown below – Page 3, 3.1 Long Short -term Memory Unit
PNG
media_image9.png
282
1351
media_image9.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg in view of Binkowski et al with the teaching of Chung because applying such an activation function within the audio generator of Zeghidour will introduce non-linearity and improve representational capacity and modeling accuracy.
Regarding Claim 13, Zeghidour et al in view of Battenberg et al in view of Binkowski et al do not
teach the audio generator of claim 12, wherein the activation function is TanH.
But Chung teaches wherein the activation function is TanH – [Page 2, Background: Recurrent Neural Network “The other approach, in which we are more interested in this paper, is to design a more sophisticated activation function than a usual activation function, consisting of affine transformation followed by a simple element-wise nonlinearity by using gating units. The earliest attempt in this direction resulted in an activation function, or a recurrent unit, called a long short-term memory (LSTM) unit [Hochreiter and Schmidhuber, 1997”]; [Page 3, 3.1 Long Short-Term Memory Unit
PNG
media_image10.png
550
1355
media_image10.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg in view of Binkowski et al with the teaching of Chung because using a standard, well known tanh activation function would improve stability and representational capability.
Regarding claim 14, Zeghidour et al in view of Battenberg et al in view of Binkowski et al do not teach the audio generator of claim 10, wherein the recurrent learnable layer is configured to generate the candidate state and/or output by at least: weighting, through weight parameter W obtained by training, a vector which is conditioned by both: the input xt for the current time instant and the state and/or output ht-1 of the preceding time step weighted onto a reset gate vector rt.
However, Chung teaches the recurrent learnable layer is configured to generate the candidate state and/or output by at least: weighting, through weight parameter W obtained by training, a vector which is conditioned by both: the input xt for the current time instant and the state and/or output ht-1 of the preceding time step weighted onto a reset gate vector rt – [Page 4, 3.2 Gated Recurrent Unit
PNG
media_image11.png
1010
1349
media_image11.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg in view of Binkowski et al with the teaching of Chung because using this well-known GRU formulation within the audio generator of Zeghidour would enable selective retention and suppression of prior state information, thereby improving temporal accuracy and stability of generated audio.
Regarding claim 15, Zeghidour et al in view of Battenberg et al in view of Binkowski et al do not teach the audio generator of claim 9, wherein the recurrent learnable layer is configured to generate the update gate vector zt by applying a parameter Wz to a concatenation of: the input ht-1 of the recurrent module ht-1 concatenated with the input xt for the current time instant.
However, Chung teaches the recurrent learnable layer is configured to generate the update gate vector zt by applying a parameter Wz to a concatenation of: the input ht-1 of the recurrent module ht-1 concatenated with the input xt for the current time instant – [Page 4, 3.2 Gated Recurrent Unit
PNG
media_image12.png
656
1358
media_image12.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg in view of Binkowski et al with the teaching of Chung because applying a parameter to a combination of a prior state and current input to generate an update gate vector within a audio generator would enable controlled updating of state information, thereby improving temporal consistency, adaptability and stability of generated audio.
Regarding claim 16, Zeghidour et al in view of Battenberg et al in view of Binkowski et al do not teach the audio generator of claim 15, configured, after having applied the parameter Wz, to apply an activation function.
However, Chung does teach having applied the parameter Wz, to apply an activation function- [Page 4, 3.2 Gated Recurrent Unit
PNG
media_image13.png
704
1385
media_image13.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg in view of Binkowski et al with the teaching of Chung because applying an activation function within a GRU layers used in Zeghidour would enable bounded gating behavior and stable state transitions. This would improve temporal consistency and robustness of generated audio.
Regarding claim 17, Zeghidour et al in view of Battenberg et al in view of Binkowski et al do not teach the audio generator of claim 16, wherein the activation function is a sigmoid, σ.
However, Chung teaches the activation function is a sigmoid, σ – [Page 4, 3.2 Gated Recurrent Unit
PNG
media_image14.png
523
1389
media_image14.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg in view of Binkowski et al with the teaching of Chung because using a sigmoid activation function in implementing GRU layers with Zeghidour’s audio generator would enable bounded gating behavior and stable state updates. This is a known and predictable design choice in recurrent neural networks.
Regarding claim 18, , Zeghidour et al in view of Battenberg et al in view of Binkowski et al do not teach the audio generator of claim 9, wherein the reset gate vector rt is obtained by applying a weight parameter Wr to a concatenation of both: the state and/or output ht-1 of the preceding time step and the input xt for the current time instant.
However, Chung teaches the reset gate vector rt is obtained by applying a weight parameter Wr to a concatenation of both: the state and/or output ht-1 of the preceding time step and the input xt for the current time instant – [Page 4, 3.2 Gated Recurrent Unit
PNG
media_image15.png
528
1410
media_image15.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg in view of Binkowski et al with the teaching of Chung because applying the reset gate technique with the audio generator would selectively control how prior state information influences current processing. This improves noise robustness and temporal accuracy of generated audio.
Regarding claim 19, Zeghidour et al in view of Battenberg et al in view of Binkowski et al do not teach the audio generator of claim 18, configured, after having applied the parameter Wr, to apply an activation function.
However, Chung teaches having applied the parameter Wr, to apply an activation function. – [Page 4, 3.2 Gated Recurrent Unit – activation function sigmoid is used below in the equation
PNG
media_image16.png
189
1087
media_image16.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg in view of Binkowski et al with the teaching of Chung because applying such activation function with GRU layers within Zeghidour would enable effective gating behavior and stable modulation of prior state contributions.
Regarding claim 20, Zeghidour et al in view of Battenberg et al in view of Binkowski et al do not teach the audio generator of claim 19, wherein the activation function is a sigmoid, σ.
However, Chung teaches the activation function is a sigmoid, σ – [Page 4, 3.2 Gated Recurrent Unit – activation function sigmoid is used below in the equation
PNG
media_image16.png
189
1087
media_image16.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings Zeghidour et al in view of Battenberg in view of Binkowski et al with the teaching of Chung because using a sigmoid activation function in implementing GRU layers with Zeghidour’s audio generator would enable bounded gating behavior and stable state updates. This is a known and predictable design choice in recurrent neural networks.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHEZA ABDUL AZIZ whose telephone number is (571)272-9610. The examiner can normally be reached Monday-Friday 7:30am-5pm Alternate Fridays off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHEZA ABDUL AZIZ/Examiner, Art Unit 2657
/DANIEL C WASHBURN/Supervisory Patent Examiner, Art Unit 2657
Prosecution Insights