Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
2. The information disclosure statement (IDS) submitted on April 02, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
3. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or
any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
4. Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
The Independent claim 1 recites “An audio decoding method, performed by a computer device, comprising: obtaining encoding vectors of audio frames in an audio frame sequence; performing, in response to a current audio frame in the audio frame sequence being to be decoded, up-sampling on an encoding vector of a historical audio frame to obtain an up-sampling feature value describing the historical audio frame, the historical audio frame including one or more audio frames decoded before the current audio frame in the audio frame sequence; and performing, based on the up-sampling feature value, up-sampling on an encoding vector of the current audio frame to obtain decoded data of the current audio frame”.
The limitations “obtaining encoding vectors of audio frames in an audio frame sequence; performing, in response to a current audio frame in the audio frame sequence being to be decoded, up-sampling on an encoding vector of a historical audio frame to obtain an up-sampling feature value describing the historical audio frame, the historical audio frame including one or more audio frames decoded before the current audio frame in the audio frame sequence; and performing, based on the up-sampling feature value, up-sampling on an encoding vector of the current audio frame to obtain decoded data of the current audio frame” as drafted, covers a mental process, as this could be done by mentally or by hand with pen and paper.
This judicial exception is not integrated into a practical application. Claim 1 recites “An audio decoding method, performed by a computer device, comprising:…”. This limitation directs towards using a computer for the method, and does not impose any meaningful limits on practicing the abstract idea.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The addition of the generic computer components recited above with regard to claim 1 do not amount to more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Claim 1 does not recite any additional limitations. The claim as drafted, is not patent eligible.
The Independent claim 15 recites “An audio encoding method, performed by a computer device, comprising: obtaining audio data of audio frames in an audio frame sequence; performing, in response to a current audio frame in the audio frame sequence being to be encoded, down-sampling on audio data of a historical audio frame to obtain a down-sampling feature value describing the historical audio frame, the historical audio frame including one or more audio frames encoded before the current audio frame in the audio frame sequence; and performing, based on the down-sampling feature value, down-sampling on the audio data of the current audio frame to obtain an encoding vector of the current audio frame”.
The limitations “obtaining audio data of audio frames in an audio frame sequence; performing, in response to a current audio frame in the audio frame sequence being to be encoded, down-sampling on audio data of a historical audio frame to obtain a down-sampling feature value describing the historical audio frame, the historical audio frame including one or more audio frames encoded before the current audio frame in the audio frame sequence; and performing, based on the down-sampling feature value, down-sampling on the audio data of the current audio frame to obtain an encoding vector of the current audio frame” as drafted, covers a mental process, as this could be done by mentally or by hand with pen and paper.
This judicial exception is not integrated into a practical application. Claim 15 recites “An audio encoding method, performed by a computer device, comprising:…”. This limitation directs towards using a computer for the method, and does not impose any meaningful limits on practicing the abstract idea.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The addition of the generic computer components recited above with regard to claim 15 do not amount to more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Claim 15 does not recite any additional limitations. The claim as drafted, is not patent eligible.
The Independent claim 18 recites “An electronic device comprising: one or more processors; and one or more memories storing one or more computer programs that, when executed by the one or more processors, cause the electronic device to: obtain encoding vectors of audio frames in an audio frame sequence; perform, in response to a current audio frame in the audio frame sequence being to be decoded, up-sampling on an encoding vector of a historical audio frame to obtain an up-sampling feature value describing the historical audio frame, the historical audio frame including one or more audio frames decoded before the current audio frame in the audio frame sequence; and perform, based on the up-sampling feature value, up-sampling on an encoding vector of the current audio frame to obtain decoded data of the current audio frame”.
The limitations “obtain encoding vectors of audio frames in an audio frame sequence; perform, in response to a current audio frame in the audio frame sequence being to be decoded, up-sampling on an encoding vector of a historical audio frame to obtain an up-sampling feature value describing the historical audio frame, the historical audio frame including one or more audio frames decoded before the current audio frame in the audio frame sequence; and perform, based on the up-sampling feature value, up-sampling on an encoding vector of the current audio frame to obtain decoded data of the current audio frame” as drafted, covers a mental process, as this could be done by mentally or by hand with pen and paper.
This judicial exception is not integrated into a practical application. Claim 18 recites “An electronic device comprising: one or more processors; and one or more memories storing one or more computer programs that, when executed by the one or more processors, cause the electronic device to:…”. This limitation directs towards using a computer for the method, and does not impose any meaningful limits on practicing the abstract idea.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The addition of the generic computer components recited above with regard to claim 18 do not amount to more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Claim 18 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim Rejections - 35 USC § 102
5. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
6. Claims 1-12 and 14-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Zeghidour (U.S. Publication No. 20230019128).
Regarding claim 1, Zeghidour discloses an audio decoding method, performed by a computer device, comprising:
obtaining encoding vectors of audio frames in an audio frame sequence ([0031] - The audio waveform 112 is processed (e.g., encoded) by the encoder 102 to generate a sequence of feature vectors 208 representing the waveform 112. Feature vectors 208 (e.g., embeddings, latent representations) are compressed representations of waveforms that extract the most relevant information about their audio content);
performing, in response to a current audio frame in the audio frame sequence being to be decoded, up-sampling on an encoding vector of a historical audio frame to obtain an up-sampling feature value describing the historical audio frame, the historical audio frame including one or more audio frames decoded before the current audio frame in the audio frame sequence ([0041] - The QFVs 212 can then be processed (e.g., decoded) by the decoder 104 to generate an audio waveform 112. The decoder 104 generally mirrors the processes of the encoder 102 by outputting waveforms starting from (quantized) feature vectors. The decoder 104 can up-sample the QFVs 212 to generate the output waveform 206 at a higher sampling rate than the input QFVs 212 [0095] - The DecoderBlocks 810 include a transposed Conv1D layer 814 for up-sampling followed by three ResidualUnits 812. Internal convolutional layers of DecoderBlocks 810 and ResidualUnits 812 are shown in FIG. 8B. The decoder 104 uses the same strides as the encoder 102, but in reverse order, to reconstruct a waveform with the same resolution as the input waveform. The number of channels is halved whenever up-sampling. A final Conv1D layer 802 with one filter, a kernel of size 7 and stride 1 projects the feature vectors 208 back to the waveform 112. A FiLM conditioning layer 806 can also be implemented to process the conditioning signal for joint decompression and enhancement);
and performing, based on the up-sampling feature value, up-sampling on an encoding vector of the current audio frame to obtain decoded data of the current audio frame ([0041] - The QFVs 212 can then be processed (e.g., decoded) by the decoder 104 to generate an audio waveform 112. The decoder 104 generally mirrors the processes of the encoder 102 by outputting waveforms starting from (quantized) feature vectors. The decoder 104 can up-sample the QFVs 212 to generate the output waveform 206 at a higher sampling rate than the input QFVs 212 [0095] - The DecoderBlocks 810 include a transposed Conv1D layer 814 for up-sampling followed by three ResidualUnits 812. Internal convolutional layers of DecoderBlocks 810 and ResidualUnits 812 are shown in FIG. 8B. The decoder 104 uses the same strides as the encoder 102, but in reverse order, to reconstruct a waveform with the same resolution as the input waveform. The number of channels is halved whenever up-sampling. A final Conv1D layer 802 with one filter, a kernel of size 7 and stride 1 projects the feature vectors 208 back to the waveform 112. A FiLM conditioning layer 806 can also be implemented to process the conditioning signal for joint decompression and enhancement).
Regarding claim 2, Zeghidour discloses the audio decoding method, wherein:
performing up-sampling on the encoding vector of the historical audio frame to obtain the up-sampling feature value includes performing up-sampling on the encoding vector of the historical audio frame using a plurality of up-sampling layers of a decoder to obtain the up-sampling feature value including a plurality of feature vectors, respectively ([0041] - The QFVs 212 can then be processed (e.g., decoded) by the decoder 104 to generate an audio waveform 112. The decoder 104 generally mirrors the processes of the encoder 102 by outputting waveforms starting from (quantized) feature vectors. The decoder 104 can up-sample the QFVs 212 to generate the output waveform 206 at a higher sampling rate than the input QFVs 212 [0095] - The DecoderBlocks 810 include a transposed Conv1D layer 814 for up-sampling followed by three ResidualUnits 812. Internal convolutional layers of DecoderBlocks 810 and ResidualUnits 812 are shown in FIG. 8B. The decoder 104 uses the same strides as the encoder 102, but in reverse order, to reconstruct a waveform with the same resolution as the input waveform. The number of channels is halved whenever up-sampling. A final Conv1D layer 802 with one filter, a kernel of size 7 and stride 1 projects the feature vectors 208 back to the waveform 112. A FiLM conditioning layer 806 can also be implemented to process the conditioning signal for joint decompression and enhancement);
and performing, based on the up-sampling feature value, up-sampling on the encoding vector of the current audio frame to obtain the decoded data of the current audio frame includes:
inputting the encoding vector of the current audio frame into the decoder, and inputting the plurality of feature vectors into the plurality of up-sampling layers correspondingly ([0041] - The QFVs 212 can then be processed (e.g., decoded) by the decoder 104 to generate an audio waveform 112. The decoder 104 generally mirrors the processes of the encoder 102 by outputting waveforms starting from (quantized) feature vectors. The decoder 104 can up-sample the QFVs 212 to generate the output waveform 206 at a higher sampling rate than the input QFVs 212 [0095] - The DecoderBlocks 810 include a transposed Conv1D layer 814 for up-sampling followed by three ResidualUnits 812. Internal convolutional layers of DecoderBlocks 810 and ResidualUnits 812 are shown in FIG. 8B. The decoder 104 uses the same strides as the encoder 102, but in reverse order, to reconstruct a waveform with the same resolution as the input waveform. The number of channels is halved whenever up-sampling. A final Conv1D layer 802 with one filter, a kernel of size 7 and stride 1 projects the feature vectors 208 back to the waveform 112. A FiLM conditioning layer 806 can also be implemented to process the conditioning signal for joint decompression and enhancement);
and performing up-sampling processing on the encoding vector of the current audio frame and the plurality of feature vectors through the plurality of up-sampling layers, to obtain the decoded data of the current audio frame ([0041] - The QFVs 212 can then be processed (e.g., decoded) by the decoder 104 to generate an audio waveform 112. The decoder 104 generally mirrors the processes of the encoder 102 by outputting waveforms starting from (quantized) feature vectors. The decoder 104 can up-sample the QFVs 212 to generate the output waveform 206 at a higher sampling rate than the input QFVs 212 [0095] - The DecoderBlocks 810 include a transposed Conv1D layer 814 for up-sampling followed by three ResidualUnits 812. Internal convolutional layers of DecoderBlocks 810 and ResidualUnits 812 are shown in FIG. 8B. The decoder 104 uses the same strides as the encoder 102, but in reverse order, to reconstruct a waveform with the same resolution as the input waveform. The number of channels is halved whenever up-sampling. A final Conv1D layer 802 with one filter, a kernel of size 7 and stride 1 projects the feature vectors 208 back to the waveform 112. A FiLM conditioning layer 806 can also be implemented to process the conditioning signal for joint decompression and enhancement).
Regarding claim 3, Zeghidour discloses the audio decoding method, further comprising, before inputting the encoding vector of the current audio frame into the decoder:
obtaining an encoder including a plurality of down-sampling layers ([0031] - The encoder 102 can down-sample the input waveform 202 to generate the compressed feature vectors 208, such that the feature vectors 208 have a lower sampling rate than the original audio waveform 112. For example, the encoder neural network 102 can use multiple convolutional layers with increasing strides to generate feature vectors 208 at the lower sampling rate (e.g., lower temporal resolution));
performing encoding and decoding processing on an audio input sample through the encoder and the decoder to obtain an audio output sample ([0041] - The QFVs 212 can then be processed (e.g., decoded) by the decoder 104 to generate an audio waveform 112. The decoder 104 generally mirrors the processes of the encoder 102 by outputting waveforms starting from (quantized) feature vectors. The decoder 104 can up-sample the QFVs 212 to generate the output waveform 206 at a higher sampling rate than the input QFVs 212);
determining a first loss error between the encoder and the decoder based on the audio input sample and the audio output sample ([0050] - the objective function 214 can utilize a multi-scale spectral reconstruction loss that measures an error between mel-spectrograms of the target waveform 204 and the output waveform 206);
performing type discrimination on the audio input sample and the audio output sample through a sample discriminator to obtain a discrimination result, and determining a second loss error of the sample discriminator based on the discrimination result ([0067] - the discriminator 216 introduces a reconstruction loss in the form of a “feature loss”. Specifically, the feature loss.sub.feat measures an error between the discriminator's internal layer outputs for the target audio waveform 204 and those for the output audio waveform 206);
and performing generative adversarial training on the encoder, the decoder, and the sample discriminator based on the first loss error and the second loss error, to update network parameters of the encoder, the decoder, and the sample discriminator ([0048] - The neural networks are trained end-to-end on an objective function 214 that can include numerous reconstruction losses. In some implementations, a discriminator neural network 216 is also trained to facilitate adversarial losses 218 and, in some cases, additional reconstruction losses).
Regarding claim 4, Zeghidour discloses the audio decoding method, wherein:
the sample discriminator includes an original sample discriminator and a sample feature discriminator ([0060] - the training system 300 exploits a discriminator neural network 216 to incorporate adversarial loss 218 into the objective function 214, and potentially additional reconstruction losses. [0067] - the discriminator 216 introduces a reconstruction loss in the form of a “feature loss”. Specifically, the feature loss.sub.feat measures an error between the discriminator's internal layer outputs for the target audio waveform 204 and those for the output audio waveform 206);
and performing type discrimination on the audio input sample and the audio output sample through the sample discriminator includes:
inputting the audio input sample and the audio output sample into the original sample discriminator to obtain a first-type discrimination result ([0061] - discriminator function that maps an input waveform to output logits. k indexes a particular discriminator output and t indexes a particular logit of the discriminator output. In some implementations, the discriminator 216 utilizes a fully convolutional neural network such that the number of logits is proportional to the length of the input waveform);
performing spectral feature extraction on the audio input sample to obtain a first Mel spectrum, and performing spectral feature extraction on the audio output sample to obtain a second Mel spectrum ([0050] - the objective function 214 can utilize a multi-scale spectral reconstruction loss that measures an error between mel-spectrograms of the target waveform 204 and the output waveform 206. A spectrogram characterizes the frequency spectrum of an audio waveform over time, e.g., using a short-time Fourier transform (STFT)).
and inputting the first Mel spectrum and the second Mel spectrum into the sample feature discriminator to obtain a second-type discrimination result, the discrimination result including the first-type discrimination result and the second-type discrimination result ([0050] - the objective function 214 can utilize a multi-scale spectral reconstruction loss that measures an error between mel-spectrograms of the target waveform 204 and the output waveform 206. A spectrogram characterizes the frequency spectrum of an audio waveform over time, e.g., using a short-time Fourier transform (STFT)).
Regarding claim 5, Zeghidour discloses the audio decoding method, wherein determining the first loss error includes:
performing spectral feature extraction on the audio input sample to obtain a first Mel spectrum, and performing spectral feature extraction on the audio output sample to obtain a second Mel spectrum ([0050] - the objective function 214 can utilize a multi-scale spectral reconstruction loss that measures an error between mel-spectrograms of the target waveform 204 and the output waveform 206. A spectrogram characterizes the frequency spectrum of an audio waveform over time, e.g., using a short-time Fourier transform (STFT));
and determining the first loss error based on a difference between the first Mel spectrum and the second Mel spectrum ([0050] - the objective function 214 can utilize a multi-scale spectral reconstruction loss that measures an error between mel-spectrograms of the target waveform 204 and the output waveform 206. A spectrogram characterizes the frequency spectrum of an audio waveform over time, e.g., using a short-time Fourier transform (STFT)).
Regarding claim 6, Zeghidour discloses the audio decoding method, wherein performing spectral feature extraction on the audio input sample to obtain the first Mel spectrum, and performing spectral feature extraction on the audio output sample to obtain the second Mel spectrum includes:
obtaining a sampling window including at least two sample scales ([0050] - Here, St denotes the t-th frame (e.g., timeslice) of a 64-bin mel-spectrogram computed with window length equal to s and hop length equal to s/4. The α.sub.s coefficient can be set to α.sub.s=√{square root over (s/2)});
and performing spectral feature extraction on the audio input sample at different ones of the at least two sample scales through the sampling window to obtain a multi-scale first Mel spectrum, and performing spectral feature extraction on the audio output sample to obtain a multi-scale second Mel spectrum ([0050] - the objective function 214 can utilize a multi-scale spectral reconstruction loss that measures an error between mel-spectrograms of the target waveform 204 and the output waveform 206. A spectrogram characterizes the frequency spectrum of an audio waveform over time, e.g., using a short-time Fourier transform (STFT)).
Regarding claim 7, Zeghidour discloses the video decoding method wherein:
the up-sampling layer includes at least two sampling channels ([0095] - The number of channels is halved whenever up-sampling);
and performing up-sampling processing on the encoding vector of the current audio frame and the plurality of feature vectors through the plurality of up-sampling layers includes:
performing feature extraction on the encoding vector of the current audio frame and the plurality of feature vectors through the at least two sampling channels in the up-sampling layer, to obtain at least two channel feature values ([0044] - The audio compression/decompression systems 100/200 can be a fully “end-to-end” machine learning approach when implementing this data-driven training solution. In the end-to-end implementation, the compression/decompression systems 100/200 leverage the neural networks for all tasks involved in training, as well as inference after training. No processing, such as feature extraction, is performed by an external system);
obtaining an average value and a variance of the at least two channel feature values ([0056] - the codebooks 110 can be repeatedly updated during training using exponential moving averages of the feature vectors 208);
and performing normalization processing on the at least two channel feature values based on the average value and the variance ([0056] - The training system 300 can also improve usage of the codebooks 110 by running a k-means algorithm on a first set of training examples 116 and using the learned centroids as initialization for the following training examples 116. Alternatively or in addition, if a code vector has not been assigned to a feature vector 208 for numerous training examples 116, the training system 300 can replace it with a random feature vector 208 sampled during a current training example 116. For example, the training system 300 can track the exponential moving average of assignments to each code vector (with a decay factor of 0.99) and replace the code vectors of which this statistic falls below 2).
Regarding claim 8, Zeghidour discloses the audio decoding method, further comprising, before performing normalization processing on the at least two channel feature values:
performing weighted smoothing processing on an average value and a variance among the audio frames to obtain a processed average value and a processed variance ([0056] - The training system 300 can also improve usage of the codebooks 110 by running a k-means algorithm on a first set of training examples 116 and using the learned centroids as initialization for the following training examples 116. Alternatively or in addition, if a code vector has not been assigned to a feature vector 208 for numerous training examples 116, the training system 300 can replace it with a random feature vector 208 sampled during a current training example 116. For example, the training system 300 can track the exponential moving average of assignments to each code vector (with a decay factor of 0.99) and replace the code vectors of which this statistic falls below 2);
wherein performing normalization processing on the at least two channel feature values includes:
performing normalization processing on the at least two channel feature values based on the processed average value and the processed variance ([0056] - The training system 300 can also improve usage of the codebooks 110 by running a k-means algorithm on a first set of training examples 116 and using the learned centroids as initialization for the following training examples 116. Alternatively or in addition, if a code vector has not been assigned to a feature vector 208 for numerous training examples 116, the training system 300 can replace it with a random feature vector 208 sampled during a current training example 116. For example, the training system 300 can track the exponential moving average of assignments to each code vector (with a decay factor of 0.99) and replace the code vectors of which this statistic falls below 2).
Regarding claim 9, Zeghidour discloses the audio decoding method, wherein obtaining the encoding vectors includes:
for one audio frame in the audio frame sequence, obtaining an encoding index value of the one audio frame ([0061] - a discriminator function that maps an input waveform to output logits. k indexes a particular discriminator output and t indexes a particular logit of the discriminator output. In some implementations, the discriminator 216 utilizes a fully convolutional neural network such that the number of logits is proportional to the length of the input waveform);
and querying a codebook for a codebook vector associated with the encoding index value, and determining an encoding vector of the one audio frame based on the codebook vector ([0035] - the first vector quantizer 108, the quantizer 106 can receive the feature vector 208 and select a code vector from its codebook 110 to represent the feature vector 208 based on a smallest distance metric. A residual vector can be computed as the difference between the feature vector 208 and the code vector representing the feature vector 208. The residual vector can be received by the next quantizer 108 in the sequence to select a code vector from its codebook 110 to represent the residual vector based on a smallest distance metric).
Regarding claim 10, Zeghidour discloses the audio decoding method, wherein:
a dimension of the codebook vector is lower than a dimension of the encoding vector ([0032] - The RVQ 108 realizes (lossy) compression by mapping the higher-dimensional space of feature vectors 208 to a discrete subspace of code vectors. As will be elaborated below, a CFV 210 specifies codewords (e.g., indices) from respective codebooks 110 of each vector quantizer 108, where each codeword identifies a code vector stored in the associated codebook 110. Consequently, a QFV 212 is an approximation of a feature vector 208 defined by the combination of code vectors specified by the corresponding CFV 212. Generally, the QFV 212 is a summation (e.g., linear combination) of code vectors specified by the CFV 212);
and determining the encoding vector of the one audio frame based on the codebook vector includes:
performing dimension raising projection on the codebook vector to obtain the encoding vector of the one audio frame ([0032] - The RVQ 108 realizes (lossy) compression by mapping the higher-dimensional space of feature vectors 208 to a discrete subspace of code vectors. As will be elaborated below, a CFV 210 specifies codewords (e.g., indices) from respective codebooks 110 of each vector quantizer 108, where each codeword identifies a code vector stored in the associated codebook 110. Consequently, a QFV 212 is an approximation of a feature vector 208 defined by the combination of code vectors specified by the corresponding CFV 212. Generally, the QFV 212 is a summation (e.g., linear combination) of code vectors specified by the CFV 212).
Regarding claim 11, Zeghidour discloses the audio decoding method, further comprising, before querying the codebook:
obtaining a quantizer configured to maintain the codebook ([0035] - the first vector quantizer 108, the quantizer 106 can receive the feature vector 208 and select a code vector from its codebook 110 to represent the feature vector 208 based on a smallest distance metric. A residual vector can be computed as the difference between the feature vector 208 and the code vector representing the feature vector 208. The residual vector can be received by the next quantizer 108 in the sequence to select a code vector from its codebook 110 to represent the residual vector based on a smallest distance metric);
and training the quantizer, including:
obtaining an encoding vector sample obtained by an encoder performing encoding processing on an audio frame sample ([0031] - The audio waveform 112 is processed (e.g., encoded) by the encoder 102 to generate a sequence of feature vectors 208 representing the waveform 112. Feature vectors 208 (e.g., embeddings, latent representations) are compressed representations of waveforms that extract the most relevant information about their audio content);
predicting a codebook vector sample matching the encoding vector sample through the quantizer ([0035] - the first vector quantizer 108, the quantizer 106 can receive the feature vector 208 and select a code vector from its codebook 110 to represent the feature vector 208 based on a smallest distance metric. A residual vector can be computed as the difference between the feature vector 208 and the code vector representing the feature vector 208. The residual vector can be received by the next quantizer 108 in the sequence to select a code vector from its codebook 110 to represent the residual vector based on a smallest distance metric);
and updating a network parameter of the quantizer based on a loss error between the encoding vector sample and the codebook vector sample, to obtained a trained quantizer ([0032] - The RVQ 108 realizes (lossy) compression by mapping the higher-dimensional space of feature vectors 208 to a discrete subspace of code vectors. As will be elaborated below, a CFV 210 specifies codewords (e.g., indices) from respective codebooks 110 of each vector quantizer 108, where each codeword identifies a code vector stored in the associated codebook 110. Consequently, a QFV 212 is an approximation of a feature vector 208 defined by the combination of code vectors specified by the corresponding CFV 212. Generally, the QFV 212 is a summation (e.g., linear combination) of code vectors specified by the CFV 212);
wherein querying the codebook includes:
querying, through the trained quantizer, the codebook for the codebook vector associated with the encoding index value ([0035] - the first vector quantizer 108, the quantizer 106 can receive the feature vector 208 and select a code vector from its codebook 110 to represent the feature vector 208 based on a smallest distance metric. A residual vector can be computed as the difference between the feature vector 208 and the code vector representing the feature vector 208. The residual vector can be received by the next quantizer 108 in the sequence to select a code vector from its codebook 110 to represent the residual vector based on a smallest distance metric).
Regarding claim 12, Zeghidour discloses the audio decoding method, further comprising, after predicting the codebook vector sample matching the encoding vector sample:
obtaining a statistical parameter of the encoding vector sample matching the codebook vector sample ([0055] - The RVQ function .sub.ψ is parametrized by codebook parameters which can be updated using the objective function 214 to minimize losses during quantization);
and updating the codebook based on the statistical parameter to obtain an updated codebook configured for predicting the codebook vector sample matching the encoding vector sample next time ([0055] - The RVQ function .sub.ψ is parametrized by codebook parameters which can be updated using the objective function 214 to minimize losses during quantization).
Regarding claim 14, Zeghidour discloses a non-transitory computer-readable storage medium storing one or more computer programs that, when executed by one or more processors, cause the one or more processors to perform the method according to claim 1 ([0097] - one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus).
Regarding claim 15, Zeghidour discloses an audio encoding method, performed by a computer device, comprising:
obtaining audio data of audio frames in an audio frame sequence ([0031] - The audio waveform 112 is processed (e.g., encoded) by the encoder 102 to generate a sequence of feature vectors 208 representing the waveform 112. Feature vectors 208 (e.g., embeddings, latent representations) are compressed representations of waveforms that extract the most relevant information about their audio content);
performing, in response to a current audio frame in the audio frame sequence being to be encoded, down-sampling on audio data of a historical audio frame to obtain a down-sampling feature value describing the historical audio frame, the historical audio frame including one or more audio frames encoded before the current audio frame in the audio frame sequence ([0031] - The encoder 102 can down-sample the input waveform 202 to generate the compressed feature vectors 208, such that the feature vectors 208 have a lower sampling rate than the original audio waveform 112. For example, the encoder neural network 102 can use multiple convolutional layers with increasing strides to generate feature vectors 208 at the lower sampling rate (e.g., lower temporal resolution). [0094] - The encoder 102 includes a Conv1D layer 802 followed by four EncoderBlocks 804. Each of the blocks includes three ResidualUnits 812, containing dilated convolutions with dilation rates of 1, 3, and 9, respectively, followed by a down-sampling layer in the form of a strided convolution. Internal convolutional layers of EncoderBlocks 804 and ResidualUnits 812 are shown in FIG. 8B. The number of channels is doubled whenever down-sampling. A final Conv1D layer 802 with a kernel of length 3 and a stride of 1 is used to set the dimensionality of the feature vectors 208 to D. A FiLM conditioning layer 806 can also be implemented to process a conditioning signal for use in joint compression and enhancement. The FiLM layer 806 carries out a feature-wise affine transformation on the neural network's feature vectors 208, conditioned on the conditioning signal).
and performing, based on the down-sampling feature value, down-sampling on the audio data of the current audio frame to obtain an encoding vector of the current audio frame ([0031] - The encoder 102 can down-sample the input waveform 202 to generate the compressed feature vectors 208, such that the feature vectors 208 have a lower sampling rate than the original audio waveform 112. For example, the encoder neural network 102 can use multiple convolutional layers with increasing strides to generate feature vectors 208 at the lower sampling rate (e.g., lower temporal resolution). [0094] - The encoder 102 includes a Conv1D layer 802 followed by four EncoderBlocks 804. Each of the blocks includes three ResidualUnits 812, containing dilated convolutions with dilation rates of 1, 3, and 9, respectively, followed by a down-sampling layer in the form of a strided convolution. Internal convolutional layers of EncoderBlocks 804 and ResidualUnits 812 are shown in FIG. 8B. The number of channels is doubled whenever down-sampling. A final Conv1D layer 802 with a kernel of length 3 and a stride of 1 is used to set the dimensionality of the feature vectors 208 to D. A FiLM conditioning layer 806 can also be implemented to process a conditioning signal for use in joint compression and enhancement. The FiLM layer 806 carries out a feature-wise affine transformation on the neural network's feature vectors 208, conditioned on the conditioning signal).
Regarding claim 16, Zeghidour discloses a non-transitory computer-readable storage medium storing one or more computer programs that, when executed by one or more processors, cause the one or more processors to perform the method according to claim 15 ([0097] - one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus).
Regarding claim 17, Zeghidour discloses an electronic device comprising:
one or more processors ([0098] - programmable processor, a computer, or multiple processors or computers);
and one or more memories storing one or more computer programs that, when executed by the one or more processors, cause the electronic device to perform the method according to claim 15 ([0097] - The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device).
Regarding claim 18, Zeghidour discloses an electronic device comprising:
one or more processors ([0098] - programmable processor, a computer, or multiple processors or computers);
and one or more memories storing one or more computer programs that, when executed by the one or more processors, cause the electronic device to ([0097] - The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device):
obtain encoding vectors of audio frames in an audio frame sequence ([0031] - The audio waveform 112 is processed (e.g., encoded) by the encoder 102 to generate a sequence of feature vectors 208 representing the waveform 112. Feature vectors 208 (e.g., embeddings, latent representations) are compressed representations of waveforms that extract the most relevant information about their audio content);
perform, in response to a current audio frame in the audio frame sequence being to be decoded, up-sampling on an encoding vector of a historical audio frame to obtain an up-sampling feature value describing the historical audio frame, the historical audio frame including one or more audio frames decoded before the current audio frame in the audio frame sequence ([0041] - The QFVs 212 can then be processed (e.g., decoded) by the decoder 104 to generate an audio waveform 112. The decoder 104 generally mirrors the processes of the encoder 102 by outputting waveforms starting from (quantized) feature vectors. The decoder 104 can up-sample the QFVs 212 to generate the output waveform 206 at a higher sampling rate than the input QFVs 212 [0095] - The DecoderBlocks 810 include a transposed Conv1D layer 814 for up-sampling followed by three ResidualUnits 812. Internal convolutional layers of DecoderBlocks 810 and ResidualUnits 812 are shown in FIG. 8B. The decoder 104 uses the same strides as the encoder 102, but in reverse order, to reconstruct a waveform with the same resolution as the input waveform. The number of channels is halved whenever up-sampling. A final Conv1D layer 802 with one filter, a kernel of size 7 and stride 1 projects the feature vectors 208 back to the waveform 112. A FiLM conditioning layer 806 can also be implemented to process the conditioning signal for joint decompression and enhancement);
and perform, based on the up-sampling feature value, up-sampling on an encoding vector of the current audio frame to obtain decoded data of the current audio frame ([0041] - The QFVs 212 can then be processed (e.g., decoded) by the decoder 104 to generate an audio waveform 112. The decoder 104 generally mirrors the processes of the encoder 102 by outputting waveforms starting from (quantized) feature vectors. The decoder 104 can up-sample the QFVs 212 to generate the output waveform 206 at a higher sampling rate than the input QFVs 212 [0095] - The DecoderBlocks 810 include a transposed Conv1D layer 814 for up-sampling followed by three ResidualUnits 812. Internal convolutional layers of DecoderBlocks 810 and ResidualUnits 812 are shown in FIG. 8B. The decoder 104 uses the same strides as the encoder 102, but in reverse order, to reconstruct a waveform with the same resolution as the input waveform. The number of channels is halved whenever up-sampling. A final Conv1D layer 802 with one filter, a kernel of size 7 and stride 1 projects the feature vectors 208 back to the waveform 112. A FiLM conditioning layer 806 can also be implemented to process the conditioning signal for joint decompression and enhancement).
Regarding claim 19, Zeghidour discloses the device, wherein the one or more computer programs, when executed by the one or more processors, further cause the electronic device to:
perform up-sampling on the encoding vector of the historical audio frame using a plurality of up-sampling layers of a decoder to obtain the up-sampling feature value including a plurality of feature vectors, respectively ([0041] - The QFVs 212 can then be processed (e.g., decoded) by the decoder 104 to generate an audio waveform 112. The decoder 104 generally mirrors the processes of the encoder 102 by outputting waveforms starting from (quantized) feature vectors. The decoder 104 can up-sample the QFVs 212 to generate the output waveform 206 at a higher sampling rate than the input QFVs 212 [0095] - The DecoderBlocks 810 include a transposed Conv1D layer 814 for up-sampling followed by three ResidualUnits 812. Internal convolutional layers of DecoderBlocks 810 and ResidualUnits 812 are shown in FIG. 8B. The decoder 104 uses the same strides as the encoder 102, but in reverse order, to reconstruct a waveform with the same resolution as the input waveform. The number of channels is halved whenever up-sampling. A final Conv1D layer 802 with one filter, a kernel of size 7 and stride 1 projects the feature vectors 208 back to the waveform 112. A FiLM conditioning layer 806 can also be implemented to process the conditioning signal for joint decompression and enhancement);
input the encoding vector of the current audio frame into the decoder, and input the plurality of feature vectors into the plurality of up-sampling layers correspondingly ([0041] - The QFVs 212 can then be processed (e.g., decoded) by the decoder 104 to generate an audio waveform 112. The decoder 104 generally mirrors the processes of the encoder 102 by outputting waveforms starting from (quantized) feature vectors. The decoder 104 can up-sample the QFVs 212 to generate the output waveform 206 at a higher sampling rate than the input QFVs 212 [0095] - The DecoderBlocks 810 include a transposed Conv1D layer 814 for up-sampling followed by three ResidualUnits 812. Internal convolutional layers of DecoderBlocks 810 and ResidualUnits 812 are shown in FIG. 8B. The decoder 104 uses the same strides as the encoder 102, but in reverse order, to reconstruct a waveform with the same resolution as the input waveform. The number of channels is halved whenever up-sampling. A final Conv1D layer 802 with one filter, a kernel of size 7 and stride 1 projects the feature vectors 208 back to the waveform 112. A FiLM conditioning layer 806 can also be implemented to process the conditioning signal for joint decompression and enhancement);
and perform up-sampling processing on the encoding vector of the current audio frame and the plurality of feature vectors through the plurality of up-sampling layers, to obtain the decoded data of the current audio frame ([0041] - The QFVs 212 can then be processed (e.g., decoded) by the decoder 104 to generate an audio waveform 112. The decoder 104 generally mirrors the processes of the encoder 102 by outputting waveforms starting from (quantized) feature vectors. The decoder 104 can up-sample the QFVs 212 to generate the output waveform 206 at a higher sampling rate than the input QFVs 212 [0095] - The DecoderBlocks 810 include a transposed Conv1D layer 814 for up-sampling followed by three ResidualUnits 812. Internal convolutional layers of DecoderBlocks 810 and ResidualUnits 812 are shown in FIG. 8B. The decoder 104 uses the same strides as the encoder 102, but in reverse order, to reconstruct a waveform with the same resolution as the input waveform. The number of channels is halved whenever up-sampling. A final Conv1D layer 802 with one filter, a kernel of size 7 and stride 1 projects the feature vectors 208 back to the waveform 112. A FiLM conditioning layer 806 can also be implemented to process the conditioning signal for joint decompression and enhancement)
Regarding claim 20, Zeghidour discloses the device, wherein the one or more computer programs, when executed by the one or more processors, further cause the electronic device to, before inputting the encoding vector of the current audio frame into the decoder:
obtain an encoder including a plurality of down-sampling layers ([0031] - that extract the most relevant information about their audio content. The encoder 102 can down-sample the input waveform 202 to generate the compressed feature vectors 208);
perform encoding and decoding processing on an audio input sample through the encoder and the decoder to obtain an audio output sample ([0041] - The QFVs 212 can then be processed (e.g., decoded) by the decoder 104 to generate an audio waveform 112. The decoder 104 generally mirrors the processes of the encoder 102 by outputting waveforms starting from (quantized) feature vectors. The decoder 104 can up-sample the QFVs 212 to generate the output waveform 206 at a higher sampling rate than the input QFVs 212);
determine a first loss error between the encoder and the decoder based on the audio input sample and the audio output sample ([0050] - the objective function 214 can utilize a multi-scale spectral reconstruction loss that measures an error between mel-spectrograms of the target waveform 204 and the output waveform 206);
perform type discrimination on the audio input sample and the audio output sample through a sample discriminator to obtain a discrimination result, and determining a second loss error of the sample discriminator based on the discrimination result ([0067] - the discriminator 216 introduces a reconstruction loss in the form of a “feature loss”. Specifically, the feature loss.sub.feat measures an error between the discriminator's internal layer outputs for the target audio waveform 204 and those for the output audio waveform 206);
and perform generative adversarial training on the encoder, the decoder, and the sample discriminator based on the first loss error and the second loss error, to update network parameters of the encoder, the decoder, and the sample discriminator ([0048] - The neural networks are trained end-to-end on an objective function 214 that can include numerous reconstruction losses. In some implementations, a discriminator neural network 216 is also trained to facilitate adversarial losses 218 and, in some cases, additional reconstruction losses).
Allowable Subject Matter
7. Claim 13 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims with the rejection under 35 U.S.C. 101 overcome.
The following is an examiner’s statement of reasons for allowance: claim 13 contains the limitations “the statistical parameter includes at least one of a vector sum or a quantity of hits, the vector sum representing an average value vector obtained by performing weighted average processing on encoding vector samples, and the quantity of hits representing a quantity of encoding vector samples matching the codebook vector sample; and updating the codebook based on the statistical parameter includes: performing exponential weighted smoothing on the codebook based on the vector sum; and performing Laplacian smoothing on the codebook based on the quantity of hits”. At the time of the effective filing date of the application, these limitations had not been fully anticipated and it would not have been obvious to one of ordinary skill in the art to combine elements of the prior art to meet this limitation.
The closest prior art, Zeghidour (U.S. Publication No. 20230019128) either singularly or in combination fail to anticipate or render obvious the above described limitations. Zeghidour discloses compressing audio waveforms using neural networks and vector quantizers.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
8. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ravelli (U.S. Publication No. 20200265855) teaches encoding and decoding audio signals. Sharma (U.S. Publication No. 20140142958) teaches multi-mode audio recognition and auxiliary data encoding and decoding.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ETHAN DANIEL KIM whose telephone number is (571) 272-1405. The examiner can normally be reached on Monday - Friday 9:00 - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richemond Dorvil can be reached on (571) 272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ETHAN DANIEL KIM/
Examiner, Art Unit 2658
/RICHEMOND DORVIL/Supervisory Patent Examiner, Art Unit 2658
Prosecution Insights