DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments with respect to 35 U.S.C. 102 in regards to claims 1 and 21 have been considered but are moot due to new grounds of rejection necessitated by amendments. See detailed rejection below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6, 8, 11-12 and 21-24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Baby et al. (“iSEGAN: Improved Speech Enhancement Generative Adversarial Networks”; 2/20/2020; Idiap Research Institute, Martigny, Switzerland) in view of Nesta et al. (US 2020/0349965).
Claims 1 and 21,
Baby teaches a neural network based system for speech enhancement of an audio signal ([Abstract] [Intro] Popular neural network-based speech enhancement systems operate on the magnitude spectrogram and ignore the phase mismatch between the noisy and clean speech signals; SPEECH enhancement systems aim to improve the quality and intelligibility of acquired speech signals by removing artefacts caused by background noise or other interferences such as room reverberation),
the system comprising a generative network for generating an enhanced audio signal ([Intro] [II.] this paper investigates the use of generative neural networks which can directly map the raw noisy speech waveform to the underlying clean speech waveform; G generates an estimate that is closer to the underlying clean speech signal x (denoted as ˆx , G(y, z)));
wherein the generative network is separate from the conditioning network and is configured to: receive a noise vector ([Fig. 1 caption] [III.B.] yc is the encoder output of noisy speech and z are samples from the prior distribution Z; The latent noise input z of size 8×1024 was drawn from a normal distribution N(0, I)) and
the conditioning information as input ([II.] [Fig. 2 caption] The encoder (Genc) projects the input noisy signal into a condensed representation yc = Genc(y), which is concatenated with the latent samples z; yc denotes the encoder output corresponding to the input noisy speech signal and z are the samples from the prior distribution Z); and
generate the enhanced audio signal based on the noise vector and the conditioning information ([II.] The decoder (Gdec) then reconstructs the signal such that its output ˆx = Gdec(yc, z) fools D into classifying it as real; G generates an estimate that is closer to the underlying clean speech signal x (denoted as ˆx , G(y, z))).
The difference between the prior art and the claimed invention is that Baby does not explicitly teach a conditioning network for generating conditioning information for the generative network, wherein the conditioning network comprises an encoder-decoder network comprising a plurality of layers and is configured to: receive the audio signal as input; propagate the audio signal through the plurality of layers; and provide one or more first internal representations of the audio signal or processed versions thereof as the conditioning information, wherein the one or more first internal representations of the audio signal are extracted at respective layers of the conditioning network.
Nesta teaches a conditioning network for generating conditioning information for the generative network ([0018] The present disclosure provides improved systems and methods for denoising and target signal enhancement. In various embodiments, prior information about both a target signal (e.g., target speech) and noise in the form of deep embeddings is provided. Two embedding networks are trained to encode and disentangle specific characteristics of the noise and the target signal so that similar sounds within these categories have close embeddings),
wherein the conditioning network comprises an encoder-decoder network comprising a plurality of layers ([[0020] [Fig. 1B] Fig. 1B illustrates an autoencoder neural network 120 that is trained using clean speech samples 122 to extract a clean speech waveform 124; The autoencoder neural network 120 is a pre-processing neural network that includes an input layer 126, an output layer 128 and a plurality of hidden layers, including a speech embedding vector 130 in one of the hidden layers) and is configured to:
receive the audio signal as input ([0009] [0019] a pre-processing neural network configured to receive a segment of the input audio signal; an input layer 104 that receives audio samples 102);
propagate the audio signal through the plurality of layers ([0020] [claim 14] an input layer and plural hidden layers: an input layer 126, an output layer 128 and a plurality of hidden layers; claim 14 further states: processing the segment of the audio signal through the speech pre-processing neural network; and constructing the embedding vector from the hidden layer); and
provide one or more first internal representations of the audio signal or processed versions thereof as the conditioning information ([0009] [0019] [0022] the pre-processing neural network including at least one hidden layer comprising an embedding vector, and a noise reduction neural network configured to receive the segment of the input audio signal, and the embedding vector and generate the enhanced target signal; An activation vector 112 of the last hidden layer is used to describe a particular output noise classification; the embedding is computed through the respective supervising embedding DNNs (e.g., through an embedding vector extracted from the DNN) and an aggregated average embedding (e.g., aggregate average embedding vector) is computed),
wherein the one or more first internal representations of the audio signal are extracted at respective layers of the conditioning network ([0026] [claim 13] a corresponding noise embedding vector 232 is generated from a hidden layer; a corresponding speech embedding vector 242 is generated from a hidden layer; see Figs. 1A-C; claim 13 states: the embedding vector corresponding to a hidden layer of a pre-processing neural network).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Baby with teachings of Nesta by modifying the speech enhancement generative adversarial networks as taught by Baby to include a conditioning network for generating conditioning information for the generative network, wherein the conditioning network comprises an encoder-decoder network comprising a plurality of layers and is configured to: receive the audio signal as input; propagate the audio signal through the plurality of layers; and provide one or more first internal representations of the audio signal or processed versions thereof as the conditioning information, wherein the one or more first internal representations of the audio signal are extracted at respective layers of the conditioning network as taught by Nesta for the benefit of enhancing target audio signals corresponding to one or more acoustic sources (Nesta [0002]).
Claims 2 and 22,
Baby further teaches the system according to claim 1, wherein the first internal representations of the conditioning information relate to a hierarchy of representations of the audio signal, at different temporal resolutions ([Fig. 1] [Section II.A] a multi-layer hierarchy: Genc projects and compresses the input noisy signal through several strided convolutions layers; progressive temporal resolutions: This work used 11 different convolutional layers each for Genc and Gdec with filter-length 31 and stride =2. Thus, after every layer, the temporal dimension of the features gets halved; Fig. 2 depicts output shapes, see Fig. 2; using intermediate representations: G uses U-Shaped skip-connections that bypass the intermediate compression stages and these directly pass the fine-grained information such as phase and alignment to the decoder; this a typical hierarchical multi-resolution representation).
Claims 3 and 23,
Baby further teaches the system according to claim 1, wherein each first internal representation of the conditioning information or a processed version thereof is combined with a respective second internal representation in the generative network ([Fig. 2 caption] [II.A] [III.B] Generator architecture: encoder-decoder structure featuring U-shaped skip-connections employed for speech enhancement; Also notice that G uses U-Shaped skip-connections that bypass the intermediate compression stages, see Fig. 2; These skip connections directly pass the fine-grained information such as phase and alignment to the decoder; Gdec followed the reverse procedure together with skip-connections that doubled the temporal dimension after every layer).
Claims 4 and 24,
Nesta further teaches the system according to claim 1, wherein the conditioning network is further configured to receive first side information as input ([0023] [0030] [0033] metadata side information: metadata is used to retrieve a predefined embedding from a collection of embeddings describing known categories. The metadata may include any information identifying the nature of the audio signals; user provided side information: a user interface 510 may be provided allowing the user to input information on the source speaker; the pre-processing program receives side information: The data pre-processing program takes as input the speaker feature settings provided by the user and the noise recording), and
wherein processing of the audio signal by the conditioning network depends on the first side information ([0033] [claim 19] [claim 20] dependence on metadata: the embedding vector is generated based at least in part on metadata associated with the segment of the audio signal, see claim 19; dependence on user designation: receiving a user designation of at least one audio signal property, and wherein the embedded vector is generated in accordance with the user designation, see claim 20; side-information driven embedding generation: It outputs a fixed-sized embedding for each of these that can be used in the denoising program along with the mixed signal).
Claim 6,
Baby further teaches the system according to claim 1, wherein the generative network is further configured to receive second side information as input ([II.] [Fig. 1 caption] that cGAN generation is controlled by an additional input: With cGANs, we direct this data generation process based on the input noisy speech y such that G generates an estimate that is closer to the underlying clean speech signal x (denoted as ˆx , G(y, z)); yc is the encoder output of noisy speech and z are samples from the prior distribution Z), and
wherein processing of the noise vector by the generative network depends on the second side information ([II.] joint processing of z and the conditioning/side representation: yc = Genc(y), which is concatenated with the latent samples z; the decoder (Gdec) then reconstructs the signal such that its output ˆx = Gdec(yc, z) fools D into classifying it as real).
Claim 8,
Nesta further teaches the system according to claim 1, wherein the plurality of layers of the conditioning network comprise one or more intermediate layers ([0019-0020] The autoencoder neural network 120 is a pre-processing neural network that includes an input layer 126, an output layer 128 and a plurality of hidden layers, including a speech embedding vector 130 in one of the hidden layers; a DNN with an input layer 104 that receives audio samples 102, a plurality of hidden layers 110 and an output layer 106), and
wherein the one or more first internal representations of the audio signal are extracted from the one or more intermediate layers (the hidden layer include a speech embedding vector 130 in one of the hidden layers, and that the inner-most hidden layer is used for speech embedding; claim 13 recites: the embedding vector corresponding to a hidden layer of a pre-processing neural network; claim 14 recites: constructing the embedding vector from the hidden layer).
Claim 11,
Baby further teaches the system according to claim 1, wherein the generative network is based on one of a diffusion-based model, a variational autoencoder, an autoregressive model, and a Generative Adversarial Network formulation ([Introduction] GAN).
Claim 12,
Baby further teaches the system according to claim 1, wherein the generative network is based on an encoder-decoder structure ([II.] G makes us of an encoder-decoder structure; The encoder (Genc) projects the input noisy signal into a condensed representation yc = Genc (y)… and the decoder (Gdec) then reconstructs the signal),
wherein the encoder-decoder structure uses ResNets and/or the encoder part of the encoder- decoder structure comprises one or more skip connections ([Fig. 2 caption] [III.B] Fig. 2 caption: Generator architecture: Encoder-decoder structure featuring U-shaped skip-connections employed for speech enhancement; G uses U-shaped skip-connections that bypass the intermediate compression stages (Fig. 2). These skip connections directly pass the fine-grained information such as phase and alignment to the decoder; Gdec followed the reverse procedure together with skip connections).
Claims 5 and 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Baby et al. (“iSEGAN: Improved Speech Enhancement Generative Adversarial Networks”; 2/20/2020; Idiap Research Institute, Martigny, Switzerland) in view of Nesta et al. (US 2020/0349965) and further in view of Valentini-Botinhao et al. (“Speech Enhancement of Noisy and Reverberant Speech for Text-to-Speech”, IEEE 2018).
Claim 5,
Baby and Nesta teach all the limitations in claim 4. The difference between the prior art and the claimed invention is that Baby nor Nesta explicitly teach wherein the first side information comprises a numeric description of one or more of: a type of artifact present in the audio signal, a level of noise present in the audio signal, an enhancement operation to be performed on the audio signal.
Valentini-Botinhao teaches wherein the first side information comprises a numeric description of one or more of: a type of artifact present in the audio signal, a level of noise present in the audio signal, an enhancement operation to be performed on the audio signal ([III.B] [VIII.A-B] numeric noise level description: Each of these noises were added to the speech signal at four different signal-to-noise (SNR) values: 15 dB, 10 dB, 5 dB and 0 dB; use of noise level and characteristics at enhancement time: This motivates the use, at run time, of modules that estimate SNR and noise characteristics so that a model trained with matched conditions can be applied; additional input use: Another way to improve performance … would be to append speaker and environment information as additional input of the network).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Baby with teachings of Valentini-Botinhao by modifying the speech enhancement generative adversarial networks as taught by Baby to include wherein the first side information comprises a numeric description of one or more of: a type of artifact present in the audio signal, a level of noise present in the audio signal, an enhancement operation to be performed on the audio signal as taught by Valentini-Botinhao for the benefit of enhancing text-to-speech voices created from noisy and reverberant recordings at a lower quality (Valentine-Botinhao [Abstract]).
Claim 7,
Baby and Nesta teach all the limitations in claim 6. The difference between the prior art and the claimed invention is that Baby nor Nesta explicitly teach wherein the second side information comprises a numeric description of one or more of: a type of artifact present in the audio signal, a level of noise present in the audio signal, an enhancement operation to be performed on the audio signal, and information on characteristics of the audio signal.
Valentini-Botinhao teaches wherein the second side information comprises a numeric description of one or more of: a type of artifact present in the audio signal, a level of noise present in the audio signal, an enhancement operation to be performed on the audio signal, and information on characteristics of the audio signal ([III.B] [VIII.A-B] numeric noise level description: Each of these noises were added to the speech signal at four different signal-to-noise (SNR) values: 15 dB, 10 dB, 5 dB and 0 dB; use of noise level and characteristics at enhancement time: This motivates the use, at run time, of modules that estimate SNR and noise characteristics so that a model trained with matched conditions can be applied; additional input use: Another way to improve performance … would be to append speaker and environment information as additional input of the network).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Baby with teachings of Valentini-Botinhao by modifying the speech enhancement generative adversarial networks as taught by Baby to include wherein the second side information comprises a numeric description of one or more of: a type of artifact present in the audio signal, a level of noise present in the audio signal, an enhancement operation to be performed on the audio signal, and information on characteristics of the audio signal as taught by Valentini-Botinhao for the benefit of enhancing text-to-speech voices created from noisy and reverberant recordings at a lower quality (Valentine-Botinhao [Abstract]).
Claim 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Baby et al. (“iSEGAN: Improved Speech Enhancement Generative Adversarial Networks”; 2/20/2020; Idiap Research Institute, Martigny, Switzerland) in view of Nesta et al. (US 2020/0349965) and further in view of Pandey et al. (“A New Framework for CNN-Based Speech Enhancement in the Time Domain”, IEEE 2019).
Claim 10,
Baby and Nesta teach all the limitations in claim 1. The difference between the prior art and the claimed invention is that Baby nor Nesta explicitly teach wherein the one or more first internal representations are extracted at the one or more respective layers of the decoder network of the encoder-decoder network of the conditioning network.
Pandey teaches wherein the one or more first internal representations are extracted at the one or more respective layers of the decoder network of the encoder-decoder network of the conditioning network ([III.C] a speech enhancement autoencoder: We use an autoencoder based fully convolutional neural network with skip connections first proposed for time domain speech enhancement in [8], which was adopted from U-Net [23]; decoder layers: The decoder mirrors the encoder, consisting of a series of deconvolutional layers…; decoder-layer outputs/internal representations: The output of each layer in the decoder is concatenated with the output from the corresponding symmetric layer of the encoder along the channel axis; lists the dimensionality of the outputs from the successive layers in the proposed network…).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Baby with teachings of Pandey by modifying the speech enhancement generative adversarial networks as taught by Baby to include wherein the one or more first internal representations are extracted at the one or more respective layers of the decoder network of the encoder-decoder network of the conditioning network as taught by Pandey for the benefit of proposing an easy way to implement and apply speech processing tasks that require time-frequency masking or spectral mapping (Pandey [Abstract]).
Allowable Subject Matter
Claims 13, 15 and 17-19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: Baby et al. (“iSEGAN: Improved Speech Enhancement Generative Adversarial Networks”; 2/20/2020; Idiap Research Institute, Martigny, Switzerland) in view of Nesta et al. (US 2020/0349965):
Baby and Nesta teach all the limitations in claim 1. Claims 15 and 17-19 depend on claim 13.
With respect to claim 13, Baby further teaches the system according to claim 1, wherein the system has been trained prior to inference ([II.] The training phrases of a cGAN-based speech enhancement system are depicted in Fig. 1; training a cGAN-based speech enhancement setting is comprised of repeating the following three updates for every mini-batch over several epochs),
using data pairs each comprising a clean audio signal and a distorted audio signal corresponding to or derived from the clean audio signal ([Fig. 1] [II.] y, x and x^ are the noisy speech, clean speech and the clean speech estimate generated by G, respectively; The goal of a speech enhancement system is to estimate the clean speech signal x from the noisy mixture y = x + w, where w is the added background noise),
wherein the distorted audio signal comprises noise and/or artifacts ([I.] speech enhancement systems aim to improve the quality and intelligibility of acquired speech signals by removing artefacts caused by background noise or other interferences such as room reverberation; w is the added background noise),
wherein one or more of the data pairs comprise a respective clean audio signal and a respective distorted audio signal that has been generated by programmatic transformation of the clean audio signal and/or addition of noise ([III.A] The recordings were added with 10 different noise conditions … at signal to noise ratios (SNRs) of 0, 5, 10 and 15 dB), and
The difference between the prior art and the claimed invention is that Baby nor Nesta explicitly teach wherein the programmatic transformation comprises one or more of adding reverberation, low-pass filtering, clipping, packet loss simulation, transcoding, random equalization and level dynamics distortion.
Therefore, it would not have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Baby and Nesta to include wherein the programmatic transformation comprises one or more of adding reverberation, low-pass filtering, clipping, packet loss simulation, transcoding, random equalization and level dynamics distortion.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHREYANS A PATEL whose telephone number is (571)270-0689. The examiner can normally be reached Monday-Friday 8am-5pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SHREYANS A. PATEL
Primary Examiner
Art Unit 2653
/SHREYANS A PATEL/Examiner, Art Unit 2659
Prosecution Insights