Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claims 1-20 are pending. Claims 1, 12, and 20 are independent.
This Application was published as US 20230253003.
Apparent priority is 27 November 2020.
Response to Arguments
35 USC 103
Applicant’s arguments with respect to 35 USC 103 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-10 and 12-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hu et al. ("DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement") in view of Yu et al. (US 10283140 B1).
Regarding claim 1, Hu discloses: 1. A speech processing method, comprising: obtaining a first spectrum of a noisy speech in a complex number domain; ("Conv-STFT" Fig. 1 - See also Pg. 3, section 3.2. "The input and output are the real and imaginary part of the noisy and estimated STFT complex spectrogram.")
performing subband division on the first spectrum to obtain first subband spectra of the noisy speech in the complex number domain, (Fig. 2, “Input Complex spectrum”) the first subband spectra each comprising a plurality of complex numbers including real parts and imaginary parts, the first subband spectra each being in a form of a two-dimensional vector sequence or a matrix, ("At the same time, we define the input complex matrix X = Xr +jXi" pg. 2 section 2.2.) (subband division is not explicitly disclosed)
the subband division comprising dividing a frequency domain of the first spectrum into a plurality of subbands of non-equal width; (not explicitly disclosed)
processing the first subband spectra based on a pre-trained noise reduction model to obtain second subband spectra of a target speech in the noisy speech in the complex number domain; ("(Complex) LSTM" Fig. 1 - See also Pg. 3, section 3.2. which discloses that the LSTM is trained.)
performing subband restoration on the second subband spectra to obtain a second spectrum in the complex number domain, the subband restoration comprising splicing the second subband spectra along a frequency dimension to obtain the second spectrum; (not explicitly disclosed)
and synthesizing the target speech based on the second spectrum, ("Conv-STFT" Fig. 1 - See also Pg. 3, section 2.3. "In details, we use STFT kernel initialized convolution/deconvolution module to analyze/synthesize waveform [29] before sending to network and calculating the loss function.")
wherein: the noise reduction model is obtained based on training of a deep complex convolutional recurrent network; ("In this paper, we build upon previous network architectures to design a new complex-valued speech enhancement network, called deep complex convolution recurrent network (DCCRN)" Pg. 2, Section 1.2.)
the deep complex convolutional recurrent network comprises an encoding network in the complex number domain, (Fig. 2 shows an encoding network in the complex domain )
a decoding network in the complex number domain, ("Complex Decoder" Fig. 1)
and a long short-term memory network in the complex number domain, ("(Complex) LSTM" Fig. 1)
and the encoding network and the decoding network are connected through the long short-term memory network, (Fig. 1 shows the Complex Encoder and Complex Decoder are connected through the LSTM. )
the encoding network comprises a plurality of layers of complex encoders, (Fig. 1 shows at least 2 Complex Encoders. )
the decoding network comprises a plurality of layers of complex decoders, (Fig. 1 shows at least 2 Complex Decoders.)
the processing the first subband spectra based on the pre-trained noise reduction model to obtain the second subband spectra of the target speech in the noisy speech in the complex number domain comprises inputting the first subband spectra of the noisy speech in the complex number domain into a first layer of complex encoder of the plurality of layers of complex encoders of the encoding network, and (Fig. 1 shows the Input Noisy Wav is input to the Complex Encoder after the STFT)
a last layer of complex decoder of the plurality of layers of complex decoders of the decoding network outputs the second subband spectra of the target speech in the noisy speech in the complex number domain. (Fig. 1 shows Complex Decoder outputs the target speech through the STFT)
Hu does not disclose performing subband division and restoration. Hu discloses processing a single complex spectrum to remove noise.
Yu discloses: performing subband division on the first spectrum to obtain a first subband spectra of the noisy speech… ("Spectrum analyzer 106 splits the feature matrix into individual sub-bands (illustrated as sub-bands 1 through N)." Col 5; 14-15)
the subband division comprising dividing a frequency domain of the first spectrum into a plurality of subbands of non-equal width; (“In other embodiments, the size of each sub-band may be non-uniform. For example, the method may partition the previous 8000 Hz signal into sub-bands of 0-1000 Hz, 1000-4000 Hz, 4000-6000 Hz, and 6000-8000 Hz.” Col 6; 66 – Col 7; 3)
and performing subband restoration on the second subband spectra to obtain a second spectrum… ("In step 212, the method concatenates the clean signals into an output matrix output. In one embodiment, the method may combine each of the output vectors generated by the DNNs in the proper order based on the location of each sub-band. In one embodiment, the resulting output is a clean audio feature matrix having the same dimensions as the feature matrix the method generated in step 204 but having clean audio signal DFT coefficients instead of the original DFT coefficients." Col 7; 56-64)
the subband restoration comprising splicing the second subband spectra along a frequency dimension to obtain the second spectrum; (Fig. 1A shows that the CONCATENATOR 110 combines the sub-band outputs before the output is transformed to the time-domain.)
Hu and Yu are considered analogous art to the claimed invention because they disclose noise suppression methods. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hu to process the audio in sub-bands using the method disclosed by Yu. Doing so would have been beneficial because clean speech has different time-frequency characteristics in different sub-bands. (Yu Col 2, para 4)
Regarding claim 2, Hu discloses: 2. The method according to claim 1, wherein the obtaining the first spectrum of the noisy speech in the complex number domain comprises: performing short-time Fourier transform on the noisy speech to obtain the first spectrum of the noisy speech in the complex number domain; ("Conv-STFT" Fig. 1- STFT stands for short-time Fourier transform. (ref pg. 1, section 1.1.))
and the synthesizing the target speech based on the second spectrum comprises: performing inverse transform of short-time Fourier transform on the second spectrum to obtain the target speech. (In Fig. 1, the second Conv-STFT shows data goes from a complex domain to a waveform, so this is clearly an inverse STFT.)
Regarding claim 3, Hu does not disclose the additional limitations.
Yu discloses: 3. The method according to claim 1, wherein the performing subband division on the first spectrum to obtain the first subband spectra in the complex number domain further comprises: dividing the first spectrum according to the divided subbands to obtain the first subband spectra in one-to-one correspondence with the divided subbands. ("Spectrum analyzer 106 splits the feature matrix into individual sub-bands (illustrated as sub-bands 1 through N)." Col 5; 14-15)
Rationale for combination as provided for claim 1.
Regarding claim 4, Hu discloses: 4. The method according to claim 1, wherein each layer of complex encoder comprises a complex convolution layer, ("Real Conv"; "Imag Conv" Fig. 2)
a batch normalization layer, ("Complex BatchNormalazation" Fig. 2)
and an activation unit layer; ("Real PReLU" Fig. 2 - PReLU is an activation function)
each layer of complex decoder comprises a complex deconvolution layer, a batch normalization layer, and an activation unit layer; ("Conv2d block is composed of a convolution/
deconvolution layer followed by batch normalization and activation function." Pg. 2, section 2.1.)
and a number of layers of the complex encoder in the encoding network is the same as a number of layers of the complex decoder in the decoding network, (Fig. 1 shows 2 encoders and 2 decoders )
and the complex encoder in the encoding network and the complex decoder in a reverse order in the decoding network are in a one-to-one correspondence and are connected. (Fig. 1 shows the decoders in a one-to-one correspondence and connected to the encoders. See Pg. 2, Section 2.1. "Subsequently, the decoder reconstructs the low resolution features to the original size of the input, leading the encoder-decoder structure to a symmetric design."; See also pg. 2, Section 2.1. "Skip-connection is conducive to flowing the gradient by concentrating the encoder and decoder. )
Regarding claim 5, Hu discloses: 5. The method according to claim 4, wherein the complex convolution layer comprises a first real part convolution kernel and a first imaginary part convolution kernel; ("The complex-valued convolutional filter W is defined as W = Wr+jWi, where the real-valued matrices Wr and Wi represent the real and imaginary part of a complex convolution kernel, respectively." Pg. 2, Section 2.2.)
and the complex encoder is configured to perform operations comprising: convolving a received real part and a received imaginary part through the first real part convolution kernel, to obtain a first output and a second output, and convolving the received real part and the received imaginary part through the first imaginary part convolution kernel, to obtain a third output and a fourth output; ("Therefore, we can get complex output Y from the complex convolution operation X ~W:" Pg. 2, Section 2.2.)
performing a complex multiplication operation on the first output, the second output, the third output, and the fourth output based on a complex multiplication rule, to obtain a first operation result in the complex number domain; ("Fout = (Xr *Wr - Xi *Wi)+j(Xr *Wi +Xi *Wr) (1)" Pg. 2, Section 2.2.)
sequentially processing the first operation result through the batch normalization layer and the activation unit layer in the complex encoder, to obtain an encoding result in the complex number domain, wherein the encoding result comprises a real part and an imaginary part; (Fig. 2 shows batch normalization and PReLU (activation) which outputs a complex spectrum. )
and inputting the real part and the imaginary part of the encoding result to a network structure of a next layer. (Fig. 1 shows the complex encoder output is an input to the next layer. )
Regarding claim 6, Hu discloses: 6. The method according to claim 5, wherein the long short-term memory network comprises a first long short-term memory network and a second long short-term memory network; ("LSTMr and LSTMi represent two traditional LSTMs of real part and imaginary part," Pg. 2, Section 2.2.)
and the long short-term memory network is configured to perform operations comprising: processing, through the first long short-term memory network, a real part and an imaginary part in an encoding result outputted by the last layer of complex encoder, to obtain a fifth output and a sixth output, ("Frr = LSTMr(Xr); Fir = LSTMr(Xi) (2)" Pg. 2, Section 2.2.)
and processing, through the second long short-term memory network, a real part and an imaginary part of an encoding result outputted by the last layer of complex encoder, to obtain a seventh output and an eighth output; ("Fri = LSTMi(Xr); Fii = LSTMi(Xi) (3)" Pg. 2, Section 2.2.)
performing a complex multiplication operation on the fifth output, the sixth output, the seventh output, and the eighth output based on a complex multiplication rule, to obtain a second operation result in the complex number domain, ("Fout = (Frr - Fii) + j(Fri + Fir) (4)" Pg. 2, Section 2.2.)
wherein the second operation result comprises a real part and an imaginary part; ("complex LSTM output Fout" Pg. 2, Section 2.2.)
and inputting the real part and the imaginary part of the second operation result to a first layer of complex decoder in the decoding network in the complex number domain. (Fig. 1 shows the LSTM output is the input to a complex decoder. )
Regarding claim 7, Hu discloses: 7. The method according to claim 6, wherein the complex deconvolution layer comprises a second real part convolution kernel and a second imaginary part convolution kernel; and the complex decoder is configured to perform operations comprising: convolving a received real part and a received imaginary part through the second real part convolution kernel, to obtain a ninth output and a tenth output,
and convolving the received real part and the received imaginary part through the second imaginary part convolution kernel, to obtain an eleventh output and a twelfth output;
performing a complex multiplication operation on the ninth output, the tenth output, the eleventh output, and the twelfth output based on a complex multiplication rule, to obtain a third operation result in the complex number domain;
sequentially processing the third operation result through the batch normalization layer and the activation unit layer in the complex decoder, to obtain a decoding result in the complex number domain,
wherein the decoding result comprises a real part and an imaginary part; ("Subsequently, the decoder reconstructs the low resolution features to the original size of the input, leading the encoder-decoder structure to a symmetric design." Pg. 2, Section 2.1. - Hu discloses that the encoder-decoder structure is symmetric. The same structure is disclosed for the encoder in claim 5.)
and when there is a next layer of complex decoder, inputting the real part and the imaginary part in the decoding result to the next layer of complex decoder. (Fig. 1 shows that the output from the first decoder is the input to the next decoder.)
Regarding claim 8, Hu discloses: 8. The method according to claim 4, wherein the deep complex convolutional recurrent network further comprises a short-time Fourier transform layer and an inverse short-time Fourier transform layer; (Disclosed by Hu as shown in claim 2.)
and the noise reduction model is obtained through training in steps comprising: obtaining a speech sample set, ("In our experiments, we first evaluated the proposed models as well as several baselines on a dataset simulated on WSJ0" Pg. 3, Section 3.1.)
wherein the speech sample set comprises a sample of the noisy speech, and the sample of the noisy speech is obtained by synthesizing a pure speech sample and noise; ("The speech-noise mixtures in training and validation are generated by randomly selecting utterances from the speech set and the noise set and mixing them at random SNR between -5 dB and 20 dB." Pg. 3, Section 3.1.)
and using the sample of the noisy speech as an input of the short-time Fourier transform layer, ("Input Noisy Wav" Fig. 1)
performing subband division on a spectrum outputted by the short-time Fourier transform layer, (not explicitly disclosed)
using, as an input of the encoding network, a subband spectrum obtained after the subband division, (Fig. 1 shows the spectrum goes into the Complex Encoder )
performing subband restoration on a spectrum outputted by the decoding network, (not explicitly disclosed)
using, as an input of the short-time inverse Fourier transform layer, a spectrum obtained after the subband restoration, (Fig. 1 shows the Complex Decoder goes into the inverse Fourie transform. )
using the pure speech sample as an output target of the short-time Fourier inverse transform layer, ("The input and output are the noisy and estimated clean spectrogram with MSA, respectively." Pg. 3, section 3.2. – Hu discloses the Frequency transform of the pure speech is the target before the inverse Fourier transform)
and training the deep complex convolutional recurrent network by using a machine learning method, to obtain the noise reduction model. ("We use Pytorch to train the models" Pg. 3, Section 3.2.)
Hu does not disclose subband division and restoration, or using clean speech in the time domain as the training target.
Yu discloses: using the sample of the noisy speech as an input of the short-time Fourier transform layer, performing subband division on a spectrum outputted by the short-time Fourier transform layer, using, as an input of the encoding network, a subband spectrum obtained after the subband division, ("In some embodiments, spectrum analyzer 106 may employ a short time Fourier transform (STFT) to generate a feature matrix representing the audio sequence." Col 5 para 2 )
performing subband restoration on a spectrum outputted by the decoding network, using, as an input of the short-time inverse Fourier transform layer, a spectrum obtained after the subband restoration, (See Fig. 1A, “FULLY CONCATENATED INPUT”; see also "In one embodiment, the time-domain synthesizer 112 uses an overlap-add algorithm to convert the feature matrix into the time domain.” Col 5; 52-54 – an overlap-add algorithm includes an inverse Fourier transform. )
using the pure speech sample as an output target of the short-time Fourier inverse transform layer, (“Each DNN 108A-108D is trained using a mapping of reverberant audio frames and known, clean audio frames associated with a given sub-band.” Col 5; 39-42 )
Rationale for combination of sub-band processing as provided for Claim 1.
Hu and Yu are considered analogous art to the claimed invention because they disclose noise suppression methods. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hu with clean speech in the time domain as a training target, as disclosed by Yu. Doing so would have been beneficial to ensure that the speech was not distorted in the inverse Fourier transform.
Regarding claim 9, Hu discloses: 9. The method according to claim 8, wherein the obtaining the first spectrum of the noisy speech in the complex number domain comprises: inputting the noisy speech to the short-time Fourier transform layer in the pre-trained noise reduction model, to obtain the first spectrum of the noisy speech in the complex number domain; ("Conv-STFT" Fig. 1- STFT stands for short-time Fourier transform. (ref pg. 1, section 1.1.))
and the synthesizing the target speech based on the second spectrum comprises: inputting the second spectrum to the inverse short-time Fourier transform layer in the noise reduction model, to obtain the target speech. (The second Conv-STFT shows data goes from a complex domain to a waveform, so this is clearly an inverse STFT.)
Regarding claim 10, Hu discloses: 10. The method according to claim 8, wherein the processing the first subband spectra based on the pre-trained noise reduction model to obtain the second subband spectra of the target speech in the noisy speech in the complex number domain comprises: inputting the first subband spectra to the encoding network in the pre-trained noise reduction model, (Fig. 2, “Input Complex spectrum”)
and using, as the second subband spectra of the target speech in the noisy speech in the complex number domain, the spectrum outputted by the decoding network in the noise reduction model. (Fig. 1 shows the spectrum is processed by the noise reduction model then, the Complex Decoder output spectrum is the “second” spectrum which is converted back to Time domain for clean speech)
Hu does not disclose sub-band division or restoration.
Yu discloses: 10. The method according to claim 8, wherein the processing the first subband spectrum based on a pre-trained noise reduction model to obtain a second subband spectrum of a target speech in the noisy speech in the complex number domain comprises: inputting the first subband spectrum to the encoding network in the pre-trained noise reduction model, (Fig. 1A shows SUB-BAND 1 is input to DNN 1 108A)
and using, as the second subband spectrum of the target speech in the noisy speech in the complex number domain, the spectrum outputted by the decoding network in the noise reduction model.
(See Fig. 1A OUTPUT 1)
Rationale for combination as provided for Claim 1.
Claim 12 is an apparatus claim with limitations corresponding to the limitations of Claim 1 and is rejected under similar rationale. Additionally, a memory and a processor of the Claim is taught by Hu (For example, Pg. 3, Section 3.2. discloses the models are trained with Pytorch which would require a computer to run. One of ordinary skill in the art would understand that this includes a memory and processor. See also “Intel i5-8250U PC” pg. 4 para 7)
Claim 13 is an apparatus claim with limitations corresponding to the limitations of Claim 2 and is rejected under similar rationale.
Claim 14 is an apparatus claim with limitations corresponding to the limitations of Claim 3 and is rejected under similar rationale.
Claim 15 is an apparatus claim with limitations corresponding to the limitations of Claim 4 and is rejected under similar rationale.
Claim 16 is an apparatus claim with limitations corresponding to the limitations of Claim 5 and is rejected under similar rationale.
Claim 17 is an apparatus claim with limitations corresponding to the limitations of Claim 6 and is rejected under similar rationale.
Claim 18 is an apparatus claim with limitations corresponding to the limitations of Claim 7 and is rejected under similar rationale.
Claim 19 is an apparatus claim with limitations corresponding to the limitations of Claim 8 and is rejected under similar rationale.
Claim 20 is a non-transitory computer-readable medium claim with limitations corresponding to the limitations of Claim 1 and is rejected under similar rationale. Additionally, “computer-readable medium” of the Claim are taught by Hu (For example, Pg. 3, Section 3.2. discloses the models are trained with Pytorch which would require a computer including a computer-readable medium to run.)
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hu in view of Yu, in further view of Ramakrishnan et al. ("Efficient post-processing techniques for speech enhancement").
Regarding claim 11, Hu does not disclose the additional limitations. Neither does Yu.
Ramakrishnan discloses: 11. The method according to claim 1, wherein after the synthesizing the target speech, the method further comprises: filtering the target speech based on a post-filtering algorithm to obtain the enhanced target speech. ("After performing the above two steps, when the signal is transformed in to the time domain, a small amount of residual noise was observed. The noise spreads all through in the time domain. We propose to further denoise the signal using a modified soft-threshold operator in the time domain [9], [10] ." Pg. 3, Section C.)
Hu, Yu, and Ramakrishnan are considered analogous art to the claimed invention because the disclose methods of noise removal. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Hu in view of Yu with additional filtering in the time domain. Doing so would have been beneficial to make the sound more pleasing to a human listener. (Ramakrishnan Pg. 1, Section I.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JON C MEIS whose telephone number is (703)756-1566. The examiner can normally be reached Monday - Thursday, 8:30 am - 5:30 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached on 571-272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JON CHRISTOPHER MEIS/Examiner, Art Unit 2654
/HAI PHAN/Supervisory Patent Examiner, Art Unit 2654