DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status - 35 USC § 101
The examiner would like to note that all claims have been deemed to contain eligible subject matter under 35 U.S.C. 101 because the steps of extracting features from audio, transforming those features into global representations, generating a masking matrix based on the two sets of features, and masking noise based on the masking matrix are not deemed to fall under a judicial exception category. These steps cannot be reasonably performed in the mind with the assistance of pen and/or paper, nor are they performed using claimed generic mathematical operations.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-2, 4, 6-8, 10, 12-14, 16, 18 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Mesgarani et al. (US-11373672-B2), hereinafter Mesgarani.
Regarding claim 1, Mesgarani discloses: a voice noise reduction method ([Col. 8, Line 41] a noise cancellation device), comprising:
determining a to-be-denoised voice spectrum of a to-be-denoised voice signal ([Fig. 2, 210, FFT output], [A representation of audio consisting of a mixture of speakers tracking frequency as a function of time indicates the representation to be a to-be-denoised voice spectrum]);
performing feature extraction on the to-be-denoised voice spectrum to obtain a local voice spectral feature of the to-be-denoised voice spectrum ([Col. 12, Lines 33-35] acquiring mixture speech signal with a (single) sensor, processing the signal to estimate a set of features);
determining a global voice spectral feature of the to-be-denoised voice spectrum according to the local voice spectral feature of the to-be-denoised voice spectrum ([Col. 12, Lines 35-40] processing the features by a deep neural network (DNN) to assign a descriptor for each element in the signal, estimating the attribution of each element according to the physical properties of the descriptors, [Processing features to determine an attribution of the features/elements indicates the attribution is a global spectral feature based on extracted local spectral features]);
determining a masking matrix of an original voice signal in the to-be-denoised voice signal according to the local voice spectral feature and the global voice spectral feature ([Col. 34, Lines 5-10] performing 1540 neural-network-based processing on the respective weighted sums of the learnable overcomplete basis of signals to derive a plurality of mask matrices corresponding to different groups of the multiple sound sources, [Generating masking matrices based on weights, i.e. attributions, of signals/elements (global features), wherein those attributions are based on local features (original extracted features), indicates a mask matrix generated for an original voice, i.e. any voice within the mixture, based on local and global spectral features]); and,
determining the original voice signal according to the to-be-denoised voice spectrum and the masking matrix ([Col. 17, Lines 15-20] estimating a plurality of reconstructed sounds signals from the derived plurality of mask matrices corresponding to the different groups of the multiple sound sources, [Estimating sound signals corresponding to different groups of the multiple sound sources using a masking matrix, wherein extraction of one speech source is disclosed (see [Fig. 30, 3060]), indicates determining an original voice signal according to the to-be-denoised voice spectrum and the masking matrix, wherein any of the multiple signals of Mesgarani could represent an original voice signal]).
Regarding claim 2, Mesgarani discloses: the method of claim 1.
Mesgarani further discloses:
wherein the to-be-denoised voice spectrum and the local voice spectral feature have a same dimension in a time domain and a same dimension in a frequency domain ([Col. 17, Lines 45-50] projecting the time-frequency mixture signal into an embedding space comprising multiple embedded time-frequency bins, [Col. 27, Lines 13-14] only the embeddings of the most salient T-F bins may be included [A mixture signal tracks to a to-be-denoised voice spectrum and embeddings of time-frequency bins correspond to spectral features. Considering both sets of data are in the time-frequency domain, it is indicated that the dimensions, i.e. time and frequency, will be the same as the same bins are used for both sets of data. Further, in view of the definition of a dimension in [0032] of the spec, a dimension can be reasonably assumed to represent a bin or other sort of division of a larger spectrum. Considering this, it is unclear to the examiner how a spectrum of features and a singular feature can ever contain the same dimensionality, i.e. number of bins, unless the spectrum is consisting of one feature, in which case the dimensionalities will inherently be the same]).
Regarding claim 4, Mesgarani discloses: the method of claim 1.
Mesgarani further discloses:
wherein performing the feature extraction on the to-be- denoised voice spectrum to obtain the local voice spectral feature of the to-be-denoised voice spectrum comprises:
performing the feature extraction on the to-be-denoised voice spectrum through a convolutional layer to obtain an initial spectral feature ([Fig. 18A, Input Mixture passed into “Conv1d”], [Col. 4, Lines 35-40] estimating the respective weighted sums of the learnable overcomplete basis of signals using a gated 1-D convolution layer [Estimating weighted sums of signals through a convolution layer indicates each individual weight is representative of a feature, i.e. a contribution to the overall signal; wherein contribution can be reasonably construed to represent a feature]);
activating the initial spectral feature through an activation layer ([Fig. 18A, 1810 “ReLu”], [Col. 4, Lines 45-50] inputting the respective weighted sums of the learnable overcomplete basis of signals to a deep long-short term network (LSTM) followed by a fully connected layer with Softmax activation function [ReLu is a well-known activation function]);
performing a batch normalization operation on the activated initial spectral feature through a normalization layer ([Fig. 18A, 1820 “LayerNorm”], [Col. 18, Lines 35-40] following the S-conv operation, with a non-linear activation function (PReLU) and a normalization process added between each two convolution operations [Following activation with normalization indicates the normalization is performed on an activated initial spectral feature]); and,
combining an output result of the normalization layer and the to-be-denoised voice spectrum through a residual connection structure to obtain the local voice spectral feature of the to-be-denoised voice spectrum ([Fig. 18A, Convolving/Multiplying a Mixture with Masks, end of section 1820], [Wherein the masks are normalized through “LayerNorm”, indicating the convolution operation performed with the masks and mixture is a combining an output from a normalization layer, i.e. though undergoing additional convolutions, the information will still be normalized, and a to-be-denoised voice, i.e. mixture signal, through a residual connection structure, i.e. a convolution/multiplication]).
Regarding claim 6, Mesgarani discloses: the method of claim 1.
Mesgarani further discloses:
wherein determining the masking matrix of the original voice signal in the to-be-denoised voice signal according to the local voice spectral feature and the global voice spectral feature comprises:
combining the local voice spectral feature and the global voice spectral feature to obtain a combination result ([Col. 31, Lines 10-15] The time-domain signal can then be represented by a weight matrix, reformulating the waveform separation problem into estimating the weight matrices that correspond to different sources given the weight matrix of the mixture… The synthesis of the sources signals is done by calculating the weighted sum of the bas[i]s with the estimated source weight matrices, [Determining a weighted sum, wherein the weights are dependent upon features, indicates the weighted sum to be a combination of local, i.e. basis feature signals representing the individual weights, and global features, i.e. each distinct individual estimated signal weight summed together]); and,
performing a convolution operation on the combination result to obtain the masking matrix of the original voice signal in the to-be-denoised voice signal ([Col. 32, Lines 18-23] the separation module 1420 is configured the estimation of the source masks is done with a deep LSTM network to model the time dependencies across the K segments, followed by a fully-connected layer with Softmax activation function for mask generation, [Col. 32, Lines 60-65] For each segment, this operation can also be formulated as a linear deconvolutional operation (also known as transposed convolution) [In view of the plurality of signals in the mix, see [Fig. 14, 1430], any of which can represent an original voice signal]).
Regarding claim 7, Mesgarani discloses: an electronic device ([Col. 9, Lines 45-46] which may be implemented as a hearing device to facilitate hearing for a person 104), comprising:
at least one processor ([Col. 32, Lines 3-4] a dedicated unit/processor may be configured to perform the pre-processing operations); and
a memory communicatively connected to the at least one processor ([Fig. 31, Storage 3114]);
wherein the memory stores instructions executable by the at least one processor ([Col. 53, Lines 40-45] The storage device 3114 may thus include a computer program product that when executed on the computing/controller-based device 3110 causes the computing-based device to perform operations to facilitate the implementation of procedures and operations described herein [Wherein connection between memory and processor, see Fig. 31, indicates the memory stores instructions executable by the processor]), and the instructions, when executed by the at least one processor, cause the at least on processor to perform:
determining a to-be-denoised voice spectrum of a to-be-denoised voice signal ([Fig. 2, 210, FFT output], [A representation of audio consisting of a mixture of speakers tracking frequency as a function of time indicates the representation to be a to-be-denoised voice spectrum]);
performing feature extraction on the to-be-denoised voice spectrum to obtain a local voice spectral feature of the to-be-denoised voice spectrum ([Col. 12, Lines 33-35] acquiring mixture speech signal with a (single) sensor, processing the signal to estimate a set of features);
determining a global voice spectral feature of the to-be-denoised voice spectrum according to the local voice spectral feature of the to-be-denoised voice spectrum ([Col. 12, Lines 35-40] processing the features by a deep neural network (DNN) to assign a descriptor for each element in the signal, estimating the attribution of each element according to the physical properties of the descriptors, [Processing features to determine an attribution of the features/elements indicates the attribution is a global spectral feature based on extracted local spectral features]);
determining a masking matrix of an original voice signal in the to-be-denoised voice signal according to the local voice spectral feature and the global voice spectral feature ([Col. 34, Lines 5-10] performing 1540 neural-network-based processing on the respective weighted sums of the learnable overcomplete basis of signals to derive a plurality of mask matrices corresponding to different groups of the multiple sound sources, [Generating masking matrices based on weights, i.e. attributions, of signals/elements (global features), wherein those attributions are based on local features (original extracted features), indicates a mask matrix generated for an original voice, i.e. any voice within the mixture, based on local and global spectral features]); and,
determining the original voice signal according to the to-be-denoised voice spectrum and the masking matrix ([Col. 17, Lines 15-20] estimating a plurality of reconstructed sounds signals from the derived plurality of mask matrices corresponding to the different groups of the multiple sound sources, [Estimating sound signals corresponding to different groups of the multiple sound sources using a masking matrix, wherein extraction of one speech source is disclosed (see [Fig. 30, 3060]), indicates determining an original voice signal according to the to-be-denoised voice spectrum and the masking matrix, wherein any of the multiple signals of Mesgarani could represent an original voice signal]).
Regarding claim 8, Mesgarani discloses: the electronic device of claim 7.
Mesgarani further discloses:
wherein the to-be-denoised voice spectrum and the local voice spectral feature have a same dimension in a time domain and a same dimension in a frequency domain ([Col. 17, Lines 45-50] projecting the time-frequency mixture signal into an embedding space comprising multiple embedded time-frequency bins, [Col. 27, Lines 13-14] only the embeddings of the most salient T-F bins may be included [A mixture signal tracks to a to-be-denoised voice spectrum and embeddings of time-frequency bins correspond to spectral features. Considering both sets of data are in the time-frequency domain, it is indicated that the dimensions, i.e. time and frequency, will be the same as the same bins are used for both sets of data. Further, in view of the definition of a dimension in [0032] of the spec, a dimension can be reasonably assumed to represent a bin or other sort of division of a larger spectrum. Considering this, it is unclear to the examiner how a spectrum of features and a singular feature can ever contain the same dimensionality, i.e. number of bins, unless the spectrum is consisting of one feature, in which case the dimensionalities will inherently be the same]).
Regarding claim 10, Mesgarani discloses: the electronic device of claim 7.
Mesgarani further discloses:
wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform performing the feature extraction on the to-be-denoised voice spectrum to obtain the local voice spectral feature of the to-be-denoised voice spectrum in the following way:
performing the feature extraction on the to-be-denoised voice spectrum through a convolutional layer to obtain an initial spectral feature ([Fig. 18A, Input Mixture passed into “Conv1d”], [Col. 4, Lines 35-40] estimating the respective weighted sums of the learnable overcomplete basis of signals using a gated 1-D convolution layer [Estimating weighted sums of signals through a convolution layer indicates each individual weight is representative of a feature, i.e. a contribution to the overall signal; wherein contribution can be reasonably construed to represent a feature]);
activating the initial spectral feature through an activation layer ([Fig. 18A, 1810 “ReLu”], [Col. 4, Lines 45-50] inputting the respective weighted sums of the learnable overcomplete basis of signals to a deep long-short term network (LSTM) followed by a fully connected layer with Softmax activation function [ReLu is a well-known activation function]);
performing a batch normalization operation on the activated initial spectral feature through a normalization layer ([Fig. 18A, 1820 “LayerNorm”], [Col. 18, Lines 35-40] following the S-conv operation, with a non-linear activation function (PReLU) and a normalization process added between each two convolution operations [Following activation with normalization indicates the normalization is performed on an activated initial spectral feature]); and,
combining an output result of the normalization layer and the to-be-denoised voice spectrum through a residual connection structure to obtain the local voice spectral feature of the to-be-denoised voice spectrum ([Fig. 18A, Convolving/Multiplying a Mixture with Masks, end of section 1820], [Wherein the masks are normalized through “LayerNorm”, indicating the convolution operation performed with the masks and mixture is a combining an output from a normalization layer, i.e. though undergoing additional convolutions, the information will still be normalized, and a to-be-denoised voice, i.e. mixture signal, through a residual connection structure, i.e. a convolution/multiplication]).
Regarding claim 12, Mesgarani discloses: the electronic device of claim 7.
Mesgarani further discloses:
wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform determining the masking matrix of the original voice signal in the to-be-denoised voice signal according to the local voice spectral feature and the global voice spectral feature in the following way:
combining the local voice spectral feature and the global voice spectral feature to obtain a combination result ([Col. 31, Lines 10-15] The time-domain signal can then be represented by a weight matrix, reformulating the waveform separation problem into estimating the weight matrices that correspond to different sources given the weight matrix of the mixture… The synthesis of the sources signals is done by calculating the weighted sum of the bas[i]s with the estimated source weight matrices, [Determining a weighted sum, wherein the weights are dependent upon features, indicates the weighted sum to be a combination of local, i.e. basis feature signals representing the individual weights, and global features, i.e. each distinct individual estimated signal weight summed together]); and,
performing a convolution operation on the combination result to obtain the masking matrix of the original voice signal in the to-be-denoised voice signal ([Col. 32, Lines 18-23] the separation module 1420 is configured the estimation of the source masks is done with a deep LSTM network to model the time dependencies across the K segments, followed by a fully-connected layer with Softmax activation function for mask generation, [Col. 32, Lines 60-65] For each segment, this operation can also be formulated as a linear deconvolutional operation (also known as transposed convolution) [In view of the plurality of signals in the mix, see [Fig. 14, 1430], any of which can represent an original voice signal]).
Regarding claim 13, Mesgarani discloses: a non-transitory computer-readable storage medium storing computer instructions ([Col. 54, Lines 10-15] any suitable computer readable media can be used for storing instructions for performing the processes/operations/procedures described herein. For example, in some embodiments computer readable media can be transitory or non-transitory), wherein the computer instructions are configured to cause a computer to perform:
determining a to-be-denoised voice spectrum of a to-be-denoised voice signal ([Fig. 2, 210, FFT output], [A representation of audio consisting of a mixture of speakers tracking frequency as a function of time indicates the representation to be a to-be-denoised voice spectrum]);
performing feature extraction on the to-be-denoised voice spectrum to obtain a local voice spectral feature of the to-be-denoised voice spectrum ([Col. 12, Lines 33-35] acquiring mixture speech signal with a (single) sensor, processing the signal to estimate a set of features);
determining a global voice spectral feature of the to-be-denoised voice spectrum according to the local voice spectral feature of the to-be-denoised voice spectrum ([Col. 12, Lines 35-40] processing the features by a deep neural network (DNN) to assign a descriptor for each element in the signal, estimating the attribution of each element according to the physical properties of the descriptors, [Processing features to determine an attribution of the features/elements indicates the attribution is a global spectral feature based on extracted local spectral features]);
determining a masking matrix of an original voice signal in the to-be-denoised voice signal according to the local voice spectral feature and the global voice spectral feature ([Col. 34, Lines 5-10] performing 1540 neural-network-based processing on the respective weighted sums of the learnable overcomplete basis of signals to derive a plurality of mask matrices corresponding to different groups of the multiple sound sources, [Generating masking matrices based on weights, i.e. attributions, of signals/elements (global features), wherein those attributions are based on local features (original extracted features), indicates a mask matrix generated for an original voice, i.e. any voice within the mixture, based on local and global spectral features]); and,
determining the original voice signal according to the to-be-denoised voice spectrum and the masking matrix ([Col. 17, Lines 15-20] estimating a plurality of reconstructed sounds signals from the derived plurality of mask matrices corresponding to the different groups of the multiple sound sources, [Estimating sound signals corresponding to different groups of the multiple sound sources using a masking matrix, wherein extraction of one speech source is disclosed (see [Fig. 30, 3060]), indicates determining an original voice signal according to the to-be-denoised voice spectrum and the masking matrix, wherein any of the multiple signals of Mesgarani could represent an original voice signal]).
Regarding claim 14, Mesgarani discloses: the non-transitory computer-readable storage medium of claim 13.
Mesgarani further discloses:
wherein the to-be-denoised voice spectrum and the local voice spectral feature have a same dimension in a time domain and a same dimension in a frequency domain ([Col. 17, Lines 45-50] projecting the time-frequency mixture signal into an embedding space comprising multiple embedded time-frequency bins, [Col. 27, Lines 13-14] only the embeddings of the most salient T-F bins may be included [A mixture signal tracks to a to-be-denoised voice spectrum and embeddings of time-frequency bins correspond to spectral features. Considering both sets of data are in the time-frequency domain, it is indicated that the dimensions, i.e. time and frequency, will be the same as the same bins are used for both sets of data. Further, in view of the definition of a dimension in [0032] of the spec, a dimension can be reasonably assumed to represent a bin or other sort of division of a larger spectrum. Considering this, it is unclear to the examiner how a spectrum of features and a singular feature can ever contain the same dimensionality, i.e. number of bins, unless the spectrum is consisting of one feature, in which case the dimensionalities will inherently be the same]).
Regarding claim 16, Mesgarani discloses: the non-transitory computer-readable storage medium of claim 13.
Mesgarani further discloses:
wherein the computer instructions are configured to cause a computer to perform performing the feature extraction on the to-be-denoised voice spectrum to obtain the local voice spectral feature of the to-be-denoised voice spectrum in the following way:
performing the feature extraction on the to-be-denoised voice spectrum through a convolutional layer to obtain an initial spectral feature ([Fig. 18A, Input Mixture passed into “Conv1d”], [Col. 4, Lines 35-40] estimating the respective weighted sums of the learnable overcomplete basis of signals using a gated 1-D convolution layer [Estimating weighted sums of signals through a convolution layer indicates each individual weight is representative of a feature, i.e. a contribution to the overall signal; wherein contribution can be reasonably construed to represent a feature]);
activating the initial spectral feature through an activation layer ([Fig. 18A, 1810 “ReLu”], [Col. 4, Lines 45-50] inputting the respective weighted sums of the learnable overcomplete basis of signals to a deep long-short term network (LSTM) followed by a fully connected layer with Softmax activation function [ReLu is a well-known activation function]);
performing a batch normalization operation on the activated initial spectral feature through a normalization layer ([Fig. 18A, 1820 “LayerNorm”], [Col. 18, Lines 35-40] following the S-conv operation, with a non-linear activation function (PReLU) and a normalization process added between each two convolution operations [Following activation with normalization indicates the normalization is performed on an activated initial spectral feature]); and,
combining an output result of the normalization layer and the to-be-denoised voice spectrum through a residual connection structure to obtain the local voice spectral feature of the to-be-denoised voice spectrum ([Fig. 18A, Convolving/Multiplying a Mixture with Masks, end of section 1820], [Wherein the masks are normalized through “LayerNorm”, indicating the convolution operation performed with the masks and mixture is a combining an output from a normalization layer, i.e. though undergoing additional convolutions, the information will still be normalized, and a to-be-denoised voice, i.e. mixture signal, through a residual connection structure, i.e. a convolution/multiplication]).
Regarding claim 18, Mesgarani discloses: the non-transitory computer-readable storage medium of claim 13.
Mesgarani further discloses:
wherein the computer instructions are configured to cause a computer to perform determining the masking matrix of the original voice signal in the to-be-denoised voice signal according to the local voice spectral feature and the global voice spectral feature in the following way:
combining the local voice spectral feature and the global voice spectral feature to obtain a combination result ([Col. 31, Lines 10-15] The time-domain signal can then be represented by a weight matrix, reformulating the waveform separation problem into estimating the weight matrices that correspond to different sources given the weight matrix of the mixture… The synthesis of the sources signals is done by calculating the weighted sum of the bas[i]s with the estimated source weight matrices, [Determining a weighted sum, wherein the weights are dependent upon features, indicates the weighted sum to be a combination of local, i.e. basis feature signals representing the individual weights, and global features, i.e. each distinct individual estimated signal weight summed together]); and,
performing a convolution operation on the combination result to obtain the masking matrix of the original voice signal in the to-be-denoised voice signal ([Col. 32, Lines 18-23] the separation module 1420 is configured the estimation of the source masks is done with a deep LSTM network to model the time dependencies across the K segments, followed by a fully-connected layer with Softmax activation function for mask generation, [Col. 32, Lines 60-65] For each segment, this operation can also be formulated as a linear deconvolutional operation (also known as transposed convolution) [In view of the plurality of signals in the mix, see [Fig. 14, 1430], any of which can represent an original voice signal]).
Claim(s) 3, 9, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mesgarani in view of Taghizadeh et al. (US-20220150661-A1), hereinafter Taghizadeh.
Regarding claim 3, Mesgarani discloses: the method of claim 1.
Mesgarani does not disclose:
wherein determining the global voice spectral feature of the to-be-denoised voice spectrum according to the local voice spectral feature of the to-be- denoised voice spectrum comprises:
combining a channel dimension of the local voice spectral feature and a time dimension of the local voice spectral feature to obtain first combined data;
performing a self-attention operation on the first combined data in a frequency dimension through a frequency-axis attention mechanism layer to obtain a self-attention operation result of the frequency dimension;
combining a channel dimension of the self-attention operation result of the frequency dimension and a frequency dimension of the self-attention operation result of the frequency dimension to obtain second combined data; and,
performing the self-attention operation on the second combined data in a time dimension through a time-axis attention mechanism layer to obtain the global voice spectral feature of the to-be-denoised voice spectrum.
Taghizadeh discloses:
wherein determining the global voice spectral feature of the to-be-denoised voice spectrum according to the local voice spectral feature of the to-be- denoised voice spectrum comprises:
combining a channel dimension of the local voice spectral feature and a time dimension of the local voice spectral feature to obtain first combined data ([0038] extract a respective feature set having at least one feature for each audio frame of each of the plurality of audio frames, [0039] the plurality of features of the different feature sets define a three-dimensional feature array 201. In the embodiment shown in FIG. 2, the three-dimensional feature array 201 has a dimension corresponding to time represented by a time stamp t… and a dimension corresponding to the audio channel c, [0057] two-dimensional sub-array of shape [time×channels] [A 3D feature matrix with time and channel dimensions indicates a combination of those dimension to form the matrix; wherein the extracted features of Taghizadeh are representative of local features and are interchangeable with those previously defined in Mesgarani]);
performing a self-attention operation on the first combined data in a frequency dimension through a frequency-axis attention mechanism layer to obtain a self-attention operation result of the frequency dimension ([Fig. 2, 203c], [0040] process the three-dimensional feature array 201 using a neural network (herein also referred to as “rotational self-attention block”), wherein the neural network comprises a self-attention layer configured to process a plurality of two-dimensional sub-arrays 203a-c of the three-dimensional feature array 201… a second two-dimensional sub-array 203c for a respective given frequency f [Referring to the 2D matrices defined in Fig. 2, it is seen that 203c time and channel dimensions are for a given frequency, indicating self-attention on first combined (time + channel) data through a frequency axis]);
combining a channel dimension of the self-attention operation result of the frequency dimension and a frequency dimension of the self-attention operation result of the frequency dimension to obtain second combined data ([0057] two-dimensional sub-array of shape [channels×frequencies], [Fig. 2, 205], [The summation of a plurality of time/channel matrices, wherein those matrices have had self-attention performed, for various frequencies indicates the combination of a channel dimension of the self-attention operation, i.e. 203a-c’, of the frequency dimension, i.e. 203a and/or 203b (matrices with frequency dimension), and a frequency dimension of the self-attention operation result of the frequency dimension, i.e. 203a’ and/or 203b’]); and,
performing the self-attention operation on the second combined data in a time dimension through a time-axis attention mechanism layer to obtain the global voice spectral feature of the to-be-denoised voice spectrum ([Fig. 2, 203a’, 203b’], [0060] self-attention block is applied three times, [0061] the output of the last rotational self-attention block of the processing blocks 403, 503 is a matrix D of shape [time×frequencies×channels], [In view of the first self-attention being performed to generate 2D matrices 203a-c’, indicating an additional self-attention operation for resulting in a 3D matrix, wherein one of the dimension is time, further indicating a self-attention operation performed in a time dimension through a time-axis mechanism layer, i.e. that used for 203a, wherein the combined data is consisting of at least channel and frequency information, as required for resulting in the dimensionality of the claimed matrix 205]).
Mesgarani and Taghizadeh are considered analogous art within audio enhancement through masking. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Mesgarani to incorporate the teachings of Taghizadeh, because of the novel way to use a self-attention mechanism which represents every time-frequency bin of a multi-channel audio signal in every channel that has awareness of the time-frequency bins of all other channels, enhancing multi-channel audio while simultaneously denoising (Taghizadeh, [0007]).
Regarding claim 9, Mesgarani discloses: the electronic device of claim 7.
Mesgarani does not disclose:
wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform determining the global voice spectral feature of the to-be-denoised voice spectrum according to the local voice spectral feature of the to-be-denoised voice spectrum in the following way:
combining a channel dimension of the local voice spectral feature and a time dimension of the local voice spectral feature to obtain first combined data;
performing a self-attention operation on the first combined data in a frequency dimension through a frequency-axis attention mechanism layer to obtain a self-attention operation result of the frequency dimension;
combining a channel dimension of the self-attention operation result of the frequency dimension and a frequency dimension of the self-attention operation result of the frequency dimension to obtain second combined data; and,
performing the self-attention operation on the second combined data in a time dimension through a time-axis attention mechanism layer to obtain the global voice spectral feature of the to-be-denoised voice spectrum.
Taghizadeh discloses:
wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform determining the global voice spectral feature of the to-be-denoised voice spectrum according to the local voice spectral feature of the to-be-denoised voice spectrum in the following way:
combining a channel dimension of the local voice spectral feature and a time dimension of the local voice spectral feature to obtain first combined data ([0038] extract a respective feature set having at least one feature for each audio frame of each of the plurality of audio frames, [0039] the plurality of features of the different feature sets define a three-dimensional feature array 201. In the embodiment shown in FIG. 2, the three-dimensional feature array 201 has a dimension corresponding to time represented by a time stamp t… and a dimension corresponding to the audio channel c, [0057] two-dimensional sub-array of shape [time×channels] [A 3D feature matrix with time and channel dimensions indicates a combination of those dimension to form the matrix; wherein the extracted features of Taghizadeh are representative of local features and are interchangeable with those previously defined in Mesgarani]);
performing a self-attention operation on the first combined data in a frequency dimension through a frequency-axis attention mechanism layer to obtain a self-attention operation result of the frequency dimension ([Fig. 2, 203c], [0040] process the three-dimensional feature array 201 using a neural network (herein also referred to as “rotational self-attention block”), wherein the neural network comprises a self-attention layer configured to process a plurality of two-dimensional sub-arrays 203a-c of the three-dimensional feature array 201… a second two-dimensional sub-array 203c for a respective given frequency f [Referring to the 2D matrices defined in Fig. 2, it is seen that 203c time and channel dimensions are for a given frequency, indicating self-attention on first combined (time + channel) data through a frequency axis]);
combining a channel dimension of the self-attention operation result of the frequency dimension and a frequency dimension of the self-attention operation result of the frequency dimension to obtain second combined data ([0057] two-dimensional sub-array of shape [channels×frequencies], [Fig. 2, 205], [The summation of a plurality of time/channel matrices, wherein those matrices have had self-attention performed, for various frequencies indicates the combination of a channel dimension of the self-attention operation, i.e. 203a-c’, of the frequency dimension, i.e. 203a and/or 203b (matrices with frequency dimension), and a frequency dimension of the self-attention operation result of the frequency dimension, i.e. 203a’ and/or 203b’]); and,
performing the self-attention operation on the second combined data in a time dimension through a time-axis attention mechanism layer to obtain the global voice spectral feature of the to-be-denoised voice spectrum ([Fig. 2, 203a’, 203b’], [0060] self-attention block is applied three times, [0061] the output of the last rotational self-attention block of the processing blocks 403, 503 is a matrix D of shape [time×frequencies×channels], [In view of the first self-attention being performed to generate 2D matrices 203a-c’, indicating an additional self-attention operation for resulting in a 3D matrix, wherein one of the dimension is time, further indicating a self-attention operation performed in a time dimension through a time-axis mechanism layer, i.e. that used for 203a, wherein the combined data is consisting of at least channel and frequency information, as required for resulting in the dimensionality of the claimed matrix 205]).
Mesgarani and Taghizadeh are considered analogous art within audio enhancement through masking. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Mesgarani to incorporate the teachings of Taghizadeh, because of the novel way to use a self-attention mechanism which represents every time-frequency bin of a multi-channel audio signal in every channel that has awareness of the time-frequency bins of all other channels, enhancing multi-channel audio while simultaneously denoising (Taghizadeh, [0007]).
Regarding claim 15, Mesgarani discloses: the non-transitory computer-readable storage medium of claim 13.
Mesgarani does not disclose:
wherein the computer instructions are configured to cause a computer to perform determining the global voice spectral feature of the to-be-denoised voice spectrum according to the local voice spectral feature of the to-be-denoised voice spectrum in the following way:
combining a channel dimension of the local voice spectral feature and a time dimension of the local voice spectral feature to obtain first combined data;
performing a self-attention operation on the first combined data in a frequency dimension through a frequency-axis attention mechanism layer to obtain a self-attention operation result of the frequency dimension;
combining a channel dimension of the self-attention operation result of the frequency dimension and a frequency dimension of the self-attention operation result of the frequency dimension to obtain second combined data; and,
performing the self-attention operation on the second combined data in a time dimension through a time-axis attention mechanism layer to obtain the global voice spectral feature of the to-be-denoised voice spectrum.
Taghizadeh discloses:
wherein the computer instructions are configured to cause a computer to perform determining the global voice spectral feature of the to-be-denoised voice spectrum according to the local voice spectral feature of the to-be-denoised voice spectrum in the following way:
combining a channel dimension of the local voice spectral feature and a time dimension of the local voice spectral feature to obtain first combined data ([0038] extract a respective feature set having at least one feature for each audio frame of each of the plurality of audio frames, [0039] the plurality of features of the different feature sets define a three-dimensional feature array 201. In the embodiment shown in FIG. 2, the three-dimensional feature array 201 has a dimension corresponding to time represented by a time stamp t… and a dimension corresponding to the audio channel c, [0057] two-dimensional sub-array of shape [time×channels] [A 3D feature matrix with time and channel dimensions indicates a combination of those dimension to form the matrix; wherein the extracted features of Taghizadeh are representative of local features and are interchangeable with those previously defined in Mesgarani]);
performing a self-attention operation on the first combined data in a frequency dimension through a frequency-axis attention mechanism layer to obtain a self-attention operation result of the frequency dimension ([Fig. 2, 203c], [0040] process the three-dimensional feature array 201 using a neural network (herein also referred to as “rotational self-attention block”), wherein the neural network comprises a self-attention layer configured to process a plurality of two-dimensional sub-arrays 203a-c of the three-dimensional feature array 201… a second two-dimensional sub-array 203c for a respective given frequency f [Referring to the 2D matrices defined in Fig. 2, it is seen that 203c time and channel dimensions are for a given frequency, indicating self-attention on first combined (time + channel) data through a frequency axis]);
combining a channel dimension of the self-attention operation result of the frequency dimension and a frequency dimension of the self-attention operation result of the frequency dimension to obtain second combined data ([0057] two-dimensional sub-array of shape [channels×frequencies], [Fig. 2, 205], [The summation of a plurality of time/channel matrices, wherein those matrices have had self-attention performed, for various frequencies indicates the combination of a channel dimension of the self-attention operation, i.e. 203a-c’, of the frequency dimension, i.e. 203a and/or 203b (matrices with frequency dimension), and a frequency dimension of the self-attention operation result of the frequency dimension, i.e. 203a’ and/or 203b’]); and,
performing the self-attention operation on the second combined data in a time dimension through a time-axis attention mechanism layer to obtain the global voice spectral feature of the to-be-denoised voice spectrum ([Fig. 2, 203a’, 203b’], [0060] self-attention block is applied three times, [0061] the output of the last rotational self-attention block of the processing blocks 403, 503 is a matrix D of shape [time×frequencies×channels], [In view of the first self-attention being performed to generate 2D matrices 203a-c’, indicating an additional self-attention operation for resulting in a 3D matrix, wherein one of the dimension is time, further indicating a self-attention operation performed in a time dimension through a time-axis mechanism layer, i.e. that used for 203a, wherein the combined data is consisting of at least channel and frequency information, as required for resulting in the dimensionality of the claimed matrix 205]).
Mesgarani and Taghizadeh are considered analogous art within audio enhancement through masking. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Mesgarani to incorporate the teachings of Taghizadeh, because of the novel way to use a self-attention mechanism which represents every time-frequency bin of a multi-channel audio signal in every channel that has awareness of the time-frequency bins of all other channels, enhancing multi-channel audio while simultaneously denoising (Taghizadeh, [0007]).
Claim(s) 5, 11, 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mesgarani in view of Watanabe et al. (US-20180261225-A1), hereinafter Watanabe.
Regarding claim 5, Mesgarani discloses: the method of claim 1.
Mesgarani further discloses:
wherein determining the to-be-denoised voice spectrum of the to-be-denoised voice signal comprises:
performing an inverse Fourier transform on the original voice spectrum to obtain the original voice signal in the to-be-denoised voice signal ([Fig. 2, 250, “iFFT”], [Col. 11, Lines 10-20] A spectrogram of this speaker is reconstructed from the neural recordings of the user. This reconstruction is then compared with the outputs of each of the DNNs using a normalized correlation analysis (as shown at stage 230) in order to select the appropriate spectrogram, which is then converted into an acoustic waveform [Selecting an appropriate spectrogram based on a plurality of spectrograms generated from a mixed audio signal indicates the selected spectrum represent an original voice; wherein the conversion into an acoustic waveform is performed using an iFFT]).
Mesgarani does not discloses:
wherein determining the to-be-denoised voice spectrum of the to-be-denoised voice signal comprises:
performing a short-time Fourier transform on the to-be-denoised voice signal to obtain the to-be-denoised voice spectrum of the to-be-denoised voice signal; and,
wherein determining the original voice signal according to the to-be-denoised voice spectrum and the masking matrix comprises:
determining an original voice spectrum of the original voice signal according to the to-be-denoised voice spectrum and the masking matrix.
Watanabe discloses:
wherein determining the to-be-denoised voice spectrum of the to-be-denoised voice signal comprises:
performing a short-time Fourier transform on the to-be-denoised voice signal to obtain the to-be-denoised voice spectrum of the to-be-denoised voice signal ([0102] The multichannel inputs 912 are introduced into the first feature extractor 920 that extracts signal features 921 from the outputs of individual microphones 910 based on short-term Fourier-transformation (STFT) algorithm); and,
wherein determining the original voice signal according to the to-be-denoised voice spectrum and the masking matrix comprises:
determining an original voice spectrum of the original voice signal according to the to-be-denoised voice spectrum and the masking matrix ([0102] The signal features 921 of the multichannel inputs 912 are processed using the mask estimation network 930, and the mask estimation network 930 estimates and generates masks 931 including speech relevant masks and noise-relevant masks, which are the time-frequency masks corresponding to respective channels [A time-frequency mask is reasonably assumed to represent a matrix, i.e. with both time and frequency components representing dimensions and/or a 1D matrix, i.e. vector. Further, determining “speech relevant masks” and “noise-relevant masks” indicates that the speech relevant masks (when applied) correspond to an original voice spectrum of the original signal, determined through an original spectrum (required for generating masks) and the masking matrix generated]).
Mesgarani and Watanabe are considered analogous art within speaker/speech analysis using neural networks (G10L25/30). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Mesgarani to incorporate the teachings of Watanabe, because of the novel way to incorporate neural beamforming mechanisms as a differentiable component to allow joint optimization of a multichannel speech enhancement system within an end-to-end system, improving the performance of automatic speech recognition (Watanabe, [0006]).
Regarding claim 11, Mesgarani discloses: the electronic device of claim 7.
Mesgarani further discloses:
wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform determining the to-be- denoised voice spectrum of the to-be-denoised voice signal in the following way:
performing an inverse Fourier transform on the original voice spectrum to obtain the original voice signal in the to-be-denoised voice signal ([Fig. 2, 250, “iFFT”], [Col. 11, Lines 10-20] A spectrogram of this speaker is reconstructed from the neural recordings of the user. This reconstruction is then compared with the outputs of each of the DNNs using a normalized correlation analysis (as shown at stage 230) in order to select the appropriate spectrogram, which is then converted into an acoustic waveform [Selecting an appropriate spectrogram based on a plurality of spectrograms generated from a mixed audio signal indicates the selected spectrum represent an original voice; wherein the conversion into an acoustic waveform is performed using an iFFT]).
Mesgarani does not discloses:
wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform determining the to-be- denoised voice s