DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The examiner has determined that a second non-final action is appropriate.
Second Non-Final Office Action
In their remarks to the first non-final office action, the applicant submits that nothing in table 1 of Dong et al. or the accompanying description discloses the signal pathway wherein the output of the residual network bock is received by the SE block. In order to make this teaching explicitly clear, the examiner has decided to introduce reference Hu et al. and issue a second non-final office action. The original claims submitted 28 May 2024 is addressed below.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1 and 2 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by “Multi-perspective Information Fusion Res2Net with Random Specmix for Fake Speech Detection”, hereinafter referred to as Dong et al.
Regarding claim 1, Dong et al. discloses a method comprising:
receiving a first speech waveform (“In this work, we use a random Specmix strategy to help
the model to locate the discriminative information and enhance the generalization of the model. For the training of deep neural networks, we always transform the raw audio from time domain into time-frequency domain.… At the same time, to avoid the conduction of Specmix on all the samples, inspired by [21], we randomly choose speech samples according to the hyperparameter 𝑝_ℎ𝑦𝑝𝑒𝑟 in advance to conduct Specmix operation,” Dong et al., p. 3, sec. 2.3. It is clear from this excerpt that the input to the encoder comprised speech samples.);
processing an input based on the first speech waveform using a trained machine learning model to generate a second speech waveform (“In this work, we propose multi-perspective information fusion Res2Net (MPIF-Res2Net) with random Specmix.…Specmix can increase the diversity of training data, thereby improving the generalization ability of the model,” Dong et al., p. 2, col. 1, 2nd para.), wherein the trained machine learning model comprises:
an encoder (Dong et al., p. 3, Table 1.), wherein the encoder comprises a plurality of encoder layers (Dong et al., p. 3, Table 1.), each encoder layer comprising:
receiving, by a residual network block of a respective encoder layer, as input one of: a first speech waveform or multi-scale feature maps outputted from a previous encoder layer of the plurality of encoder layers (“Res2Net [17] was proposed by Gao et al. to enhance its ability to capture multi-scale information by transferring information among channel groups,” Dong et al., p. 1, col. 1, last sentence. And, Dong et al., p. 2, col. 1, 2nd para.); and
generating, based on the input, multi-scale feature maps of the respective encoder layer (“Res2Net [17] was proposed by Gao et al. to enhance its ability to capture multi-scale information by transferring information among channel groups,” Dong et al., p. 1, right col., last sentence.).
Regarding claim 2, Dong et al. discloses the method of claim 1, wherein the residual network block is a Res2Net (“In this section, we introduce the structure of the proposed
MPIF-Res2Net, it reduces redundancy caused by learning single-perspective forgery information by integrating information from multiple perspectives,” Dong et al., sec. 2, first sentence.).
Claim(s) 9 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by US 20250104727, hereinafter referred to as Zhang et al.
Regarding claim 9, Zhang et al. discloses a non-transitory computer-readable medium comprising instructions (Zhang et al., para [0006]) that, responsive to execution by a processing device (Zhang et al., para [0006]), cause the processing device to perform operations comprising:
receiving a plurality of speech waveform pairs, wherein each speech waveform pair of the plurality of speech waveform pairs includes a first speech waveform and a second speech waveform (“An element of the cGAN is the similarity metric used by the discriminator 404. Unlike traditional GAN applications (which compare between the same type of features), the cGAN is cross-modal, so the cGAN needs to discriminate between different modalities, such as whether the enhanced T-F speech spectrogram matches the ultrasound Doppler spectrogram (e.g., whether they are a “real” or “fake” pair),” Zhang et al., para [0046]. See also fig. 6A, which shows pairs of speech waveform model inputs.);
training a machine learning model to generate a third speech waveform based on a first speech waveform of a respective speech waveform, wherein training the machine learning model (“FIG. 4 depicts an example of a conditional generative adversarial network (cGAN) used to train the ML model of FIGS. 3A-3B, in accordance with some embodiments,” Zhang et al., para [0013].) comprises:
inputting, into the machine learning model, the first speech waveform of the respective speech waveform (“In the example of FIG. 4, the cGAN is used to add a conditional goal to guide a generator (G) 120 to automatically learn a loss function which well approximates the goal. The generator 120 (which is represented as “G(S.sub.noise.sup.a,U.sup.s)”) takes the noisy speech amplitude spectrogram ultrasound sensing spectrogram U.sup.s as the input, wherein the generator G(.Math.) is trained to output amplitude-enhanced T-F spectrogram of the speech S.sub.out.sup.a, which not only minimizes the traditional amplitude MSE loss, but also tries to “fool” an adversarially trained discriminator 404…,” Zhang et al., para [0050].);
predicting, based on the inputted first speech waveform, the third speech waveform (Zhang et al., para [0050]);
inputting, into a loss function, a second speech waveform of the respective speech waveform and a third speech waveform (“As shown in FIG. 6A, the Triplet loss is used to train the cross-modal Siamese network. The triplet loss function accepts 3 inputs, i.e., an anchor input U.sup.s is compared to a positive input S.sub.gr.sup.a and a negative input S.sub.out.sup.a. It aims to minimize the distance between “real” pair U.sup.s and S.sub.gr.sup.a, and maximize the distance between “fake” pair U.sup.s and S.sub.out′.sup.a,” Zhang et al., para [0047].);
optimizing, based on a result of the loss function, the machine learning model, wherein the result of the loss function is calculated based on (i) a difference between the second speech waveform of the respective speech waveform and the third speech waveform (Zhang et al., para [0047]) and (ii) a difference between a magnitude spectrogram of the second speech waveform of the respective speech waveform and a magnitude spectrogram of the third speech waveform (Zhang et al., para [0047]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over “Multi-perspective Information Fusion Res2Net with Random Specmix for Fake Speech Detection”, hereinafter referred to as Dong et al., in view of “Squeeze-and-Excitation”, hereinafter referred to as Hu et al., and further in view of “Channel Interdependence Enhanced Speaker Embeddings for Far-Field Speaker Verification”, hereinafter referred to as Zhao-Mak.
Regarding claim 3, Dong et al. discloses the method of claim 1, but not wherein each encoder layer further comprising:
receiving, by a squeeze-and-excitation (SE) block of the respective encoder layer, an output of the residual network block of the respective encoder layer.
Hu et al. is cited to disclose receiving, by a squeeze-and-excitation (SE) block of the respective encoder layer, an output of the residual network block of the respective encoder layer (Hu et al., fig. 3. This figure shows that an output of the residual network block of the respective encoder layer feeds to a squeeze-and-excitation (SE) block of the respective encoder layer. The examiner notes that this figure of Hu et al. mirrors the applicant’s figure 5.). Hu et al. benefits Dong et al. by incorporating SE blocks to produce significant performance improvement for existing architectures at minimal additional computational cost (Hu et al., Abstract). Therefore, it would be obvious for one skilled in the art to combine the teachings of Dong et al. with those of Hu et al. to improve the speech detection architecture of Dong et al.
Neither Hu et al. nor Dong et al., though, does not disclose adaptively recalibrating, based on the received output of the residual network block of the respective encoder layer, the multi-scale feature maps.
Zhao-Mak is cited to disclose adaptively recalibrating, based on the received output of the residual network block of the respective encoder layer, the multi-scale feature maps (“In this block, the SE unit recalibrates the channel activations based on the multi-scale features obtained from the conv-ops in the Res2block,” Zhao-Mak, p. 2, highlighted passage.). Zhao-Mak benefits Dong et al. by providing an SE variant which incorporates a statistics pooling mechanism into the squeeze operation to generate better channel-wise statistics (Zhao-Mak, Introduction). Therefore, it would be obvious for one skilled in the art to combine the teachings of Dong et al. with those of Zhao-Mak to better discern useful channel information of Dong et al.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over “Multi-perspective Information Fusion Res2Net with Random Specmix for Fake Speech Detection”, hereinafter referred to as Dong et al., in view of “Channel Interdependence Enhanced Speaker Embeddings for Far-Field Speaker Verification”, hereinafter referred to as Zhao-Mak.
Regarding claim 4, Dong et al. discloses the method of claim 1, wherein each encoder layer further comprising:
receiving, by a dilated convolution of the respective encoder layer, an output of the SE block of the respective encoder layer (“Figure 1: Illustration of Res2Net backbone(a) and the proposed MPIF-Res2Net(b). (SE Block: the squeeze-and-excitation block [31],” Dong et al., fig. 1, highlighted portion of caption. And, “As shown on the right of Figure 1(b), MPIF module in
current channel group 𝑖 performs different convolution operation on 𝑝𝑖 or 𝑝𝑖 +𝑦𝑖−1 to get the spoofing information from different perspective. Firstly, 𝑝𝑖 is sent into the convolution operations with different dilation parameter 𝑗, where 𝑗 ∈ [1, 2], at the beginning of MPIF module (Eq. 4). The results are then passing through the dilated convolution to recalculate the energy distribution of each channel, normalize them through the Sigmoid function, and then the average pooling layer is used to get the results 𝜔𝑗𝑘 as the weight of each channel 𝑘 from 𝐶𝑜𝑛𝑣2𝑑,” Dong et al., sec. 2.2, highlighted portion.).
Dong et al., though, does not explicitly disclose increasing, by the dilated convolution of the respective encoder layer, receptive fields of multi-scale feature maps without losing spatial resolution; receiving, by a rectified linear unit (ReLU) activation function of the respective encoder layer, an output of the dilated convolution of the respective encoder layer; introducing, by the ReLU activation function of the respective encoder layer, non-linearity; receiving, by a batch normalization (BN) of the respective encoder layer, an output of the ReLU activation function of the respective encoder layer; and normalizing, by the BN of the respective encoder layer, an output of the ReLU activation function of the respective encoder layer.
Zhao-Mak is cited to disclose increasing, by the dilated convolution of the respective encoder layer, receptive fields of multi-scale feature maps without losing spatial resolution (“Specifically, we restructure the dilated convolutional layers into Res2Net blocks to generate multi-scale frame-level features,” Zhao-Mak, Abstract. It is noted that dilated convolutions are a key technique for creating multi-scale feature maps without reducing the spatial resolution of the feature maps, which addresses a major challenge in computer vision tasks like semantic segmentation.);
receiving, by a rectified linear unit (ReLU) activation function of the respective encoder layer, an output of the dilated convolution of the respective encoder layer (“The configuration of our system is illustrated in Fig. 1. The dilation of the Conv1D layers is 2, 3, and 4 for the three SERes2blocks, respectively… Each of the convolutional layers in all systems is connected
to a leaky ReLU layer, followed by batch normalization. Speaker embeddings were extracted from the affine layer’s output after the statistics pooling layer,” Zhao-Mak, sec. 3.2, highlighted portion.);
introducing, by the ReLU activation function of the respective encoder layer, non-linearity (The ReLU activation function is, by definition, non-linear.);
receiving, by a batch normalization (BN) of the respective encoder layer, an output of the ReLU activation function of the respective encoder layer (“Each of the convolutional layers in all systems is connected to a leaky ReLU layer, followed by batch normalization,” Zhao-Mak, sec. 3.2, highlighted portion.); and
normalizing, by the BN of the respective encoder layer, an output of the ReLU activation function of the respective encoder layer (“The configuration of our system is illustrated in Fig. 1. The dilation of the Conv1D layers is 2, 3, and 4 for the three SERes2blocks, respectively. The dimension of speaker embedding, Demb, is 192 and the number of training speakers, Nspk, is 7602. Each of the convolutional layers in all systems is connected to a leaky ReLU layer, followed by batch normalization,” Zhao-Mak, sec. 3.2, highlighted portion.). Zhao-Mak benefits Dong et al. by providing an SE variant which incorporates a statistics pooling mechanism into the squeeze operation to generate better channel-wise statistics (Zhao-Mak, Introduction). Therefore, it would be obvious for one skilled in the art to combine the teachings of Dong et al. with those of Zhao-Mak to better discern useful channel information of Dong et al.
Claim(s) 5 and 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over “Multi-perspective Information Fusion Res2Net with Random Specmix for Fake Speech Detection”, hereinafter referred to as Dong et al., in view of “An attention based densely connected U-NET with convolutional GRU for speech enhancement”, hereinafter referred to as Jannu et al.
Regarding claim 5, Dong et al. discloses the method of claim 1, but not wherein the trained machine learning model further comprises:
a bottleneck disposed between the encoder and a decoder of the machine learning model, wherein the bottleneck comprises: receiving, by a first gated recurrent unit (GRU) layer, an output of the encoder; performing, by the first GRU layer, a first non-linear transformation;
receiving, by a second GRU layer, an output of the first GRU layer; and performing, by the second GRU layer, a second non-linear transformation.
Jannu et al. is cited to disclose a bottleneck disposed between the encoder and a decoder of the machine learning model (“The proposed framework consists of a stacked GRU in the bottleneck,” Jannu et al., p.1, right column, highlighted portion.), wherein the bottleneck comprises:
receiving, by a first gated recurrent unit (GRU) layer, an output of the encoder (“To represent the correlation between neighboring noisy speech frames, a two Layer GRU is added in the bottleneck of encoder-decoder,” Jannu et al., p. 2, right column, highlighted portion.);
performing, by the first GRU layer, a first non-linear transformation (“After each convolution, parametric ReLU (PReLU) nonlinearity and layer normalisation are applied,” Jannu et al., p. 2, left column, highlighted portion.);
receiving, by a second GRU layer, an output of the first GRU layer (“After each convolution, parametric ReLU (PReLU) nonlinearity and layer normalisation are applied,” Jannu et al., p. 2, left column, highlighted portion.); and
performing, by the second GRU layer, a second non-linear transformation (“After each convolution, parametric ReLU (PReLU) nonlinearity and layer normalisation are applied,” Jannu et al., p. 2, left column, highlighted portion.). Jannu et al. benefits Dong et al. by introducing a U-Net with convolutional GRU to improve the performance of SE (Jannu et al., Introduction). Therefore, it be obvious for one skilled in the art to combine the teachings of Dong et al. with those of Jannu et al. to improve the speech enhancement of Dong et al.
Regarding claim 7, Dong et al. discloses the method of claim 1, but not wherein the trained machine learning model further comprises: a decoder, wherein the decoder comprises a decoder of encoder layers; and a plurality of skip connections, wherein each skip connection connects a decoder layer of the decoder layers to an encoder layer of the encoder.
Jannu et al. is cited to disclose a decoder, wherein the decoder comprises a decoder of encoder layers (“The output from the encoder layer is multiplied with an attention mask instead of concatenating them directly with same hierarchical level among the decoder
Layers,” Jannu et al., p. 2, left column, second highlighted portion.); and
a plurality of skip connections, wherein each skip connection connects a decoder layer of the decoder layers to an encoder layer of the encoder (“The model proposed is a densely connected U-Net with attention-based skip links. It consists of encoder and decoder with stacked GRU as bottleneck. The encoder and decoder have 8 layers,” Jannu et al., p. 2, left column, first highlighted portion. Also, “In the proposed model we used attention mechanism in the skip links to identify most important features from encoder layer. The output from the encoder layer is multiplied with an attention,” Jannu et al., p. 2, left column, second highlighted portion. And, “Attention-based skip links are used from encoder to decoder,” Jannu et al., right column, 2nd highlighted portion.). Jannu et al. benefits Dong et al. by introducing a U-Net with convolutional GRU to improve the performance of SE (Jannu et al., Introduction). Therefore, it be obvious for one skilled in the art to combine the teachings of Dong et al. with those of Jannu et al. to improve the speech enhancement of Dong et al.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over “Multi-perspective Information Fusion Res2Net with Random Specmix for Fake Speech Detection”, hereinafter referred to as Dong et al., in view of “An attention based densely connected U-NET with convolutional GRU for speech enhancement”, hereinafter referred to as Jannu et al., and further in view of “Improving RNN Transducer Modeling for End-to-End Speech Recognition”, hereinafter referred to Li et al. (2).
Regarding claim 6, Dong et al., as modified by Jannu et al., discloses the method of claim 5, but not wherein the first GRU layer and the second GRU layer is a uni-directional GRU layer.
Li et al. is cited to disclose wherein the first GRU layer and the second GRU layer is a uni-directional GRU layer (“Note that all the networks in this study are uni-directional and our layer normalized GRU is different from the bi-directional GRU with batch normalization used in [5],” Li et al (2).). Li et al (2) benefits Dong et al. by proposing better model structures (than the layer-normalized long short-term memory proposed in prior work) so that RNN-T models obtain better accuracy but smaller footprint (Li et al (2), Introduction). Therefore, it be obvious for one skilled in the art to combine the teachings of Dong et al. with those of Li et al (2) to improve the model architecture of Dong et al.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over “Multi-perspective Information Fusion Res2Net with Random Specmix for Fake Speech Detection”, hereinafter referred to as Dong et al., in view of “Direct and Residual Subspace Decomposition of Spatial Room Impulse Responses”, hereinafter referred to as Deppisch et al.
Regarding claim 8, Dong et al. discloses the method of claim 1, but not wherein the first speech waveform is substantially equivalent to a second speech waveform including at least one of: a convolutive room impulse response (RIR) and additive background noise. Deppisch et al. is cited to disclose wherein the first speech waveform is substantially equivalent to a second speech waveform including at least one of: a convolutive room impulse response (RIR) and additive background noise (“Following a convolutive multiple-input-multiple-output (MIMO) signal model [40, Ch. 2.1.4], the array signals are convolutive mixtures of the source signals s(t) plus additive noise n(t), x(t) = Hs(t) + n(t),” Deppisch et al., p.2, highlighted portion.). Deppisch et al. benefits Dong et al. by incorporating methods to capture the directional properties of an acoustic environment, thereby improving human auditory perception (Deppisch et al., Introduction). Therefore, it be obvious for one skilled in the art to combine the teachings of Dong et al. with those of Deppisch et al. to improve the auditory perception of Dong et al.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20250104727, hereinafter referred to as Zhang et al., in view of “Direct and Residual Subspace Decomposition of Spatial Room Impulse Responses”, hereinafter referred to as Deppisch et al.
Regarding claim 10, Zhang et al. discloses the non-transitory computer-readable medium of claim 9, but not wherein the first speech waveform of the respective speech waveform is the second speech waveform of the respective speech waveform including at least one of: a convolutive room impulse response (RIR) and additive background noise. Deppisch et al. is cited to disclose wherein the first speech waveform of the respective speech waveform is the second speech waveform of the respective speech waveform including at least one of: a convolutive room impulse response (RIR) and additive background noise (“Following a convolutive multiple-input-multiple-output (MIMO) signal model [40, Ch. 2.1.4], the array signals are convolutive mixtures of the source signals s(t) plus additive noise n(t), x(t) = Hs(t) + n(t),” Deppisch et al., p.2, highlighted portion.). Deppisch et al. benefits Zhang et al. by incorporating methods to capture the directional properties of an acoustic environment, thereby improving human auditory perception (Deppisch et al., Introduction). Therefore, it be obvious for one skilled in the art to combine the teachings of Zhang et al. with those of Deppisch et al. to improve the auditory perception of Dong et al.
Claim(s) 11 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20250104727, hereinafter referred to as Zhang et al., in view of “Multi-perspective Information Fusion Res2Net with Random Specmix for Fake Speech Detection”, hereinafter referred to as Dong et al.
Regarding claim 11, Zhang et al. discloses the non-transitory computer-readable medium of claim 9, but not wherein the machine learning model comprises: an encoder, wherein the encoder comprises a plurality of encoder layers, each encoder layer comprising: a residual network block, wherein the residual network block receives one of: the first speech waveform of the respective speech waveform or multi-scale feature maps outputted from a previous encoder layer of the plurality of encoder layers to generate multi-scale feature maps of a respective encoder layer.
Dong et al. is cited to disclose an encoder, wherein the encoder comprises a plurality of encoder layers, each encoder layer comprising:
a residual network block, wherein the residual network block receives one of: the first speech waveform of the respective speech waveform or multi-scale feature maps outputted from a previous encoder layer of the plurality of encoder layers to generate multi-scale feature maps of a respective encoder layer (“Res2Net [17] was proposed by Gao et al. to enhance its ability to capture multi-scale information by transferring information among channel groups,” Dong et al., p. 1, col. 1, last sentence. And, Dong et al., p. 2, col. 1, 2nd para.). Dong et al. benefits Zhang et al. by proposing an MPIF-Res2Net model to fuse information from different perspectives, making the information learned by the model more diverse, thereby reducing the redundancy
caused by similar information and avoiding interference with the learning of discriminative information (Dong et al., Abstract). Therefore, it would be obvious for one skilled in the art to combine the teachings of Zhang et al. with those of Dong et al. to improve the speech enhancement techniques of Zhang et al.
Regarding claim 12, Zhang et al., as modified by Dong et al., discloses the non-transitory computer-readable medium of claim 11, wherein the residual network block is a Res2Net (“In this section, we introduce the structure of the proposed MPIF-Res2Net, it reduces redundancy caused by learning single-perspective forgery information by integrating information from multiple perspectives,” Dong et al., sec. 2, first sentence.).
Claim(s) 13-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20250104727, hereinafter referred to as Zhang et al., in view of “Multi-perspective Information Fusion Res2Net with Random Specmix for Fake Speech Detection”, hereinafter referred to as Dong et al., and further in view of “Channel Interdependence Enhanced Speaker Embeddings for Far-Field Speaker Verification”, hereinafter referred to as Zhao-Mak.
Regarding claim 13, Zhang et al., as modified by Dong et al., discloses the non-transitory computer-readable medium of claim 11, but not wherein each encoder layer further comprising: a squeeze-and-excitation (SE) block, wherein the SE block receives an output of the residual network block of the respective encoder layer and adaptively recalibrates the multi-scale feature maps.
Zhao-Mak is cited to disclose wherein each encoder layer further comprises: a squeeze-and-excitation (SE) block, wherein the SE block receives an output of the residual network block of the respective encoder layer and adaptively recalibrates the multi-scale feature maps (“In this block, the SE unit recalibrates the channel activations based on the multi-scale features obtained from the conv-ops in the Res2block,” Zhao-Mak, p. 2, highlighted passage.). Zhao-Mak benefits Zhang et al. by providing an SE variant which incorporates a statistics pooling mechanism into the squeeze operation to generate better channel-wise statistics (Zhao-Mak, Introduction). Therefore, it would be obvious for one skilled in the art to combine the teachings of Zhang et al. with those of Zhao-Mak to better discern useful channel information of Zhang et al.
Regarding claim 14, Zhang et al., as modified by Dong et al., discloses the non-transitory computer-readable medium of claim 11, but not wherein each encoder layer further comprising: a dilated convolution, wherein the dilated convolution receives an output of the SE block of the respective encoder layer and increases receptive fields of multi-scale feature maps without losing spatial resolution; a rectified linear unit (ReLU) activation function, wherein the ReLU activation function receives the output of the dilated convolution of the respective encoder layer and introduces non-linearity; and
a batch normalization (BN), wherein BN receives the output of the ReLU activation function of the respective encoder layer and normalizes the output of the ReLU activation function of the respective encoder layer.
Zhao-Mak is cited to disclose a dilated convolution, wherein the dilated convolution receives an output of the SE block of the respective encoder layer and increases receptive fields of multi-scale feature maps without losing spatial resolution (“Specifically, we restructure the dilated convolutional layers into Res2Net blocks to generate multi-scale frame-level features,” Zhao-Mak, Abstract. It is noted that dilated convolutions are a key technique for creating multi-scale feature maps without reducing the spatial resolution of the feature maps, which addresses a major challenge in computer vision tasks like semantic segmentation.);
a rectified linear unit (ReLU) activation function, wherein the ReLU activation function receives the output of the dilated convolution of the respective encoder layer and introduces non-linearity (“The configuration of our system is illustrated in Fig. 1. The dilation of the Conv1D layers is 2, 3, and 4 for the three SERes2blocks, respectively… Each of the convolutional layers in all systems is connected to a leaky ReLU layer, followed by batch normalization. Speaker embeddings were extracted from the affine layer’s output after the statistics pooling layer,” Zhao-Mak, sec. 3.2, highlighted portion.); and
a batch normalization (BN), wherein BN receives the output of the ReLU activation function of the respective encoder layer and normalizes the output of the ReLU activation function of the respective encoder layer (“The configuration of our system is illustrated in Fig. 1. The dilation of the Conv1D layers is 2, 3, and 4 for the three SERes2blocks, respectively. The dimension of speaker embedding, Demb, is 192 and the number of training speakers, Nspk, is 7602. Each of the convolutional layers in all systems is connected to a leaky ReLU layer, followed by batch normalization,” Zhao-Mak, sec. 3.2, highlighted portion.). Zhao-Mak benefits Zhang et al. by providing an SE variant which incorporates a statistics pooling mechanism into the squeeze operation to generate better channel-wise statistics (Zhao-Mak, Introduction). Therefore, it would be obvious for one skilled in the art to combine the teachings of Zhang et al. with those of Zhao-Mak to better discern useful channel information of Zhang et al.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 20250104727, hereinafter referred to as Zhang et al., in view of “An attention based densely connected U-NET with convolutional GRU for speech enhancement”, hereinafter referred to as Jannu et al.
Regarding claim 15, Zhang et al. discloses the non-transitory computer-readable medium of claim 9, but not wherein the machine learning model further comprises: a bottleneck disposed between an encoder and a decoder of the machine learning model, wherein the bottleneck comprises: a first gated recurrent unit (GRU) layer, wherein the GRU layer receives an output of the encoder and performs a first non-linear transformation; and a second GRU layer, wherein the second GRU layer receives an output of the first GRU layer and performs a second non-linear transformation.
Jannu et al. is cited to disclose a bottleneck disposed between the encoder and a decoder of the machine learning model (“The proposed framework consists of a stacked GRU in the bottleneck,” Jannu et al., p.1, right column, highlighted portion.), wherein the bottleneck comprises:
a first gated recurrent unit (GRU) layer, wherein the GRU layer receives an output of the encoder and performs a first non-linear transformation (“To represent the correlation between neighboring noisy speech frames, a two Layer GRU is added in the bottleneck of encoder-decoder,” Jannu et al., p. 2, right column, highlighted portion.); and
a second GRU layer, wherein the second GRU layer receives an output of the first GRU layer and performs a second non-linear transformation (“After each convolution, parametric ReLU (PReLU) nonlinearity and layer normalisation are applied,” Jannu et al., p. 2, left column, highlighted portion.). Jannu et al. benefits Zhang et al. by introducing a U-Net with convolutional GRU to improve the performance of SE (Jannu et al., Introduction). Therefore, it be obvious for one skilled in the art to combine the teachings of Zhang et al. with those of Jannu et al. to improve the speech enhancement of Zhang et al.
Claim(s) 16, 19, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over “Multi-perspective Information Fusion Res2Net with Random Specmix for Fake Speech Detection”, hereinafter referred to as Dong et al., in view of “An attention based densely connected U-NET with convolutional GRU for speech enhancement”, hereinafter referred to as Jannu et al., and further in view of US 20250104727, hereinafter referred to as Zhang et al.
Regarding claim 16, Dong et al. discloses a system comprising:
a memory device; and
a processing device coupled to the memory device, wherein the processing device is to perform operations comprising:
receiving, by a speech enhancement system, a first speech waveform (“In this work, we use a random Specmix strategy to help the model to locate the discriminative information and enhance the generalization of the model. For the training of deep neural networks, we always transform the raw audio from time domain into time-frequency domain.… At the same time, to avoid the conduction of Specmix on all the samples, inspired by [21], we randomly choose speech samples according to the hyperparameter 𝑝_ℎ𝑦𝑝𝑒𝑟 in advance to conduct Specmix operation,” Dong et al., p. 3, sec. 2.3. It is clear from this excerpt that the input to the encoder comprised speech samples.); and
generating, by an encoder of the speech enhancement system, a latent output of the first speech waveform, wherein each layer of a plurality of encoder layers of the encoder comprises a residual network block and a squeeze-and-excitation (SE) block (Dong et al., p. 3, Table 1.).
Dong et al., though, does not disclose performing, by a bottleneck of the speech enhancement system, non-linear transformation on the latent output of the first speech waveform, wherein the bottleneck comprises a first gated recurrent unit (GRU) layers and a second GRU layer; and predicting, by a decoder of the speech enhancement system, a second speech waveform from the non-linear transformation of the latent output of the first speech waveform.
Jannu et al. is cited to disclose performing, by a bottleneck of the speech enhancement system (“The proposed framework consists of a stacked GRU in the bottleneck,” Jannu et al., p.1, right column, highlighted portion.), non-linear transformation on the latent output of the first speech waveform (“After each convolution, parametric ReLU (PReLU) nonlinearity and layer normalisation are applied,” Jannu et al., p. 2, left column, highlighted portion.), wherein the bottleneck comprises a first gated recurrent unit (GRU) layers and a second GRU layer (“To represent the correlation between neighboring noisy speech frames, a two Layer GRU is added in the bottleneck of encoder-decoder,” Jannu et al., p. 2, right column, highlighted portion.); and
predicting, by a decoder of the speech enhancement system, a second speech waveform from the non-linear transformation of the latent output of the first speech waveform (Jannu et al., p. 2, left column, highlighted portion.). Jannu et al. benefits Dong et al. by introducing a U-Net with convolutional GRU to improve the performance of SE (Jannu et al., Introduction). Therefore, it be obvious for one skilled in the art to combine the teachings of Dong et al. with those of Jannu et al. to improve the speech enhancement of Dong et al.
Neither Jannu et al. nor Dong et al. explicitly teach a memory device or a processing device.
Zhang et al. is additionally cited to teach a memory device (Zhang et al., para [0006].); and
a processing device coupled to the memory device (Zhang et al., para [0006].). Zhang et al. benefits Jannu et al. by providing the hardware necessary for implementing the teachings of Zhang et al.
Regarding claim 19, Dong et al., as modified by Jannu et al. and Zhang et al., discloses the system of claim 16, wherein performing, by the bottleneck of the speech enhancement system, non-linear transformation on the latent output of the first speech waveform includes:
receiving latent output of the first speech waveform (“To represent the correlation between neighboring noisy speech frames, a two Layer GRU is added in the bottleneck of encoder-decoder,” Jannu et al., p. 2, right column, highlighted portion.);
performing, by the first GRU layer, a first non-linear transformation on the latent output of the first speech waveform (“After each convolution, parametric ReLU (PReLU) nonlinearity and layer normalisation are applied,” Jannu et al., p. 2, left column, highlighted portion.) ); and
receiving, by the second GRU layer, an output of the first GRU layer (“After each convolution, parametric ReLU (PReLU) nonlinearity and layer normalisation are applied,” Jannu et al., p. 2, left column, highlighted portion.); and
performing, by the second GRU layer, a second non-linear transformation (“After each convolution, parametric ReLU (PReLU) nonlinearity and layer normalisation are applied,” Jannu et al., p. 2, left column, highlighted portion.).
Regarding claim 20, Dong et al., as modified by Jannu et al. and Zhang et al., discloses the system of claim 16, wherein predicting the second speech waveform from the non-linear transformation of the latent output of the first speech waveform includes:
for each decoder layer of a plurality of decoder layers of the decoder, receiving a latent output of an encoder layer of the plurality of encoder layers connected to a respective decoder layer via a skip connection (“The model proposed is a densely connected U-Net with attention-based skip links. It consists of encoder and decoder with stacked GRU as bottleneck. The encoder and decoder have 8 layers,” Jannu et al., p. 2, left column, first highlighted portion. Also, “In the proposed model we used attention mechanism in the skip links to identify most important features from encoder layer. The output from the encoder layer is multiplied with an attention,” Jannu et al., p. 2, left column, second highlighted portion. And, “Attention-based skip links are used from encoder to decoder,” Jannu et al., right column, 2nd highlighted portion.).
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over “Multi-perspective Information Fusion Res2Net with Random Specmix for Fake Speech Detection”, hereinafter referred to as Dong et al., in view of “An attention based densely connected U-NET with convolutional GRU for speech enhancement”, hereinafter referred to as Jannu et al., further in view of US 20250104727, hereinafter referred to as Zhang et al., and further in view of “Direct and Residual Subspace Decomposition of Spatial Room Impulse Responses”, hereinafter referred to as Deppisch et al.
Regarding claim 17, Dong et al., as modified by Jannu et al. and Zhang et al., discloses the system of claim 16, but not wherein the first speech waveform is substantially equivalent to a second speech waveform including at least one of: a convolutive room impulse response (RIR) and additive background noise. Deppisch et al. is cited to dislcose wherein the first speech waveform is substantially equivalent to a second speech waveform including at least one of: a convolutive room impulse response (RIR) and additive background noise (“Following a convolutive multiple-input-multiple-output (MIMO) signal model [40, Ch. 2.1.4], the array signals are convolutive mixtures of the source signals s(t) plus additive noise n(t), x(t) = Hs(t) + n(t),” Deppisch et al., p.2, highlighted portion.). Deppisch et al. benefits Dong et al. by incorporating methods to capture the directional properties of an acoustic environment, thereby improving human auditory perception (Deppisch et al., Introduction). Therefore, it be obvious for one skilled in the art to combine the teachings of Dong et al. with those of Deppisch et al. to improve the auditory perception of Dong et al.
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over “Multi-perspective Information Fusion Res2Net with Random Specmix for Fake Speech Detection”, hereinafter referred to as Dong et al., in view of “An attention based densely connected U-NET with convolutional GRU for speech enhancement”, hereinafter referred to as Jannu et al., further in view of US 20250104727, hereinafter referred to as Zhang et al., and further in view of “Channel Interdependence Enhanced Speaker Embeddings for Far-Field Speaker Verification”, hereinafter referred to as Zhao-Mak.
Regarding claim 18, Dong et al., as modified by Jannu et al. and Zhang et al., discloses the system of claim 16, wherein generating the latent output of the first speech waveform includes:
for each encoder layer of the plurality of encoder layers, generating, by the residual network block of a respective encoder layer, multi-scale feature maps associated with an input of the respective encoder layer (Dong et al., p. 3, Table 1.).
Dong et al., though, does not describe adaptively recalibrating, by the SE block of the respective encoder layer, multi-scale feature maps, wherein an output of the respective encoder layer is a latent output of the input of the respective encoder layer.
Zhao-Mak is cited to disclose adaptively recalibrating, by the SE block of the respective encoder layer, multi-scale feature maps, wherein an output of the respective encoder layer is a latent output of the input of the respective encoder layer (“In this block, the SE unit recalibrates the channel activations based on the multi-scale features obtained from the conv-ops in the Res2block,” Zhao-Mak, p. 2, highlighted passage.). Zhao-Mak benefits Dong et al. by providing an SE variant which incorporates a statistics pooling mechanism into the squeeze operation to generate better channel-wise statistics (Zhao-Mak, Introduction). Therefore, it would be obvious for one skilled in the art to combine the teachings of Dong et al. with those of Zhao-Mak to better discern useful channel information of Dong et al.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See attached PTO-892. Other prior art of note includes Rybicka et al. and Luo-Zhou, which describe Res2NET and Squeeze-Excitation architectures.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANNE L THOMAS-HOMESCU whose telephone number is (571)272-0899. The examiner can normally be reached Mon-Fri 8-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh M Mehta can be reached on 5712727453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANNE L THOMAS-HOMESCU/Primary Examiner, Art Unit 2656
Prosecution Insights