Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103 is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Priority
Acknowledgment is made of applicant's claim for domestic priority based on US provisional application 63680889 filed on 08/08/2024.
Claim Rejections - 35 USC § 103
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 103 that form the basis for the rejections under this section made in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 8-11, and 17-20 are rejected under 35 USC 103 for being unpatentable over Huang et al. (US 2024/0290320 A1) in view of Li et al. (US 2017/0278513 A1).
Regarding Claims 1, 10, and 19, Huang discloses an electronic device (Fig. 1, user device 10 ¶26 or remote computing system 60 per ¶27) comprising:
at least one processing device (data processing hardware 12 on user device 10 per ¶26 or data processing hardware 62 on remote computing system 60 per ¶27) configured to:
receive a user utterance via an audio input device (¶28, audio subsystem 108 receives utterance 106 spoken by the user captured by audio capture device 16a);
provide the user utterance to a first recurrent neural network (RNN) classifier and a second RNN classifier to process the user utterance (Fig. 2, ¶37, a first RNN-T model comprising encoder network 210 and first decoder 220a and a second RNN-T model comprising the encoder network 210 and the second decoder 220b; ¶41, provide sequence of acoustic frames 110 / d-dimensional feature vectors x to encoder network 210) and provide outputs to a first joint layer (Fig. 2, ¶37, first RNN-T model provides, at each of a plurality of output steps, probability distribution 224a over possible speech recognition hypotheses and second RNN-T model provides, at each of a plurality of output steps, end of segment EOS 232 to Softmax layer 240);
provide the user utterance to an automated speech recognition (ASR) model to process the user utterance and provide a text transcript to a text classifier (¶29, ASR model 200 perform streaming speech recognition to produce an initial speech recognition result 120a; e.g., initial speech recognition results 120a during time 1 per ¶30; Fig. 3, ¶47, ASR model 200 / Softmax layer 240 provides initial speech recognition results (i.e., initial speech recognition result 120a per ¶29) to prediction network 300, which generates single embedding vector pui 350 per ¶53, which captures linguistic dependencies between non-blank symbols 242 predicted during the previous output steps so far per ¶43);
combine the outputs from the first convolutional RNN classifier and the second convolutional RNN classifier using the first joint layer (¶43, softmax layer 240 generates a respective sequence of non-blank symbols 242; i.e., a non-blank symbol 242 at the current output step corresponding to a current output step probability distribution 224a and a current output step EOS 232);
combine outputs from the first joint layer and the text classifier using a second joint layer (¶44, each Joint Network 222 in decoder 220a combines higher-order feature representation 214 and embedding vector 350 produced by prediction network 300 to generate probability distribution 224a for the final speech recognition result 120b such that decoder 220a’s prediction of each symbol yu 242 is conditioned not only on the acoustics but also on the sequence of labels 242 yu-n,…,yui-1 output so far per ¶46); and
determine an audio class based on a result from the second joint layer (¶46, the final Softmax layer 240 receives probability distributions 224a for the final speech recognition result 120b and selects the output label / symbol with the highest probability to produce transcription 120; ¶29, the final speech recognition result 120b improves the initial speech recognition result 120a), wherein the audio class indicates whether the user utterance includes speech intended for further processing (¶30 and ¶35, using NLU processing of the transcription 120 to execute user command / query).
Huang does not disclose the first RNN classifier is a first convolutional recurrent neural network (RNN) classifier and the second RNN classifier is a second convolutional RNN classifier.
Li discloses a speech recognition system (Fig. 1) using a first convolutional recurrent neural network classifier and a second convolutional RNN classifier to process user utterance and provide outputs to a first joint layer (¶13, neural network trained as an acoustic model comprises a convolutional layer, one or more long-short term memory layers, and multiple hidden layers to convolve audio data for a first channel with a first filter (spatial or spectral filtering) to generate a first convolution output, convolving audio data for a second channel with a second filter (spectral or spatial filtering) to generate second convolution outputs, and combining the first convolution outputs and the second convolution outputs).
It would’ve been obvious to one ordinarily skilled in the art before the effective filing date of the invention to implement the first RNN classifier as a first convolutional RNN classifier and the second RNN classifier as a second convolutional RNN classifier to provide spatial filtering and spectral filtering to perform acoustic processing (Li, ¶13; compare Huang, ¶9, RNN-T encoder as acoustic model processing acoustic frames to generate higher order feature representations for corresponding acoustic frames).
Further regarding claim 19, Huang discloses a non-transitory machine readable medium comprising instructions that when executed cause at least one processor of an electronic device to execute the functions of claims 1 and 10 (data processing hardware 12 executing instructions stored in memory hardware 14 on user device 10 per ¶26 or data processing hardware 62 executing instructions stored in memory hardware 64 on remote computing system 60 per ¶27).
Regarding Claims 2, 11, and 20, Huang discloses wherein:
the first joint layer uses concatenation, cross attention, or context layers to combine the outputs from the first convolutional RNN classifier and the second convolutional RNN classifier (¶37 and ¶¶39-40, encoder network 210 of the first RNN-T model and the second RNN-T model use self-attention layers to generate respective higher order feature representations 214a and 214b; ¶39, first encoder 212a in the encoder network 210 performs concatenation operations to generate the higher order feature representations 214a and 214b; ¶43, softmax layer 240 generates dense representation / embedding vector 350 based on the higher order feature representations 214a and 214b as shown in Fig. 2; thus, Softmax layer uses concatenation and cross attention); and
the second joint layer uses concatenation, cross attention, or context layers to combine the outputs from the first joint layer and the text classifier (¶37 and ¶¶39-40, encoder network 210 of the first RNN-T model and the second RNN-T model use self-attention layers to generate respective higher order feature representations 214a and 214b; ¶39, first encoder 212a in the encoder network 210 performs concatenation operations to generate the higher order feature representations 214a and 214b; ¶44, each Joint Network 222 in decoder 220a combines higher-order feature representation 214 and embedding vector 350 produced by prediction network 300 to generate probability distribution 224a; thus, the Joint Network 222 uses concatenation and the cross attention).
Regarding Claims 8 and 17, Huang discloses wherein the text classifier is one of: an RNN classifier trained with a dataset including text transcripts (¶42, prediction network 300 is part of RNN-T decoder 220 trained using any suitable RNN-T training process for training an ASR model; e.g., ¶55, training samples 415 including audio data 422 characterizing multiple segments of long-form speech and corresponding ground truth transcription 424 of the long form speech); or a text classifier created by finetuning a pre-trained model with a dataset including text transcripts.
Regarding Claims 9 and 18, Huang discloses wherein the audio class is determined based on a confidence score output by the second joint layer (¶46, the final Softmax layer 240 receives probability distributions 224a for the final speech recognition result 120b and selects the output label / symbol with the highest probability (i.e., Softmax layer 240 outputs the highest probability distributions 224a for corresponding output labels / symbols) to produce transcription 120).
Claims 3-7 and 12-16 are rejected under 35 USC 103 for being unpatentable over Huang et al. (US 2024/0290320 A1) and Li et al. (US 2017/0278513 A1) as applied to claims 1 and 10, in view of Zou (US 2025/0391419 A1).
Regarding Claims 3 and 12, Huang does not disclose wherein: the first convolutional RNN classifier is trained by performing pretraining of a first autoencoder that receives and processes clean speech training audio; and the second convolutional RNN classifier is trained by performing pretraining of a second autoencoder that receives and processes noisy training audio.
Li teaches a multitask learning strategy (¶29 and ¶50) training convolutional RNN classifier (Fig. 1, Deep Neural Network) by performing pretraining of LSTM classifier by receiving and processing clean speech training audio and noisy training audio (¶41, training to regularize model parameters; ¶42, training based on noisy training data comprising clean utterances corrupted by noise signals added to the clean utterances at SNRs ranging from 0 to 20 dB).
According to Zou, a deep learning model is an autoencoder (¶47) such as a recurrent neural network (¶49) with encoding layer such as LSTM (¶72). Therefore, Zou demonstrated that the RNN-LSTM classifiers of Huang and Li are or can be autoencoders.
It would’ve been obvious to one ordinarily skilled in the art before the effective filing date of the invention to train the first convolutional RNN classifier / first RNN-T model by performing pretraining of a first autoencoder / RNN-LSTM that receives and processes clean speech training audio (Li, ¶42, training data includes clean utterances with 0 dB of noise added) and train the second convolutional RNN classifier / second RNN-T model by performing pretraining of a second autoencoder / RNN-LSTM that receives and processes noisy training audio (Li, ¶42, training data includes clean utterances with non-zero dB of noise added) in order to implement multitask learning to improve robustness (Li, ¶50).
Regarding Claims 4 and 13, Huang as modified by Li teaches wherein: the first convolutional RNN classifier is provided with seed weights from the first autoencoder during training; and the second convolutional RNN classifier is provided with seed weights from the second autoencoder during training (Li, ¶44, weights for CNN and DNN layers are initialized using Glorot-Bengio strategy).
Regarding Claims 5 and 14, Huang discloses wherein: the first autoencoder includes a first convolutional RNN encoder and a first convolutional RNN decoder configured to receive an output from the first convolutional RNN encoder (Huang ¶37, encoder network 210 and the first decoder 220a form a first RNN-T model; Fig. 2 shows first decoder 220a receives output from encoder network 210); and
the second autoencoder includes a second convolutional RNN encoder, and a second convolutional RNN decoder configured to receive an output from the second convolutional RNN encoder (Huang ¶37, encoder network 210 and the second decoder 22b form a second RNN-T model; Fig. 2 shows second decoder 220b receives output from encoder network 210).
As modified by Li to implement the multitask learning strategy (Li ¶29 and ¶50) training convolutional RNN classifier (Fig. 1, Deep Neural Network) by performing pretraining of LSTM classifier by receiving and processing clean speech training audio and noisy training audio (Li ¶41, training to regularize model parameters; ¶42, training based on noisy training data comprising clean utterances corrupted by noise signals added to the clean utterances at SNRs ranging from 0 to 20 dB), the training of the first RNN-T model comprises configuring the first RNN-T model / convolutional RNN encoder to receive speech training audio comprising clean speech training audio (Li ¶42, training data corresponding to clean utterances with noise signals of 0 dB) and training of the second RNN-T model comprises configuring the second RNN-T model / convolutional RNN encoder to receive noisy training audio (Li ¶42, training data corresponding to clean utterances with noise signals of non-zero dB).
Regarding Claims 6 and 15, Huang discloses wherein the convolutional RNN encoder is considered trained when a difference between input features to the convolutional RNN encoder and output features of the convolutional RNN decoder is minimized (¶56, training RNN-T model based on loss 432; ¶57, based on loss 432, train the RNN-T model to minimize loss 432).
Regarding Claims 7 and 16, Huang as modified by Li teaches wherein the first convolutional RNN classifier, having the seed weights from the first autoencoder, the second convolutional RNN classifier, having the seed weights from the second autoencoder (Li, ¶44, weights for CNN and DNN layers are initialized using Glorot-Bengio strategy; Zou demonstrated that deep learning model / DNN layer can be an autoencoder such as RNN-LSTM per Zou at ¶47, ¶49, and ¶72), and the text classifier are jointly trained using a same audio dataset (Huang, Fig. 2 showing first RNN-T model comprising Encoder 210 + First Decoder 220a with a separate respective prediction network 300 per ¶42 and second RNN-T model comprising Encoder 210 + Second Decoder 220b with a separate respective prediction network 300, though 220a and 220b can share a common prediction network 300), including: the first convolutional RNN classifier the second convolutional RNN classifier are trained using audio samples from the same audio dataset (Huang, ¶7 and 12, initially training the first joint network (see Fig. 2, Joint Network 222, 222a as part of the first RNN-T model) and initializing the second joint network (see Fig. 2, Joint Network 222, 222b as part of the second RNN-T model) with the same parameters as the trained first joint network), and the text classifier is trained using text transcriptions created using the same audio dataset (¶7 and ¶12, using the ground-truth end of segment label inserted into the corresponding transcription of the spoken utterance characterized by the audio data of each training sample).
Conclusion
Prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
CN 113297364 A discloses a natural language understanding device in a dialogue system comprising a word embedding layer, an encoding representation layer, and a joint learning layer to generate intent by concatenating the output of a CNN mode, a RCNN (recurrent convolutional neural network), and a VDCNN.
US 2022/0093091 A1 discloses analyzing received voice commands by using a convolutional gated recurrent neural network and long short-term memory-GRNN for sentence tokenization and buffering of input command to extract relevant portions or filter the sensitive information in input command (¶23).
US 2022/027734 A1 discloses a RNN-T model comprising an encoder network (transcription network), a prediction network, a joint network, and a softmax block (¶61 and Fig. 5) where the encoder network (analogous to acoustic model per ¶62) computes high-dimensional embeddings of the input sequence (¶62), the prediction network (RNN language model) generates high level representation based on previous speech recognition predicted by the RNN-T model (¶63), and the joint network combines the encoder network output and the prediction network output (¶65).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to examiner Richard Z. Zhu whose telephone number is 571-270-1587 or examiner’s supervisor Hai Phan whose telephone number is 571-272-6338. Examiner Richard Zhu can normally be reached on M-Th, 0730:1700.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RICHARD Z ZHU/Primary Examiner, Art Unit 2654 08/07/2026
