DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections – 35 USC § 103
1. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
2. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
3. Claims 1, 2, 7-13 & 18-22 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20220108688 A1 hereinafter, Wang ‘688) in view of Galle et al. (US 20220147721 A1 hereinafter, Galle ‘721).
Regarding claim 12; Wang ‘688 discloses system (Fig. 6, Computing Device 600)
comprising:
data processing hardware (Fig. 6, Processor 610);
and memory hardware (Fig. 6, Memory 620) in communication with the data processing hardware (i.e. As shown in Fig. 6, computing device 600 includes a processor 610 coupled to memory 620. Paragraph 0055), the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations (i.e. Memory 620 may be used to store software executed by computing device 600 and/or one or more data structures used during operation of computing device 600. Paragraph 0056)
comprising:
obtaining an audio encoder (Fig. 3, mBERT105) pre-trained on an initial training data set (i.e. Fig. 3 is a simplified diagram illustrating parameter transfer from a pre-trained multilingual language model mBERT. Paragraph 0008)
the audio encoder comprising a plurality of multi-head attention layers (i.e. The mBERT decoder consists of six transformer decoder layers with a hidden size of 3072 and an attention dimension of 756, and four attention heads. Paragraph 0076);
obtaining a first adapter corresponding to a first task (i.e. The adapter layer 108 includes a language-specific adapter that adapts a self-attention output into a adapter output capturing language-intrinsic knowledge from self-attended features. Paragraph 0029) obtaining a second adapter corresponding to a second task (i.e. The adapter layer 108 includes common adapter that adapts the self-attention output into a adapter output capturing language-agnostic knowledge from the self-attended features. Paragraph 0029);
and adapting the audio encoder for the first task and the second task by inserting, in parallel (i.e. A language-specific and a language-agnostic adapter are employed in parallel to augment each encoder and decoder layer. Paragraph 0023) the first adapter and the second adapter at one or more of the plurality of multi-head attention layers of the audio encoder (i.e. For example, Fig. 2A provides a simplified diagram illustrating the structure of encoder 110. As shown in Fig. 2A, an adapter layer 108 is added after the two layer norm 104a-b (not shown in Fig. 1) and self-attention layer 106. The self-attention layer 106 attends to features of an input. The adapter layer 108 includes a language-specific adapter that adapts a self-attention output into a adapter output capturing language-intrinsic knowledge from self-attended features, and a common adapter that adapts the self-attention output into a adapter output capturing language-agnostic knowledge from the self-attended features, as further described in relation to Fig. 3. Paragraph 0029)
Examiner reasonably believes that Wang ‘688 discloses adapting the audio encoder for the first task and the second task by inserting, in parallel as expressed above. However, Examiner cites Galle ‘721 to cure any deficiencies of Wang ‘688.
Galle ‘721 discloses adapting the audio encoder for the first task and the second task by inserting, in parallel (i.e. The multilingual neural machine translation system is configured where (i) each adapter layer corresponding to a language in the set of languages is trained with parallel data of at least one other language in the set of languages. Paragraphs 0009-0010)
Wang ‘688 and Galle ‘721 are combinable because they are from same field of endeavor of speech systems (Galle ‘721 at “Field”).
Before the effective filing date, it would have been obvious to a person of ordinary skill in the art to modify the speech system as taught by Wang ‘688 by adding adapting the audio encoder for the first task and the second task by inserting, in parallel as taught by Galle ‘721. The motivation for doing so would have been advantageous so that language translation quality can be improved for languages that lack sufficient training data. Therefore, it would have been obvious to combine Wang ‘688 with Galle ‘721 to obtain the invention as specified.
Regarding claim 13; Wang ‘688 discloses wherein: the first adapter is pre-trained on a first task training data set corresponding to the first task; and the second adapter is pre-trained on a second task training data set corresponding to the second task (Fig. 7B, Step 704 i.e. At step 704, decoder parameters may be transferred from a pre-trained language model (e.g., mBERT) to the transformer decoder. Paragraphs 0029 & 0063)
Regarding claim 18; Wang ‘688 discloses wherein the operations further comprise: obtaining, based on an input, a first output from the first adapter; obtaining, based on the input, a second output from the second adapter; and aggregating the first output and the second output (i.e. The self-attention layer 106 attends to features of an input. The adapter layer 108 includes a language-specific adapter that adapts a self-attention output into a adapter output capturing language-intrinsic knowledge from self-attended features, and a common adapter that adapts the self-attention output into a adapter output capturing language-agnostic knowledge from the self-attended features. A feed-forward sublayer 109 generates an encoded representation of the input (encoder hidden states) based on a combination of the first adapter output and the second adapter output. Paragraph 0029)
Regarding claim 19; Wang ‘688 discloses wherein the operations further comprise: obtaining, based on the input, an encoded representation output from the audio encoder; and combining the encoded representation with the aggregated first and second outputs (i.e. A feed-forward sublayer 109 generates an encoded representation of the input (encoder hidden states) based on a combination of the first adapter output and the second adapter output. Paragraph 0029)
Regarding claim 20; Wang ‘688 discloses wherein the initial training data set comprises a set of un-transcribed speech utterances (i.e. At step 708, a batch of multilingual utterances may be sampled from the training dataset. At step 710, a language adapter mask may be generated using the language tag in the batch of utterances. For example, the language adapter mask may indicate the specific type of Language of the multilingual utterance, e.g., Spanish, English, etc. Paragraphs 0065-0066)
Regarding claim 21; Wang ‘688 discloses wherein the audio encoder is pre-trained on the set of un-transcribed speech utterances using Bidirectional Encoder Representations from Transformers (BERT)-based speech pre-training with random projection quantizer (BEST-RQ) (i.e. At step 702, a training dataset of multilingual data may be received, e.g., at the data interface 615, and a transformer model including the encoder 110 and decoder 115 may be initialized with randomly initiated parameters. For example, the transformer model may be stored at memory 620. At step 704, decoder parameters may be transferred from a pre-trained language model (e.g., mBERT) to the transformer decoder. Paragraphs 0062-0063).
Regarding claim 22; Wang ‘688 discloses wherein the plurality of multi-head attention layers of the audio encoder comprises a plurality of Conformer layers (i.e. The encoder 110 includes a number of encoder layers. Each encoder layer includes a two-dimensional convolution layer 102 followed by a self-attention layer 106. The convolution layer 102 is used to extract more robust features before they are sent to the self-attention layer 106. Paragraph 0029).
Regarding claim 1; Claim 1 contains substantially the same subject matter as 12. Therefore, claim 1 is rejected on the same grounds as claim 12.
Regarding claim 2; Claim 2 contains substantially the same subject matter as 13. Therefore, claim 2 is rejected on the same grounds as claim 13.
Regarding claim 7; Claim 7 contains substantially the same subject matter as 18. Therefore, claim 7 is rejected on the same grounds as claim 18.
Regarding claim 8; Claim 8 contains substantially the same subject matter as 19. Therefore, claim 8 is rejected on the same grounds as claim 19.
Regarding claim 9; Claim 9 contains substantially the same subject matter as 20. Therefore, claim 9 is rejected on the same grounds as claim 20.
Regarding claim 10; Claim 10 contains substantially the same subject matter as 21. Therefore, claim 10 is rejected on the same grounds as claim 21.
Regarding claim 11; Claim 11 contains substantially the same subject matter as 22. Therefore, claim 11 is rejected on the same grounds as claim 22.
Allowable Subject Matter
1. Claims 3-6 & 14-17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
2. Claims 4-6 depend on indicated objected claim 3. Therefore, by virtue of their dependency, Claims 4-6 are also indicated as objected subject matter.
3. Claims 15-17 depend on indicated objected claim 14. Therefore, by virtue of their dependency, Claims 15-17 are also indicated as objected subject matter.
Examiners Statement of Reasons for Allowance
The cited reference (Wang ‘688) teaches wherein embodiments described provide an Adapt-and-Adjust (A2) mechanism for multilingual speech recognition model that combines both adaptation and adjustment methods as an integrated end-to-end training to improve the models' generalization and mitigate the long-tailed issue. Specifically, a multilingual language model mBERT is utilized, and converted into an autoregressive transformer decoder. In addition, a cross-attention module is added to the encoder on top of the mBERT's self-attention layer in order to explore the acoustic space in addition to the text space. The joint training of the encoder and mBERT decoder can bridge the semantic gap between the speech and the text.
The cited reference (Galle ‘721) teaches wherein a multilingual neural machine translation systems having monolingual adapter layers and bilingual adapter layers for zero-shot translation include an encoder configured for encoding an input sentence in a source language into an encoder representation and a decoder configured for processing output of the encoder adapter layer to generate a decoder representation. The encoder includes an encoder adapter selector for selecting, from a plurality of encoder adapter layers, an encoder adapter layer for the source language to process the encoder representation. The decoder includes a decoder adapter selector for selecting, from a plurality of decoder adapter layers, a decoder adapter layer for a target language for generating a translated sentence of the input sentence in the target language from the decoder representation.
The cited references (Wang ‘688 & Galle ‘721) fail to disclose obtaining a fusion training data set; and training the first adapter and the second adapter based on the fusion training data set;
wherein training the first adapter and the second adapter based on the fusion training data set comprises tuning a plurality of fusion parameters added to the plurality of multi-head attention layers of the audio encoder; wherein tuning the plurality of fusion parameters comprises: obtaining a recurrent neural network-transducer (RNN-T) loss; and tuning, based on the RNN-T loss, the plurality of fusion parameters; and wherein parameters of the first adapter and the second adapter are frozen while tuning the plurality of fusion parameters. As a result, and for these reasons, Examiner indicates Claims 3-6 & 14-17 as allowable subject matter.
Relevant Prior Art References Not Relied Upon
1. Radford et al. (US 12,079,587 A1) - Disclosed herein are methods, systems, and computer-readable media for generating an output transcript from an input audio segment using a multi-task transformer model. In some embodiments, the transformer model can be trained to transcribe or translate audio data in multiple languages using labeled audio data. The labeled audio data can include first audio segments associated with first same-language transcripts of the first audio segments and second audio segments associated with second different-language transcripts of the second audio segments. In some embodiments, a vocabulary of the model can include special purpose and time stamp tokens. The special purpose tokens can specify tasks for the model to perform.
2. Wei et al. (US 12,033,618 B1) - Techniques for determining and storing relevant context information for a user input, such as a spoken input, are described. In some embodiments, context information is determined to be relevant on an audio frame basis. Context scores for different types of context data (e.g., prior dialog turn data, user profile data, device information, etc.) are determined for individual audio frames corresponding to a spoken input. Based on the corresponding context scores, the most relevant context is stored in a local context cache. The local context cache is updated as subsequent audio frames, of the user input, are processed. The data stored in the context cache is provided to downstream components to perform tasks such as ASR, NLU and SLU.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARCUS T. RILEY, ESQ. whose telephone number is (571)270-1581. The examiner can normally be reached 9-5 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached at 571-272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MARCUS T. RILEY, ESQ.
Primary Examiner
Art Unit 2654
/MARCUS T RILEY/Primary Examiner, Art Unit 2654
