CTNF 18/826,743 CTNF 80314 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Information Disclosure Statement The information disclosure statement (IDS) submitted on 02/21/2025 was filed in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner. Claim Rejections - 35 USC § 101 Claims 1-10 are drawn to a " software " per se “automated speech recognition model” and as such is non-statutory subject matter. See MPEP § 2106.1V.B.1 .a. Data structures not claimed as embodied in computer readable media are descriptive material per se and are not statutory because they are not capable of causing functional change in the computer. See, e.g., Warmerdam, 33 F.3d at 1361, 31 USPQ2d at 1760 (claim to a data structure per se held nonstatutory). Such claimed data structures do not define any structural and functional interrelationships between the data structure and other claimed aspects of the invention, which permit the data structure's functionality to be realized. In contrast, a claimed computer readable medium encoded with a data structure defines structural and functional interrelationships between the data structure and the computer software and hardware components which permit the data structure's functionality to be realized, and is thus statutory. Similarly, computer programs claimed as computer listings per se, i.e., the descriptions or expressions of the programs are not physical "things." They are neither computer components nonstatutory processes, as they are not "acts" being performed. Such claimed computer programs do not define any structural and functional interrelationships between the computer program and other claimed elements of a computer, which permit the computer program's functionality to be realized. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim (s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sainath et al (WO 2022/203698) in view of Wenxin Hou et al. (Exploiting Adapters for Cross-lingual Low-resource Speech Recognition) . Regarding Claim 1, Sainath et al teaches streaming and non-streaming automated speech recognition (ASR) model (FIGS. 2A-2C include example models 200a-c operating in various combinations of streaming and non-streaming modes) (Page 13, paragraph [0030]), the ASR model comprising: a causal encoder (Fig. 2A-2C, first encoder 210) comprising an initial stack of multi-head attention layers (Here, the encoders 210, 220 can be cascaded irrespective of the underlying architecture for each encoder. In some examples, the encoders 210, 220 include a stack of 512-dimension conformer layers. Causal convolution and left-context attention layers may be used for each conformer layer to strictly restrict the model use no future inputs. A multi-headed (e.g., 8 heads) attention mechanisms may be used in a self-attention layer. The cascades encoders 210, 220 may include 17 conformer layers. Here, the causal encoder 210 may include 15 conformer layers while the non-causal encoder 210 may include two conformer layers that take in additional right context (e.g., 5.04 seconds). Optionally, transformer layers may be used in lieu of conformer layers) (page 13, paragraph [0030]), the causal encoder configured to: receive, as input, a sequence of acoustic frames characterizing a spoken utterance in a particular native language (Fig. 2C, X t ) (Fig. 4, Step 402) (At operation 402, the method 400 includes receiving, as input to a cascaded encoders model 200, a sequence of acoustic frames 110) (pages 22, paragraph [0051]); and generate, at each of a plurality of output steps, a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames (Fig. 4, Step 406) (At operation 406, the method 400 includes generating, by a first encoder 210, at each of a plurality of output steps, a first higher order feature representation for a corresponding acoustic frame 110 in the sequence of acoustic frames 110) (pages 22-23, paragraph [0052]); a non-causal encoder (Fig. 2C, second encoder 220) comprising a final stack of multi-head attention layers (A multi-headed (e.g., 8 heads) attention mechanisms may be used in a self-attention layer) overlain on the initial stack of multi-head attention layers (In some examples, the encoders 210, 220 include a stack of 512-dimension conformer layers. Causal convolution and left-context attention layers may be used for each conformer layer to strictly restrict the model use no future inputs. A multi-headed (e.g., 8 heads) attention mechanisms may be used in a self-attention layer. The cascades encoders 210, 220 may include 17 conformer layers. Here, the causal encoder 210 may include 15 conformer layers while the non-causal encoder 210 may include two conformer layers that take in additional right context (e.g., 5.04 seconds). Optionally, transformer layers may be used in lieu of conformer layers) (page 13, paragraph [0030]) (Since the encoders consist of stacks of conformer layers, it is being interpreted by the examiner that the stacks of the layers are overlaid on each other.), the non-causal encoder configured to: receive, as input, the first higher order feature representation generated by the causal encoder at each of the plurality of output steps (Fig. 4, Step 408) (The method 400 further includes, at operation 408, receiving, as input to a second encoder 220, the first higher order feature representation generated by the first encoder 210 at each of the plurality of output steps) (pages 22-23, paragraph [0052]); and generate, at each of the plurality of output steps, a second higher order feature representation for a corresponding first higher order feature representation (Fig. 4, Step 410) (At operation 410, the method 400 also includes generating, by the second encoder 220, at each of the plurality of output steps, a second higher order feature representation for a corresponding first higher order feature frame) (pages 22-23, paragraph [0052]); and a decoder (Fig. 2A-2C, decoder 204) configured to: receive, as input, the second higher order feature representation generated by the non-causal encoder at each of the plurality of output steps (Fig. 4, Step 412), (The method 400 also includes, at operation 412, receiving, as input to a decoder 204, the second higher order feature representation generated by the second encoder 220 at each of the plurality of output steps) (pages 22-23, paragraph [0052]); and generate, at each of the plurality of output steps, a probability distribution over possible speech recognition hypotheses in the particular native language (Figure 4, Step 414) (At operation 414, the method 400 further includes generating, at each of the plurality of time steps, a first probability distribution over possible speech recognition hypothesis and then rescoring, using an external language model 206, the first probability distribution over possible speech recognition hypothesis to generate a transcription 120 of the utterance 106) (pages 22-23, paragraph [0052]). Sainath et al fails to teach a multilingual automated speech recognition (ASR) model comprising: a plurality of language-dependent adapter (LDA) modules each comprising corresponding sets of language-dependent weights each specific to a different native language, wherein each corresponding LDA module: is inserted between two consecutive multi-head attention layers in the causal encoder or the non-causal encoder; and configured to receive, as input, a language ID vector identifying the particular native language to activate the corresponding language-dependent weights specific to the particular native language. Wenxin Hou et al. teaches a multilingual automated speech recognition (ASR) model (Multilingual and Cross-Lingual Speech Recognition) (Section II, A) comprising: a plurality of language-dependent adapter (LDA) modules (To exploit adapters for cross-lingual ASR, it is important to study the relationship between different languages. In this paper, we comprehensively analyze the MetaAdapter as well as the newly proposed SimAdapter algorithms that learn and exploit the inter-language relationships to improve cross-lingual ASR) (Section III, B) each comprising corresponding sets of language-dependent weights each specific to a different native language (Fig. 5. Illustration of the SimAdapter module. The language-specific features a Lk of different languages L k ∈ {L 1 ,L 2 ,...,L N } are attended by the language-agnostic features z to extract better features for the target language) (Section V, A), wherein each corresponding LDA module: is inserted between two consecutive multi-head attention layers in the causal encoder or the non-causal encoder (Fig. 2. Illustration of the MetaAdapter and SimAdapter module injected in the Speech-Transformer. Note that the residual connection between the feed-forward layer and layer normalization only applies to the SimAdapter) (Layers between Self -attention and CTC Head in Fig. 2); and configured to receive, as input, a language ID vector identifying the particular native language to activate the corresponding language-dependent weights specific to the particular native language (As model outputs, a shared vocabulary including characters/subwords and language tokens (e.g., , ) of 42 languages is adopted to realize language-independent training and recognizing. Furthermore, a language token is inserted to the beginning of each training label as an auxiliary language identification target. The model is trained to firstly identify the language before recognizing the speech contents. It is worth noting that we focus on monolingual transfer in this work. Therefore, language-specific heads are used and the language identification objective is dropped during fine-tuning) (Section III, C). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of Sainath with the teachings of Wenxin Hou to improve multilingual and cross-lingual speech recognition by transferring the knowledge from the existing languages to the new language. Regarding Claim 2, Sainath et al fails to teach the ASR model, wherein each corresponding multi-head attention layer subsequent to an initial multi-head attention layer in the initial stack of multi-head attention layers is configured to receive a concatenation of an output from a previous multi-head attention layer and output of the corresponding LDA module inserted between the corresponding multi-head attention layer and the previous multi-head attention layer. Wenxin Hou et al. teaches the ASR model, wherein each corresponding multi-head attention layer subsequent to an initial multi-head attention layer in the initial stack of multi-head attention layers (Fig. 2. Illustration of the MetaAdapter and SimAdapter module injected in the Speech-Transformer. Note that the residual connection between the feed-forward layer and layer normalization only applies to the SimAdapter) (Layers between Self -attention and CTC Head in Fig. 2) is configured to receive a concatenation of an output from a previous multi-head attention layer and output of the corresponding LDA module inserted between the corresponding multi-head attention layer and the previous multi-head attention layer (Fig. 2. Illustration of the MetaAdapter and SimAdapter module injected in the Speech-Transformer. Note that the residual connection between the feed-forward layer and layer normalization only applies to the SimAdapter) (Layers between Self -attention and CTC Head in Fig. 2). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of Sainath with the teachings of Wenxin Hou to improve multilingual and cross-lingual speech recognition by transferring the knowledge from the existing languages to the new language. Regarding Claim 3, Sainath et al teaches the ASR model, wherein the decoder is further configured to: receive, as input, the first higher order feature representation generated by the causal encoder (At operation 406, the method 400 includes generating, by a first encoder 210, at each of a plurality of output steps, a first higher order feature representation for a corresponding acoustic frame 110 in the sequence of acoustic frames 110. The method 400 further includes, at operation 408, receiving, as input to a second encoder 220, the first higher order feature representation generated by the first encoder 210 at each of the plurality of output steps) (Fig. 4, steps 406-408) (pages 22-23, paragraph [0052]) at each of the plurality of output steps (Fig. 4, Step 412), (The method 400 also includes, at operation 412, receiving, as input to a decoder 204, the second higher order feature representation generated by the second encoder 220 at each of the plurality of output steps) (pages 22-23, paragraph [0052]); and generate, at each of the plurality of output steps, a second probability distribution over possible speech recognition hypotheses in the particular native language (The ASR system (109) of claim 1 or 2, wherein the decoder (204) is further configured to: receive, as input, the first higher order feature representation generated by the first encoder (210) at each of the plurality of output steps; and generate, at each of the plurality of time steps, a second probability distribution over possible speech recognition hypothesis (120a)) (Claim 3). Regarding Claim 4, Sainath et al teaches the ASR model, wherein the decoder comprises: a prediction network configured to, at each of the plurality of output steps: receive, as input, a sequence of N previous non-blank symbols (In some examples, the N previous non-blank sub-word unit predictions is equal to the last five non-blank sub word unit predictions) (page 14, paragraph [0033]) output by a final softmax layer (Although not illustrated, the model 200 may include a Softmax layer that receives output of the decoder 204) (pages 15 and 16, paragraph [0036]); for each non-blank symbol of the sequence of N previous non-blank symbols, generate a respective embedding (The V2 embedding lookup table 240, given N previous non-blank sub-word unit predictions yi-.sub.h . . . , y, v, computes the embedding of each of these outputs as { di , ch, ...d.sub.n)) (page 14, paragraph [0033]); and generate an average embedding by averaging the respective embeddings (The V2 embedding lookup table 240 then computes and outputs an average d of the embeddings {di, d2, ...d.sub.n) to a projection layer 242 with SWISH activation to produce output / provided to the joint layer 230) (page 14, paragraph [0033]); and a joint network (joint network 230) configured to: receive, as input, the average embedding generated by the prediction network at each of the plurality of output steps (The V2 embedding lookup table 240 then computes and outputs an average d of the embeddings {di, d2, ...d.sub.n) to a projection layer 242 with SWISH activation to produce output / provided to the joint layer 230) (page 14, paragraph [0033]) and one of: when the ASR model is operating in a streaming mode, the first higher order feature representation generated by the causal encoder at each of the plurality of output steps (Here, the model 200b performs streaming speech recognition on the audio data 110 using only the first encoder 210 to generate the first higher order representation e.sup.s for the decoder 204) (page 18, paragraph [0042]); or when the ASR model is operating in a non-streaming mode, the second higher order representation generated by the non-causal encoder at each of the plurality of output steps (Since only one limitation needs to be addressed, the other limitation will not be addressed at this time); and generate, at each of the plurality of output steps, one of: when the ASR model is operating in the streaming mode, the second probability distribution over possible speech recognition hypotheses (The ASR system (109) of claim 1 or 2, wherein the decoder (204) is further configured to: receive, as input, the first higher order feature representation generated by the first encoder (210) at each of the plurality of output steps; and generate, at each of the plurality of time steps, a second probability distribution over possible speech recognition hypothesis (120a)) (Claim 3); or when the ASR model is operating in the non-streaming mode, the probability distribution over possible speech recognition hypotheses (Since only one limitation needs to be addressed, the other limitation will not be addressed at this time). Regarding Claim 5, Sainath et al teaches the ASR model, wherein the prediction network comprises a V2 embedding look-up table (The prediction network may include a V2 embedding look-up table (pages 4 and 5, paragraph [0008]). Regarding Claim 6, Sainath et al teaches the ASR model, wherein the multi-head attention layers in the initial and the final stacks of multi-head attention layers comprise conformer layers (In some examples, the encoders 210, 220 include a stack of 512-dimension conformer layers. Causal convolution and left-context attention layers may be used for each conformer layer to strictly restrict the model use no future inputs. A multi-headed (e.g., 8 heads) attention mechanisms may be used in a self-attention layer. The cascades encoders 210, 220 may include 17 conformer layers) (page 13, paragraph [0030]). Regarding Claim 7, Sainath et al teaches the ASR model, wherein the initial stack of multi-head attention layers comprises a greater number of multi-head attention layers than the final stack of multi- head attention layers (In some examples, the cascading encoders 202 are composed of a stack of conformer layers. For instance, the first causal encoder 210 may include an initial stack of 15 conformer layers, while the second non-causal encoder 220 may include two additional conformer layers on top of the initial stack of 15 conformer layers) (pages 16-17, paragraph [0038]), . Regarding Claim 9, Sainath et al fails to teach the ASR model, wherein each multi-head attention layer in the initial and the final stacks of multi-head attention layers is followed by a corresponding LDA module. Wenxin Hou et al. teaches the ASR model, wherein each multi-head attention layer in the initial and the final stacks of multi-head attention layers is followed by a corresponding LDA module (Fig. 2. Illustration of the MetaAdapter and SimAdapter module injected in the Speech-Transformer. Note that the residual connection between the feed-forward layer and layer normalization only applies to the SimAdapter) (Layers between Self -attention and CTC Head in Fig. 2). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of Sainath with the teachings of Wenxin Hou to improve multilingual and cross-lingual speech recognition by transferring the knowledge from the existing languages to the new language. Regarding Claim 10, Sainath et al fails to teach the ASR model, wherein each corresponding LDA module further comprises :a layer norm layer; a down-projection layer; a Rectified Linear Unit (ReLU) layer; an up-projection layer; and a residual connection. Wenxin Hou et al. teaches the ASR model, wherein each corresponding LDA module further comprises: a layer norm layer; a down-projection layer; a Rectified Linear Unit (ReLU) layer; an up-projection layer; and a residual connection (Fig. 3. Architecture of the adapter module) (Section III, D). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of Sainath with the teachings of Wenxin Hou to improve multilingual and cross-lingual speech recognition by transferring the knowledge from the existing languages to the new language. Regarding Claim 11, Sainath et al teaches a computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising: receiving, as input to a streaming and non-streaming automatic speech recognition (ASR) model, a sequence of acoustic frames characterizing a spoken utterance in a particular native language (Fig. 2C, X t ) (Fig. 4, Step 402) (At operation 402, the method 400 includes receiving, as input to a cascaded encoders model 200, a sequence of acoustic frames 110) (pages 22, paragraph [0051]); generating, by a causal encoder of the ASR model comprising an initial stack of multi-head attention layers, at each of a plurality of output steps, a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames (Fig. 4, Step 406) (At operation 406, the method 400 includes generating, by a first encoder 210, at each of a plurality of output steps, a first higher order feature representation for a corresponding acoustic frame 110 in the sequence of acoustic frames 110) (pages 22-23, paragraph [0052]); generating, by a non-causal encoder of the ASR model comprising a final stack of multi-head attention layers overlain on the initial stack of multi-head attention layers, at each of the plurality of output steps, a second higher order feature representation for a corresponding first higher order feature representation (In some examples, the cascading encoders 202 are composed of a stack of conformer layers. For instance, the first causal encoder 210 may include an initial stack of 15 conformer layers, while the second non-causal encoder 220 may include two additional conformer layers on top of the initial stack of 15 conformer layers) (pages 6 and 17), paragraph [0033]); and generating, by a decoder of the ASR model (Fig. 4, Step 412), (The method 400 also includes, at operation 412, receiving, as input to a decoder 204, the second higher order feature representation generated by the second encoder 220 at each of the plurality of output steps) (pages 22-23, paragraph [0052]), at each of the plurality of output steps, a first probability distribution over possible speech recognition hypotheses in the particular native language based on a corresponding second higher order feature representation generated the non-causal encoder (Figure 4, Step 414) (At operation 414, the method 400 further includes generating, at each of the plurality of time steps, a first probability distribution over possible speech recognition hypothesis and then rescoring, using an external language model 206, the first probability distribution over possible speech recognition hypothesis to generate a transcription 120 of the utterance 106) (pages 22-23, paragraph [0052]). Sainath et al fails to teach a multilingual automatic speech recognition (ASR) model comprising: receiving, as input at each corresponding language-dependent adapter (LDA) module of a plurality of LDA modules, a language ID vector, each corresponding LDA module comprising corresponding sets of language-dependent weights each specific to a different native language and is inserted between two consecutive multi-head attention layers in the causal encoder or the non-causal encoder, the language ID vector identifying the particular native language to activate the corresponding language-dependent weights specific to the particular native language; Wenxin Hou et al. teaches a multilingual automatic speech recognition (ASR) model (Multilingual and Cross-Lingual Speech Recognition) (Section II, A) comprising: receiving, as input at each corresponding language-dependent adapter (LDA) module of a plurality of LDA modules (To exploit adapters for cross-lingual ASR, it is important to study the relationship between different languages. In this paper, we comprehensively analyze the MetaAdapter as well as the newly proposed SimAdapter algorithms that learn and exploit the inter-language relationships to improve cross-lingual ASR) (Section III, B), a language ID vector (As model outputs, a shared vocabulary including characters/subwords and language tokens (e.g., , ) of 42 languages is adopted to realize language-independent training and recognizing. Furthermore, a language token is inserted to the beginning of each training label as an auxiliary language identification target. The model is trained to firstly identify the language before recognizing the speech contents. It is worth noting that we focus on monolingual transfer in this work. Therefore, language-specific heads are used and the language identification objective is dropped during fine-tuning) (Section III, C), each corresponding LDA module comprising corresponding sets of language-dependent weights each specific to a different native language (Fig. 5. Illustration of the SimAdapter module. The language-specific features a Lk of different languages L k ∈ {L 1 ,L 2 ,...,L N } are attended by the language-agnostic features z to extract better features for the target language) (Section V, A) and is inserted between two consecutive multi-head attention layers in the causal encoder or the non-causal encoder (Fig. 2. Illustration of the MetaAdapter and SimAdapter module injected in the Speech-Transformer. Note that the residual connection between the feed-forward layer and layer normalization only applies to the SimAdapter), the language ID vector identifying the particular native language to activate the corresponding language-dependent weights specific to the particular native language (As model outputs, a shared vocabulary including characters/subwords and language tokens (e.g., , ) of 42 languages is adopted to realize language-independent training and recognizing. Furthermore, a language token is inserted to the beginning of each training label as an auxiliary language identification target. The model is trained to firstly identify the language before recognizing the speech contents. It is worth noting that we focus on monolingual transfer in this work. Therefore, language-specific heads are used and the language identification objective is dropped during fine-tuning) (Section III, C). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teachings of Sainath with the teachings of Wenxin Hou to improve multilingual and cross-lingual speech recognition by transferring the knowledge from the existing languages to the new language. Claim 12 is rejected for the same reason as claim 2. Claim 13 is rejected for the same reason as claim 3. Claim 14 is rejected for the same reason as claim 4. Claim 15 is rejected for the same reason as claim 5. Claim 16 is rejected for the same reason as claim 6. Claim 17 is rejected for the same reason as claim 7. Claim 19 is rejected for the same reason as claim 9. Claim 20 is rejected for the same reason as claim 10 . Allowable Subject Matter 07-43 Claim 8 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and if the 35 USC 101 rejection above is overcome. 12-151-08 AIA 07-43 12-51-08 Claim 18 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Cited Art 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Zhang et al. (US 2024/0135923) discloses universal monolingual output layer for multilingual speech recognition. Le et al. (US 12,087,306) discloses contextualized streaming end-to-end speech recognition with trie-based deep learning and shallow fusion. Stooke et al. (US 12,431,122) discloses training a language model of an end-to-end automatic speech recognition model using random encoder features . Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to SATWANT K SINGH whose telephone number is (571)272-7468. The examiner can normally be reached Monday thru Friday 9:00 AM to 6:00 PM EST. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras D Shah can be reached at (571}270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /SATWANT K SINGH/Primary Examiner, Art Unit 2653 Application/Control Number: 18/826,743 Page 2 Art Unit: 2653 Application/Control Number: 18/826,743 Page 3 Art Unit: 2653 Application/Control Number: 18/826,743 Page 4 Art Unit: 2653
Prosecution Insights