DETAILED ACTION
This communication is in response to the Application filed on 09/16/2024. Claims 1-28 are pending and have been examined. Claims 1 and 15 are independent. This Application was published as U.S. Pub No. 20250095637.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 09/16/2024 was filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority
Applicant’s claims for benefit of a provisional application 63/584,051 submitted on 09/20/2023 is acknowledged.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-28 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al., (US Pub No. 2021/0350786, hereinafter, Chen) in view of Salaam et al. (US Pub No. 2023/0259718, hereinafter, Salaam) further in view of Li et al., ("Prefix-tuning: Optimizing continuous prompts for generation." Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021).
Regarding Claim 1,
Chen discloses a computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations (Chen, Fig.1, paras [034-035], "…an automated speech recognition (ASR) system 100 implementing an ASR model 200 that resides on a user device 102") comprising:
training a multilingual automatic speech recognition (ASR) model to learn how to recognize speech in the target language by injecting the unspoken textual utterance (Chen, Figs.3A-3C, par [046], "…training a generative adversarial network (GAN)-based text-to-speech (TTS) model 310 and a speech recognition model 200 in unison..."; par [062], "…The use of the unspoken text utterances 302a is to promote increases in linguistic diversity to enable training of the ASR model 200 on unseen words and sequences not present in the spoken training utterances 305...") into a text encoder associated with the multilingual ASR model (Chen, Figs.2A-2B, paras [036, 041], "…the ASR model 200 may include an end-to-end (E2E) sequence-to-sequence model, such as a frame alignment-based transducer model 200a (FIG. 2A) or an attention-based encoder-decoder (AED) model 200b (FIG. 2B)."; an encoder network 210 in Fig.2A and a listener encoding module 211 in Fig.2B).
Chen does not explicitly discloses the training of multilingual ASR by injecting synthetically generated code-switched textual content. However, Salaam, in the analogous field of semantic processing and machine learning to generation of code-switched text for training a machine learning model (Salaam, par [001]), discloses receiving a textual prompt in a first language (Salaam, Fig.2: Synthetic Data Generation Architecture 200, Fig.4, par [067], "…At step 404, the method 400 includes obtaining textual content in a first language...");
concatenating the textual prompt and the generated output text to provide an unspoken textual utterance (Salaam, Fig.4, par [073-075], "…At step 408, the method 400 includes translating the identified one or more portions to a second language using a second trained language model…"; par [075], "…At step 410, the method 400 includes reintegrating the translated one or more portions into the textual content to generate code-switched textual content...");
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified the speech recognition using unspoken text and speech synthesis of Chen with the generation of code-switched text for training a machine learning model (e.g. ASR) of Salaam with a reasonable expectation of success to adequately train the machine learning model with code-switching contents by creating synthetic code-switching textual content without relying on human-generated code-switched content (Salaam, paras [015-019]).
But, neither Chen nor Li explicitly discloses obtaining a fine-tuned prompt embedding to guide a large language model and processing the textual prompt conditioned on the prompt embedding.
Li, in the analogous field, discloses obtaining a fine-tuned prompt embedding configured to guide a large language model (LLM) to generate text in a target language from textual prompts in the first language (Li, Abstract, "…we propose prefix-tuning, which keeps language model parameters frozen and instead optimizes
a sequence of continuous task-specific vectors, which we call the prefix..."; Fig.1: prefix-tuning (bottom), e.g., Prefix (Translation); Fig.2: An annotated example of prefix-tuning, 4.2 Method, "…Prefix-tuning prepends a prefix for an autoregressive LM to obtain z = [PREFIX; x; y], or prepends prefixes for both encoder and decoder to obtain z =[PREFIX; x;PREFIX';y], as shown in Figure 2...");
processing, using the LLM, the textual prompt conditioned on the fine-tuned prompt embedding to generate output text in the target language (Li, Fig.2: An annotated example of prefix-tuning, 4 Prefix-Tuning, );
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified the training of multilingual speech recognition model using unspoken text and code-switched training text synthesis of Chen in view of Salaam with prefix-tuning for natural language generation tasks, which keeps language model parameters frozen taught by Li with a reasonable expectation of success to achieve a comparable performance with minimal parameter modification and outperform fine-tuning LLM in low-training data settings (Li, Abstract).
Regarding Claim 2,
The combination of Chen, Salaam, and Li discloses the computer-implemented method of claim 1, wherein the output text generated in the target language comprises monolingual text in the first language (Salaam, Fig.1, par [036], "…The replaced or reintegrated text 108 is at least a portion of a synthetic code-switched dataset can be used for training a language model (e.g., ASR)").
Regarding Claim 3,
The combination of Chen, Salaam, and Li discloses the computer-implemented method of claim 2, wherein the textual prompt comprises a prefix of a seed sentence in the first language (Salaam, Fig.2, par [038], "…the input includes source content 202 ( e.g., monolingual source text 102) from a source 201..."), the seed sentence sampled from a set of multilingual seed sentences, the set of multilingual seed sentences comprising a plurality of monolingual seed sentence subsets, each monolingual seed sentence subset comprising corresponding seed sentences in a respective language different than the respective language of the corresponding seed sentences of each other monolingual seed sentence subset (Salaam, Fig.2, par [044], "…the translation module 212 is configured to translate the extracted portions using a language model trained to translate between a first language ( e.g., the language of the monolingual source text 202) to a second, destination language that is different from the first language...").
Regarding Claim 4,
The combination of Chen, Salaam, and Li discloses the computer-implemented method of claim 2, wherein the fine-tuned prompt embedding is learned during a fine-tuning process by:
obtaining a randomly initialized trainable prompt embedding (Li, Fig.1: prefix-tuning (bottom), e.g., Prefix (Translation); Fig.2: An annotated example of prefix-tuning, 4.2 Method, "…Prefix-tuning prepends a prefix for an autoregressive LM to obtain z = [PREFIX; x; y], or prepends prefixes for both encoder and decoder to obtain z =[PREFIX; x;PREFIX';y], as shown in Figure 2..."; 7.4 Initialization, "…We find that how the prefix is initialized has a large impact in low-data settings. Random initialization leads to low performance with high variance….");
obtaining a multilingual training dataset comprising a plurality of training data subsets, each training data subset including corresponding monolingual training text utterances in a respective language that is different than the respective language of the corresponding monolingual training text utterances included in each other training data subset (Salaam, Fig.2, par [044], "…the translation module 212 is configured to translate the extracted portions using a language model trained to translate between a first language ( e.g., the language of the monolingual source text 202) to a second, destination language that is different from the first language...");
for each monolingual training text utterance: tokenizing the monolingual training utterance into a sequence of corresponding sub-word units; and processing, using the LLM, the sequence of corresponding sub-word units to determine a training loss that maximizes a probability of predicting a next sub-word unit based on each of the preceding sub-word units in the sequence of sub-word units (Li, Abstract, "…Prefix-tuning...allowing subsequent tokens to attend to this prefix as if it were “virtual tokens” (i.e., it is construed that tokenization process is performed)"; 4.1 Intuition, "…the context could influence the encoding of the task input x by guiding what to extract from x, and it could influence the generation of the task output y by steering the next token distribution...we can optimize the instruction as continuous word embeddings, whose effects will be propagated upward to all Transformer activation layers and rightward to subsequent tokens…"); and
fine-tuning, using the training losses, the randomly initialized trainable prompt embedding while parameters of the LLM are kept fixed (Li, 4 Prefix-Tuning, 4.2 Method, "…The training objective is the same as equation (2) (in the full fine-tuning framework, section 3.3 Fine-tuning) but the set of trainable parameters changes: the language model parameters f are fixed and the prefix parameters q are the only trainable parameters...";).
Regarding Claim 5,
The combination of Chen, Salaam, and Li discloses the computer-implemented method of claim 4, wherein: each corresponding training data subset of the plurality of training data subsets comprises one or more corresponding transcribed speech utterances each represented by a corresponding sequence of acoustic frames and paired with a corresponding transcription represented by a corresponding one of the monolingual training text utterances in the corresponding training data subset (Chen, Fig.3A, par [047], "…each spoken training utterance 305 in the set of spoken training utterances 305 includes a corresponding transcription 302b paired with a corresponding non-synthetic speech representation 304 of the corresponding spoken training utterance 305..."); and
training the multilingual speech recognition model further comprises training the multilingual speech recognition model on each of the one or more corresponding transcribed speech utterances in each corresponding training data subset of the plurality of training data subsets (Salaam, Fig.2: Synthetic Data Generation Architecture 200, par [046], "…The synthetic data 216 can be used to train a language model 220, (e.g., a multilingual ASR)..."; Li, Fig.3B, par [068], "…The synthetic speech representations 306 include unpaired synthetic speech representations 306a and paired synthetic speech representations 306b...while the paired synthetic speech representations 306b include the TTS audios converted by the GAN-based TTS model 310 from the transcriptions 302b in the set of spoken training utterances 305...").
Regarding Claim 6,
The combination of Chen, Salaam, and Li discloses the computer-implemented method of claim 1, wherein the output text generated in the target language comprises text in a second language different than the first language (Salaam, Fig.2, par [044], "…the translation module 212 is configured to translate the extracted portions using a language model trained to translate between a first language ( e.g., the language of the monolingual source text 202) to a second, destination language that is different from the first language...").
Regarding Claim 7,
The combination of Chen, Salaam, and Li discloses the computer-implemented method of claim 6, wherein the textual prompt comprises a prefix of a seed sentence in the first language, the seed sentence sampled from a set of code-mixed seed sentences, each code-mixed seed sentence comprising corresponding code-mixed text in both the first language and the second language (Li, Fig.1: prefix-tuning (bottom), e.g., Prefix (Translation); Fig.2: An annotated example of prefix-tuning, 4.2 Method, "…Prefix-tuning prepends a prefix for an autoregressive LM to obtain z = [PREFIX; x; y], or prepends prefixes for both encoder and decoder to obtain z =[PREFIX; x;PREFIX';y], as shown in Figure 2..."; i.e., the task input x is construed to be seed sentences in the first/second or code-mixed text).
Claim 8 is a system claim with limitations similar to the limitations of Claim 4 and is rejected under similar rationale. Additionally,
obtaining a code-mixed training dataset comprising a plurality of code-mixed training text utterances that each comprise code-mixed text in the first language and the second language (Salaam, Fig.2, par [044], "…the translation module 212 is configured to translate the extracted portions using a language model trained to translate between a first language ( e.g., the language of the monolingual source text 202) to a second, destination language that is different from the first language...");
Claim 9 is a system claim with limitations similar to the limitations of Claim 5 and is rejected under similar rationale.
Regarding Claim 10,
The combination of Chen, Salaam, and Li discloses the computer-implemented method of claim 1, wherein the LLM is pre-trained on a diverse range of text data sourced from web documents, books, and code (ㅣLi,5.3 Architectures and Hyperparameters, "…we use GPT-2MEDIUM and GPT-2LARGE... we use BARTLARGE...").
Regarding Claim 11,
The combination of Chen, Salaam, and Li discloses the computer-implemented method of claim 1, wherein training the multilingual ASR model to learn how to recognize speech in the target language by injecting the unspoken textual utterance into the text encoder associated with the multilingual ASR model comprises:
tokenizing the unspoken textual utterance into a sequence of sub-word units (Chen, Figs. 2A-2B, par [036-039], "…the ASR model 200 may include an end-to-end (E2E) sequence-to-sequence model, such as a frame alignment-based transducer model 200a ( FIG. 2A) or an attention-based encoder-decoder (AED) model 200b (FIG. 2B)..."; Fig.3B, par [068], "…the training process 300b trains the ASR model 200 on the synthetic speech representations 306 generated at each of the plurality of output steps for each unspoken training text utterance 302 of the plurality of unspoken text utterances 302..." );
generating, by the text encoder of an encoder, at each of a plurality of output steps, a first higher order textual feature representation for a corresponding sub-word unit in the sequence of sub-word units tokenized from the unspoken textual utterance (Chen, Fig. 2A, par [037], "…The RNN-T model 200a includes an encoder network 210, a prediction network 220, and a joint network 230. The encoder network 210, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, includes a recurrent network of stacked Long Short-Term Memory (LSTM) layers...");
receiving, as input to a first-pass decoder, the first higher order textual feature representation generated by the text encoder at each of the plurality of output steps (Chen, Fig. 2A, par [038], "…the prediction network 220 is also an LSTM network, which, like a language model (LM), processes the sequence of non-blank symbols output..."); and
generating, by the first-pass decoder, at each of the plurality of output steps, a first probability distribution over possible text units (Chen, Fig.2A, par [038], "…with the RNN-T model architecture, the representations produced by the encoder and prediction networks 210, 220 are combined by the joint network 230. The joint network then predicts a probability distribution over possible speech recognition hypotheses.
"); and
training the encoder based on the first probability distribution over possible text units generated by the first-pass decoder at each of the plurality of output steps for the unspoken textual utterance (Chen, Fig.3B, paras [068-074], "…the training process 300b trains the ASR model 200 on the synthetic speech representations 306 generated at each of the plurality of output steps for each unspoken training text utterance 302 of the plurality of unspoken text utterances 302...").
Regarding Claim 12,
The combination of Chen, Salaam, and Li discloses the computer-implemented method of claim 11, wherein the operations further comprise:
receiving, as input to a non-causal audio-text encoder of the encoder, the first higher order textual feature representation generated by the text encoder at each of the plurality of output steps (Chen, Figs. 2A-2B, par [036-039], "…the ASR model 200 may include an end-to-end (E2E) sequence-to-sequence model, such as a frame alignment-based transducer model 200a ( FIG. 2A) or an attention-based encoder-decoder (AED) model 200b (FIG. 2B)..."; Fig.3B, par [068], "…the training process 300b trains the ASR model 200 on the synthetic speech representations 306 generated at each of the plurality of output steps for each unspoken training text utterance 302 of the plurality of unspoken text utterances 302..." );
generating, by the non-causal audio-text encoder, at each of the plurality of output steps, a second higher order textual feature representation for a corresponding first higher order textual feature representation (Chen, Fig. 2A, par [037], "…The RNN-T model 200a includes an encoder network 210, a prediction network 220, and a joint network 230. The encoder network 210, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, includes a recurrent network of stacked Long Short-Term Memory (LSTM) layers...");
receiving, as input to a second-pass decoder, the second higher order textual feature representation generated by the non-causal audio-text encoder at each of the plurality of output steps (Chen, Fig. 2A, par [038], "…the prediction network 220 is also an LSTM network, which, like a language model (LM), processes the sequence of non-blank symbols output..."); and
generating, by the second decoder, at each of the plurality of output steps, a second probability distribution over possible text units, wherein training the encoder is further based on the second probability distribution over possible text units generated by the second-pass decoder at each of the plurality of output steps for the unspoken textual utterance (Chen, Fig.2A, par [038], "…with the RNN-T model architecture, the representations produced by the encoder and prediction networks 210, 220 are combined by the joint network 230. The joint network then predicts a probability distribution over possible speech recognition hypotheses...").
Regarding Claim 13,
The combination of Chen, Salaam, and Li discloses the computer-implemented method of claim 12, wherein the first-pass decoder and the second-pass decoder comprise a same decoder (Chen, Fig.2A-2B: Prediction network 220, Decoder 231)
Regarding Claim 14,
The combination of Chen, Salaam, and Li discloses the computer-implemented method of claim 12, wherein the non-causal audio- text encoder comprises one of: a plurality of unidirectional long short-term memory (LSTM) layers; a plurality of conformer layers; or a plurality of transformer layers (Chen, paras [036-040], "…the ASR model 200 may include an end-to-end (E2E) sequence-to-sequence model, such as a frame alignment-based transducer model 200a ( FIG. 2A) or an attention-based encoder-decoder (AED) model 200b (FIG. 2B)...an example frame alignment based transducer model 200a includes a Recurrent Neural Network-Transducer (RNN-T) model architecture…The encoder network 210, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, includes a recurrent network of stacked Long Short-Term Memory (LSTM) layers….").
Claim 15 is a system claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale. Additionally,
Chen discloses a system comprising: data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising (Chen, Fig.6, par [089], "…The computing device 600 includes a processor 610, memory 620, a storage device 630..."):
…
Rationale for combination is similar to that provided for Claim 1.
Claim 16 is a system claim with limitations similar to the limitations of Claim 2 and is rejected under similar rationale.
Claim 17 is a system claim with limitations similar to the limitations of Claim 3 and is rejected under similar rationale.
Claim 18 is a system claim with limitations similar to the limitations of Claim 4 and is rejected under similar rationale.
Claim 19 is a system claim with limitations similar to the limitations of Claim 5 and is rejected under similar rationale.
Claim 20 is a system claim with limitations similar to the limitations of Claim 6 and is rejected under similar rationale.
Claim 21 is a system claim with limitations similar to the limitations of Claim 7 and is rejected under similar rationale.
Claim 22 is a system claim with limitations with limitations similar to the limitations of Claim 8 and is rejected under similar rationale.
Claim 23 is a system claim with limitations with limitations similar to the limitations of Claim 9 and is rejected under similar rationale.
Claim 24 is a system claim with limitations with limitations similar to the limitations of Claim 10 and is rejected under similar rationale.
Claim 25 is a system claim with limitations with limitations similar to the limitations of Claim 11 and is rejected under similar rationale.
Claim 26 is aa system claim with limitations with limitations similar to the limitations of Claim 12 and is rejected under similar rationale.
Claim 27 is a system claim with limitations with limitations similar to the limitations of Claim 13 and is rejected under similar rationale.
Claim 28 is a system claim with limitations with limitations similar to the limitations of Claim 14 and is rejected under similar rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Wang et al., (US Pub No. 2022/0108688, hereinafter, Wang) discloses an Adapt-and-Adjust (A2) mechanism for multilingual speech recognition model that combines both adaptation and adjustment methods as an integrated end-to-end training to improve the models' generalization and mitigate the long-tailed issue.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JANGWOEN LEE whose telephone number is (703)756-5597. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, BHAVESH MEHTA can be reached at (571)272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JANGWOEN LEE/Examiner, Art Unit 2656
/BHAVESH M MEHTA/Supervisory Patent Examiner, Art Unit 2656
Prosecution Insights