Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 12/5/2024 is being considered by the examiner.
Claim Objections
Claim 13 is objected to because of the following informalities:
Claim 13 recites the limitation “which is a state hidden in an audio feature of an utterance speech” at line(s)3-4, where the phrase “which is” is unclear as to what claim part, either the prompt or the accent feature, is being further modified by the clause. It is noted that both the prompt and the accent feature are derived from the audio features (See Instant Application, as published, at [0093]-[0094]).
As it is believed that applicant intended to modify the accent feature, the following proposed amendment, if acceptable to the applicant, would overcome the objection above: “…wherein the accent feature .
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 12, 14 and 15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the
subject matter which the inventor or a joint inventor (or for applications subject to pre-
AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, and mutatis mutandis claim 7, the claim parts “prompt”, “first accent feature” and “second accent feature” as presented in the claim, lacks clarity. Claim 1 recites “a prompt generator for extracting a first accent feature from a prompt concatenation input that concatenates the prompt to the audio feature… and being adversarially trained to minimize interdependence between the first accent feature and the second accent feature to generate a prompt from the audio feature” at lines 5-9 of claim 1. It is noted that the prompt generator extracts the first accent feature from a prompt concatenation input, where the prompt concatenation input is a concatenation of the “prompt to the audio feature”. Further, the prompt generator is “minimiz[ing] interdependence between the first accent feature and the second accent feature to generate a prompt from the audio feature” then the first accent feature and the second accent feature are at least used as part of generating the prompt (one cannot minimize the interdependence between components without somehow using those components). As such, the claim as drafted creates an apparent paradox. It is unclear how the prompt generator is extracting a first accent feature from the prompt which does not exist until the interdependence between the first accent feature and second accent feature is minimized for generation of said prompt.
Regarding the claim part “prompt”, “the prompt” in the second limitation is followed by “a prompt” in the third limitation. The relationship between the claim parts is unclear and creates ambiguity as to whether they are intended to be the same prompt or different prompts.
Regarding the limitation “a prompt generator for extracting a first accent feature from a prompt concatenation input that concatenates the prompt to the audio feature…” It is unclear whether the phrase “that concatenates the prompt to the audio feature” is a description of the “prompt concatenation input” or an action being performed by the “prompt generator”. Therefore, claims 1 and 7 lack clarity and are rejected under 112(b) as being indefinite.
Further regarding claims 1, and mutatis mutandis claim 7, claim 1 recites the limitation "the prompt" in line 6. There is insufficient antecedent basis for this limitation in the claim.
Regarding claim 14, the claim is indefinite because the claim lacks a proper preamble identifying the statutory class of the claim, which renders both subject matter and numerical dependency of the claim unclear.
Regarding claim 15, the claim is indefinite because the dependency of the claim is unclear. Specifically claim 15, depends from claim 14, which does not properly identify the claim type, thereby rendering the scope of claim 15 uncertain.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-15 are rejected under 35 U.S.C. 102 as being anticipated by Yoon (Yoon, Eunseop, et al. "INTapt: Information-theoretic adversarial prompt tuning for enhanced non-native speech recognition." Findings of the Association for Computational Linguistics: ACL 2023. 2023., hereinafter Yoon).
Regarding claim 1, Yoon discloses a model training device operated by at least one processor, the model training device comprising: an accent module trained to extract an accent feature from an audio feature of an utterance speech; and a prompt generator for extracting a first accent feature from a prompt concatenation input that concatenates the prompt to the audio feature, and a second accent feature from the audio feature by using the accent module, and being adversarially trained to minimize interdependence between the first accent feature and the second accent feature to generate a prompt from the audio feature.
[Page 9894, Section 2 “In the first step, we train an Accent Module (AM) capable of isolating the accent feature from a given audio feature a of an input speech x. In the second step, we train a Prompt Generator (PG), which out puts a prompt p for a given audio feature a, using two objectives: (1) Minimize the mutual information between the accent feature z′ and z, where the former is obtained using the prompt-concatenated input ( p; a) and the latter is obtained from the original audio feature a, (2) Minimize CTC loss to improve the ASR performance of the input ( p; a)];
Regarding claim 2, the rejection of claim 1 is incorporated. Yoon discloses all limitations of claim 1 as described above. Yoon further discloses wherein: the prompt generator is trained to minimize a Connectionist Temporary Classification (CTC) loss of a speech recognition model that outputs text from the prompt concatenation input.
[Page 9894, Section 2 “In the second step, we train a Prompt Generator (PG), which out puts a prompt p for a given audio feature a, using two objectives:… (2) Minimize CTC loss to improve the ASR performance of the input (p;a)”]
Regarding claim 3, the rejection of claim 2 is incorporated. Yoon discloses all the limitations of claim 1 as described above. Yoon further discloses wherein: the accent module includes: an accent feature extractor that is trained with an accent classification head that isolates an accent feature of a given speech and extracts the accent feature from the audio feature; and an accent intensity regression head for predicting the CTC loss to capture an accent intensity by using the accent feature extracted by the accent feature extractor.
[Page 9894, Section 2.1: Accent Classification Head “The role of the accent classification head fθ2 is to isolate the accent feature of a given speech 1”]; [Page 9895, Section Accent Intensity Regression Head “Thus, an accent intensity regression head is introduced to incorporate the accent intensity into the obtained accent feature z. Based on the assumption that the intensity of the accent affects ASR performance, making the accent intensity regression head predict the CTC loss 2, obtained by inputting the corresponding speech into the back bone speech model, will allow the extracted accent feature z to capture the intensity of the accent.”].
Regarding claim 4, the rejection of claim 3 is incorporated. Yoon discloses all the limitations of claim 1 as described above. Yoon further discloses wherein: the accent feature extractor extracts the accent feature from a hidden state of the audio feature acquired through the speech recognition model.
[Page 9894, Section: Accent classification Head “Given the hidden state representation h of an audio feature input a, the feature extractor outputs the accent feature (i.e., z =fθ1 (h)) and the accent classification head fθ2 tries to assign it to the correct accent label y”].
Regarding claim 5, the rejection of claim 4 is incorporated. Yoon discloses all the limitations of claim 1 as described above. Yoon further discloses further comprising: a mutual information neural estimator that estimates the interdependence by using a neural network.
[Page 9895, Section: Mutual Information Minimization “ Mutual In formation measures the co-dependence between two random variables X and Y”].
Regarding claim 6, the rejection of claim 2 is incorporated. Yoon discloses all the limitations of claim 1 as described above. Yoon further wherein: the speech recognition model is a model trained by using native utterance data, and the utterance speech used for training the accent module and the prompt generator is a non-native utterance speech.
[Figure 2: Overview of INTapt. INTapt incorporates a two-step training process where the first step involves training the Accent Module to get the accent feature of a particular input speech and the second step involves training the Prompt Generator capable of making the non-native (L2) English speech input have a better ASR performance by re-modulating the attention of the Backbone Model so that it resembles the accent of a native (L1) English speech.]
Regarding claim 7, Yoon further discloses a method of operating a model training device operated by at least one processor, the method comprising: training an accent module to extract an accent feature from an audio feature of an utterance speech; extracting a first accent feature from a prompt concatenation input that concatenates the prompt to the audio feature, and a second accent feature from the audio feature by using the accent module; and adversarially training a prompt generator that generates a prompt from the audio feature to minimize interdependence between the first accent feature and the second accent feature.
[Page 9894, Section 2 “IN Tapt incorporates a two-step training process. In the first step, we train an Accent Module (AM) capable of isolating the accent feature from a given audio feature a of an input speech x. In the second step, we train a Prompt Generator (PG), which out puts a prompt p for a given audio feature a, using two objectives: (1) Minimize the mutual information between the accent feature z′ and z, where the former is obtained using the prompt-concatenated input (p;a) and the latter is obtained from the original audio feature a, (2) Minimize CTC loss to improve the ASR performance of the input (p;a)”]; [Page 9895, Section: Mutual Information Minimization “We use this to adversarially train the prompt generator PGθ4 to minimize the mutual information between the accent feature of the original L2 speech input and the prompt-concatenated input”]
Regarding claim 8, the rejection of claim 7 is incorporated. Yoon discloses all the limitations of claim 1 as described above. Yoon further discloses further comprising: training the prompt generator to minimize a Connectionist Temporary Classification (CTC) loss of a speech recognition model that outputs text from the prompt concatenation input.
[Page 9895, Section: CTC Loss Minimization We train the prompt generator PGθ4 to minimize the CTC loss obtained for the prompt-concatenated input (p;a)”].
Regarding claim 9, the rejection of claim 8 is incorporated. Yoon discloses all the limitations of claim 1 as described above. Yoon further discloses wherein: the training of the prompt generator includes: obtaining a hidden state by inputting the audio feature into the speech recognition model; obtaining the prompt by inputting the hidden state into the prompt generator; generating the prompt concatenation input by concatenating the audio feature and the prompt; inputting the prompt concatenation input into the speech recognition model to obtain the CTC loss; and training the prompt generator to minimize the CTC loss.
[Page 9894, Section 2 INTapt “IN Tapt incorporates a two-step training process. In the first step, we train an Accent Module (AM) capable of isolating the accent feature from a given audio feature a of an input speech x. In the second step, we train a Prompt Generator (PG), which out puts a prompt p for a given audio feature a, using two objectives: (1) Minimize the mutual information between the accent feature z′ and z, where the former is obtained using the prompt-concatenated input (p;a) and the latter is obtained from the original audio feature a, (2) Minimize CTC loss to improve the ASR performance of the input (p;a).”];
[Page 9895, Section 2.2 Prompt Generator “To address this, we propose an input dependent prompt embedding by training prompt generator PGθ4 that generates an input-specific prompt guided by Information-Theoretic Adversarial Learning. More specifically, given a hidden state h = [h1,h2,...,hL] with length L we produce a prompt of length L′, p =PGθ4 (h)”];
Regarding claim 10, the rejection of claim 9 is incorporated. Yoon discloses all the limitations of claim 1 as described above. Yoon further discloses wherein: the extracting includes: obtaining a first hidden state by inputting the prompt concatenation input to the speech recognition model, and obtaining a second hidden state by inputting the audio feature to the speech recognition model; and extracting the first accent feature by inputting the first hidden state into the accent module, and extracting the second accent feature by inputting the second hidden state into the accent module.
[Page 9894, Section: Accent classification Head “Given the hidden state representation h of an audio feature input a, the feature extractor outputs the accent feature (i.e., z =fθ1 (h)) and the accent classification head fθ2 tries to assign it to the correct accent label y”]; [Page 9894 “ Since our method requires direct access to the isolated accent feature of the corresponding audio feature input, we propose an Accent Module (AM) capable of extracting the accent feature z from the input a. The module consists of an accent feature extractor fθ1 which is trained with an accent classification head fθ2 to isolate the accent feature and an accent intensity regression head fθ3 to capture the intensity of the accent into the obtained feature”].
Regarding claim 11, the rejection of claim 10 is incorporated. Yoon discloses all the limitations of claim 1 as described above. Yoon further discloses wherein: the adversarially training includes measuring the interdependence by using a mutual information neural estimator based on a neural network model.
[Page 9895, Section Mutual Information Minimization “ Mutual In formation measures the co-dependence between two random variables X and Y….The estimation is done using a neural network parameterized by ϕ as below: I(X,Y ) ≥ Iϕ(X,Y), (3) where maximizing Iϕ(X,Y ) provides a tight lower bound of the original mutual information I(X,Y ). We use this to adversarially train the prompt generator PGθ4 to minimize the mutual information between the accent feature of the original L2 speech input and the prompt-concatenated input”];
Regarding claim 12, the rejection of claim 10 is incorporated. Yoon discloses all the limitations of claim 1 as described above. Yoon further discloses wherein: the training of the accent module includes: training the accent module to extract an accent feature from the hidden state of the audio feature acquired through the speech recognition model, and to predict the CTC loss using the extracted accent feature and capture the accent strength; and train the accent module to extract the accent feature from the audio feature by being trained with an accent classification head that isolates the accent feature of a given speech.
[Page 9894 Section 2 INTapt “In the first step, we train an Accent Module (AM) capable of isolating the accent feature from a given audio feature a of an input speech x. In the second step, we train a Prompt Generator (PG), which out puts a prompt p for a given audio feature a, using two objectives: (1) Minimize the mutual information between the accent feature z′ and z, where the former is obtained using the prompt-concatenated input (p;a) and the latter is obtained from the original audio feature a, (2) Minimize CTC loss to improve the ASR performance of the input (p;a)”][ Page 9894 Section 2.1 “The module consists of an accent feature extractor fθ1 which is trained with an accent classification head fθ2 to isolate the accent feature and an accent intensity regression head fθ3 to capture the intensity of the accent into the obtained feature.”]
Regarding claim 13, Yoon discloses an automatic speech recognition apparatus operated by at least one processor, the automatic speech recognition apparatus comprising: a prompt generator for generating a prompt from an accent feature, which is a state hidden in an audio feature of an utterance speech; and a speech recognition model for generating text for the utterance speech from a prompt concatenation input that concatenates the audio feature and the prompt.
[Page 9895, Section 2.2 Prompt Generator: “ To address this, we propose an input dependent prompt embedding by training prompt generator PGθ4 that generates an input-specific prompt guided by Information-Theoretic Adversarial Learning. More specifically, given a hidden state h = [h1,h2,...,hL] with length L we produce a prompt of length L′, p =PGθ4 (h)”];
Regarding claim 14, Yoon discloses wherein: the prompt generator is adversarially trained to minimize interdependence between a first accent feature extracted from the prompt concatenation input that concatenates the prompt to the audio feature of the utterance speech and a second accent feature extracted from the audio feature.
[Page 9895 Section Mutual Information Minimization: “We use this to adversarially train the prompt generator PGθ4 to minimize the mutual information between the accent feature of the original L2 speech input and the prompt-concatenated input”].
Regarding claim 15, Yoon discloses wherein: the prompt generator is trained to minimize Connectionist Temporal Classification (CTC) loss of a speech recognition model that outputs text from the prompt concatenation input.
[Page 9895 CTC Loss Minimization “We train the prompt generator PGθ4 to minimize the CTC loss obtained for the prompt-concatenated input (p;a).”]
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim [ 1, 2, 4, 6, 7, 8, 9, 13, 14, 15 ] are rejected under 35 U.S.C. 103 as being unpatentable over Shao (Shao, Qijie, et al. "Decoupling and interacting multi-task learning network for joint speech and accent recognition." IEEE/ACM Transactions on Audio, Speech, and Language Processing 32 (2023): 459-470., hereinafter Shao) in view of Chen (Chen, Yi-Chen, et al. "Aipnet: Generative adversarial pre-training of accent-invariant networks for end-to-end speech recognition." ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, hereinafter Chen) and in further view of Liu (S. Liu et al., "End-To-End Accent Conversion Without Using Native Utterances," ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 2020, pp. 6289-6293, doi: 10.1109/ICASSP40776.2020.9053797, hereinafter Liu(Songxiang)).
Regarding claim 1, Shao teaches
A model training device operated by at least one processor, the model training device comprising: an accent module trained to extract an accent feature from an audio feature of an utterance speech; and;
[Page 2, Column 2, lines 3- 5 "Typically, an AR model is first trained to extract accent features from the input speech, which is then utilized to assist the ASR model" where AR model corresponds to accent module].
a second accent feature from the audio feature by using the accent module
[Page 2, Column 2, lines 3- 5 "Typically, an AR model is first trained to extract accent features from the input speech, which is then utilized to assist the ASR model" where AR model corresponds to accent module];
[Page 7, Column 1, lines 1-3 "Our baseline model is based on a triple-branch structure, which utilizes the hidden embedding before the last linear layer in the accent branch to generate accent embeddings" where a triple branch structure generating accent embeddings implies multiple accents including a first and second accent];
However, Shao does not
a prompt generator for extracting a first accent feature from a prompt concatenation input that concatenates the prompt to the audio feature,
Chen teaches
a prompt generator for extracting a first accent feature
[2.1.1. Adversarial Loss “To learn accent-invariant representations, we define two mappings from speech data: accent-invariant generator GAI(xt) and accent specific generator GAS(xt)” where generator architecture comprising GAI and GAS correspond to the prompt generator because it generates representations from the acoustic features and p would be the conditioning variable that is generated that directs the generator. The acoustic feature xt corresponds to the audio feature and the accent-invariant representation generated by GAI corresponds to the claimed first accent feature and the accent specific representation generated by GAS as the second accent feature];
[2.2. Fine-tuning for End-to-End Speech Recognition “In the fine-tuning stage, the outputs of GAI which encode accent invariant linguistic content can be plugged in as inputs of any downstream speech tasks that aim to improve accent robustness”].
[2. AIPNET “Our approach consists of two stages: pre-training and fine-tuning. In the pre-training stage, the model is built through adversarial training with the goal of learning accent-invariant representations” where the generator is adversarially trained];
[2.1.2. Reconstruction Loss The adversarial loss defined between DAI and GAI enforces that accent-specific information is disentangled from GAI but preserved in GAS” where disentangled refers to minimization of interdependence between the feature accents. ]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Shao and Chen because both references are directed towards improving automatic speech recognition through accent-aware processing. Shao teaches extracting accent features from speech to improve automatic speech recognition, while Chen teaches generating accent-related representations through a generator architecture. This combination would provide more informative accent representations while reducing accent-related variabilities during speech recognition. This would improve robustness to accent variation by combing explicit accent feature extraction with accent-conditioned representation for subsequent speech recognition processing.
However, Shao in view of Chen fail to expressly recite
But Liu(Songxiang) teaches
a prompt concatenation input that concatenates the prompt to the audio feature
[4.3. Multi-task accented ASR model “Since the training data of the ASR model includes accented utterances, following [15], we concatenated an accent embedding with acoustic features at each frame as inputs to the ASR model” where the accent embedding corresponds to claimed prompt, the acoustic features correspond to audio feature, and concatenating the accent embedding with the acoustic features corresponds to the prompt concatenation input].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further combine Shao and Chen with Liu (Songxiang) because Liu (Songxiang) teaches concatenating an accent embedding with acoustic features before speech recognition processing, and integrating Liu (Songxiang) conditioning mechanism in Shao-Chen framework allows the learned accent information to directly influence speech recognition while preserving the complementary advantage of accent feature extraction and adversarial representation training. This would provide richer conditioned inputs for generating accurate representations and improving recognition of accented speech.
Regarding claim 2, the rejection of claim 1 is incorporated. Shao, Chen, and Liu (Songxiang) teach all limitations of the current invention as stated above.
Shao further teaches
[Page 3 and 4 Section A, lines 42- 52, lines 1-3 "By stacking additional encoders after the shared encoder, our approach can alleviate the sequence-length inconsistency of the two-granularity unit modeling. And then, the computation of the losses for the CTC and attention decoders are denoted as:
L𝑐𝑡𝑐 = CTC(x𝑐𝑒, y𝑓)…..The total loss of our multi-task ASR-AR consists of the ASR loss L𝑎𝑡𝑡, the CTC loss L𝑐𝑡𝑐, and the AR loss L𝑎𝑟,
which can be formulated as: L𝐴𝑆𝑅−𝐴𝑅 = L𝑎𝑡𝑡 + 𝜆1L𝑐𝑡𝑐 + 𝜆2L𝑎𝑟," where x𝑐𝑒 corresponds to the CTC encoder outputs generated by the speech recogniton model (ASR-AR) and y𝑓 corresponds to transcription text used to compute the CTC loss.
[Page 6, lines 3-6 "According to [23], the CTC forward algorithm computes the negative logarithm of the conditional probabilities of the CTC encoder outputs x𝑐𝑒 and a given text. If the given text is the transcription label y𝑓 , the CTC loss is computed" where the output text is y𝑓].
However, Shao does not teach
the prompt generator is trained
Chen teaches
the prompt generator is trained
[2.1.1. Adversarial Loss “To learn accent-invariant representations, we define two mappings from speech data: accent-invariant generator GAI(xt) and accent specific generator GAS(xt)” where generator architecture comprising GAI and GAS correspond to the prompt generator because it generates an representations from the acoustic features. The acoustic feature xt corresponds to the audio feature and the accent-invariant representation generated by GAS corresponds to the claimed first accent feature];
[2.2. Fine-tuning for End-to-End Speech Recognition “In the fine-tuning stage, the outputs of GAI which encode accent invariant linguistic content can be plugged in as inputs of any downstream speech tasks that aim to improve accent robustness”].
[2. AIPNET “Our approach consists of two stages: pre-training and fine-tuning. In the pre-training stage, the model is built through adversarial training with the goal of learning accent-invariant representations” where the generator is adversarially trained];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Shao with Chen because Chen teaches jointly training the generator architecture with the speech recognition model, while Shao teaches optimizing the speech recognition model using CTC loss. Combing them would optimize the generator using a well-established ASR loss function.
However, Shao in view of Chen do not teach
However, Liu (Songxiang) teaches
from the prompt concatenation input
[4.3. Multi-task accented ASR model “Since the training data of the ASR model includes accented utterances, following [15], we concatenated an accent embedding with acoustic features at each frame as inputs to the ASR model” where the accent embedding corresponds to claimed prompt, the acoustic features correspond to audio feature, and concatenating the accent embedding with the acoustic features corresponds to the prompt concatenation input].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Shao and Chen with Liu (Songxiang) because Liu (Songxiang) teaches providing the speech recognition model with a concatenated input comprising accent-related information and acoustic features. Combining them would jointly optimize the generator and speech recognition model using accent-conditioned speech representation.
Regarding claim 4, the rejection of claim 3 is incorporated. Shao, Chen, and Liu (Songxiang) teach all limitations of the current invention as stated above.
Shao further teaches
the accent feature extractor extracts the accent feature from a hidden state of the audio feature acquired through the speech recognition model.
[Page 2, lines 6-19 "In the multi-task ASR-AR framework, accent embeddings from the AR branch can be leveraged….The hidden states of the DNN-based accent classifier [3] provide rich and stable utterance-level accent information, while the AR posterior probabilities are more straightforward and interpretable….Moreover, incorporating accent embeddings into either the encoder [3] or decoder [22] of the ASR model allows for the model to adapt to variations in pronunciation or linguistics, leading to varying effects on ASR performance]; [Page 2, lines 34- 38 "Then, for the task interaction, the CTC branch is optimized with the same modeling units as the AR branch to provide linguistic features for the AR task, while latent accent embeddings extracted from our AR model are used to improve the ASR branch"];
[Page 7, lines 1-3 "Our baseline model is based on a triple-branch structure, which utilizes the hidden embedding before the last linear layer in the accent branch to generate accent embeddings" where hidden states are formed and then DNN based accent classifier takes these hidden states and then extracts the accent embedding. The ASR-AR model corresponds to the speech recognition model mainly the ASR branch and the utterance level accent corresponds to accent feature]
Regarding claim 6, the rejection of claim 2 is incorporated. Shao, Chen, and Liu (Songxiang) teach all limitations of the current invention as stated above.
Shao further teaches
the speech recognition model is a model trained by using native utterance data,
[Page 7, Column 1, lines 20- 25 "In English experiments, our model is first trained for 70 epochs on the mixture of the AESRC and LibriSpeech datasets and then fine-tuned 50 epochs on the AESRC alone. In the first stage, we update the CTC and attention branches using both the AESRC and LibriSpeech datasets, while only the AESRC dataset is used to update the accent branch." where CTC and attention branches are part of the speech model and LibriSpeech is recognized as native English speech which is also used to train the model];
However, Shao does not teach
However, Chen teaches
the utterance speech used for training the accent module and the prompt generator is a non-native utterance speech.
[Abstract, Column 1,lines 4-10 “For this purpose, we propose a novel pre-training framework AIPNet based on generative adversarial nets (GAN) for accent-invariant representation learning: Accent Invariant Pre-training Networks. We pre-train AIPNet to disentangle accent-invariant and accent-specific characteristics from acoustic features through adversarial training on accented data for which transcriptions are not necessarily available.”];
[Introduction, Column 1, lines 5-10 “Due to the acoustic discrepancy among accents, an ASR system that is trained on the speech data of one accent (e.g., native) often fails to recognize speech of another unseen accent (e.g., non-native). In this work, we focus on learning accent-invariant representations, aiming to build a universal ASR system that is generalizable across accents” where the AIPNET teaches pre-training the generator architecture using accented training data. A POISTA would know that in this combined system, the same non-native accented speech would also be used to train Shao’s accent module because Shao’s accent module is trained to extract accent features for accepted speech];
[2.1. Accent-Invariance Pre-training “The goal of pre-training is to learn accent-invariant representations from accented training data”].
[3.1. Dataset The dataset used in experiments contains utterances in a variety of domains, such as weather or music, collected through crowdsourced workers. There are 9 English accents in total in the dataset, including United States (US), Korea (KR), Philippines (PH), Canada (CA), India (IN), France (FR), Britain (GB), Vietnam (VN) and Latin America (LA). The training set contains 4M (3.8K hours) utterances among which 1% is split as validation data. Particularly, there are 1Mand780Kutterances in US and LA respectively and about 330K data in each of the remaining accents”];
[2.1.1. Adversarial Loss “To learn accent-invariant representations, we define two mappings from speech data: accent-invariant generator GAI(xt) and accent specific generator GAS(xt)” where generator architecture comprising GAI and GAS correspond to the prompt generator because it generates an representations from the acoustic features. The acoustic feature xt corresponds to the audio feature and the accent-invariant representation generated by GAS corresponds to the claimed first accent feature];
[2.2. Fine-tuning for End-to-End Speech Recognition “In the fine-tuning stage, the outputs of GAI which encode accent invariant linguistic content can be plugged in as inputs of any downstream speech tasks that aim to improve accent robustness”].
[2. AIPNET “Our approach consists of two stages: pre-training and fine-tuning. In the pre-training stage, the model is built through adversarial training with the goal of learning accent-invariant representations” where the generator is adversarially trained];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Shao and Chen teaches because Shao teaches a speech recognition model trained using also native speech, while Chen teaches training accent processing components using accented training data. Combing these teachings improves adaptation to non-native speech while maintaining robust baseline speech recognition performance.
Regarding claim 7, Shao teaches
A method of operating a model training device operated by at least one processor, the method comprising: an accent module trained to extract an accent feature from an audio feature of an utterance speech; and;
[Page 2, Column 2, lines 3- 5 "Typically, an AR model is first trained to extract accent features from the input speech, which is then utilized to assist the ASR model" where AR model corresponds to accent module].
[Page 2, Column 2, lines 22-24 “By extracting phoneme-level accent variations, their method effectively improves AR performance, which provides evidence of ASR’s helpfulness in AR tasks..”].
extracting a first accent feature
[Page 2, Column 1, lines 11-13 "The hidden states of the DNN-based accent classifier [3] provide rich and stable utterance-level accent information" where the utterance level accent information is the first accent feature.];
[Page 7, Column 1, lines 1-3 "Our baseline model is based on a triple-branch structure, which utilizes the hidden embedding before the last linear layer in the accent branch to generate accent embeddings" where a triple branch structure generating accent embeddings implies multiple accents including a first and second accent that can be extracted from AR];
[Page 2, Column 2, lines 3- 5 "Typically, an AR model is first trained to extract accent features from the input speech, which is then utilized to assist the ASR model" where AR model corresponds to accent module];
However, Shao fails to expressly teach
But Liu (Songxiang) teaches
a prompt concatenation input that concatenates the prompt to the audio feature
[4.3. Multi-task accented ASR model “Since the training data of the ASR model includes accented utterances, following [15], we concatenated an accent embedding with acoustic features at each frame as inputs to the ASR model” where the accent embedding corresponds to claimed prompt, the acoustic features correspond to audio feature, and concatenating the accent embedding with the acoustic features corresponds to the prompt concatenation input].
Liu (Songxiang) further teaches
a generates a prompt from the audio feature
[4.3. Multi-task accented ASR model “Since the training data of the ASR model includes accented utterances, following [15], we concatenated an accent embedding with acoustic features at each frame as inputs to the ASR model” where the accent embedding corresponds to claimed prompt, the acoustic features correspond to audio feature, and concatenating the accent embedding with the acoustic features corresponds to the prompt concatenation input]; [4.3. Multi-task accented ASR model “We postulate that different accents are associated with different speaker. In this paper, the accent embedding of a speaker is obtained by averaging all his/her speaker embeddings”].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further combine Shao with Liu (Songxiang) because Liu (Songxiang) teaches concatenating an accent embedding with acoustic features before speech recognition processing, thereby providing the speech recognition model with both acoustic information and accent related information. This would provide richer conditioned inputs for generating accent related representations and improving recognition of accented speech.
Shao in view of Liu (Songxiang) do not teach
adversarially training a prompt generator that
Chen teaches
adversarially training a prompt generator that generates a prompt from the audio feature to minimize interdependence between the first accent feature and the second accent feature.
[2.1.1. Adversarial Loss “To learn accent-invariant representations, we define two mappings from speech data: accent-invariant generator GAI(xt) and accent specific generator GAS(xt)” where generator architecture comprising GAI and GAS correspond to the prompt generator because it generates an representations from the acoustic features. The acoustic feature xt corresponds to the audio feature and the accent-invariant representation generated by GAS corresponds to the claimed first accent feature];
[2.2. Fine-tuning for End-to-End Speech Recognition “In the fine-tuning stage, the outputs of GAI which encode accent invariant linguistic content can be plugged in as inputs of any downstream speech tasks that aim to improve accent robustness”].
[2. AIPNET “Our approach consists of two stages: pre-training and fine-tuning. In the pre-training stage, the model is built through adversarial training with the goal of learning accent-invariant representations” where the generator is adversarially trained];
[2.1.2. Reconstruction Loss The adversarial loss defined between DAI and GAI enforces that accent-specific information is disentangled from GAI but preserved in GAS” where disentangled refers to minimization of interdependence between the feature accents.]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Shao, Liu (Songxiang) and Chen because Shao teaches extracting accent features from speech to improve automatic speech recognition, while Liu (Songxiang) teaches concatenating an accent embedding with acoustic features before speech recognition processing and Chen teaches generating accent-related representations through a generator architecture. This combinations would produce a more robust recognition system which combines accent feature extraction, accent-conditioned inputs, and adversarial representation learning.
Regarding claim 8, the rejection of claim 7 is incorporated. Shao, Chen, and Liu (Songxiang) teach all limitations of the current invention as stated above.
Shao further teaches
[Page 3 and 4 Section A, lines 42- 52, lines 1-3 "By stacking additional encoders after the shared encoder, our approach can alleviate the sequence-length inconsistency of the two-granularity unit modeling. And then, the computation of the losses for the CTC and attention decoders are denoted as:
L𝑐𝑡𝑐 = CTC(x𝑐𝑒, y𝑓)…..The total loss of our multi-task ASR-AR consists of the ASR loss L𝑎𝑡𝑡, the CTC loss L𝑐𝑡𝑐, and the AR loss L𝑎𝑟,
which can be formulated as: L𝐴𝑆𝑅−𝐴𝑅 = L𝑎𝑡𝑡 + 𝜆1L𝑐𝑡𝑐 + 𝜆2L𝑎𝑟," where x𝑐𝑒 corresponds to the CTC encoder outputs generated by the speech recogniton model (ASR-AR) and y𝑓 corresponds to transcription text used to compute the CTC loss.
[Page 6, lines 3-6 "According to [23], the CTC forward algorithm computes the negative logarithm of the conditional probabilities of the CTC encoder outputs x𝑐𝑒 and a given text. If the given text is the transcription label y𝑓 , the CTC loss is computed" where the output text is y𝑓].
However, Shao does not teach
training a prompt generator
Chen teaches
training a prompt generator
[2.1.1. Adversarial Loss “To learn accent-invariant representations, we define two mappings from speech data: accent-invariant generator GAI(xt) and accent specific generator GAS(xt)” where generator architecture comprising GAI and GAS correspond to the prompt generator because it generates an representations from the acoustic features. The acoustic feature xt corresponds to the audio feature and the accent-invariant representation generated by GAS corresponds to the claimed first accent feature];
[2.2. Fine-tuning for End-to-End Speech Recognition “In the fine-tuning stage, the outputs of GAI which encode accent invariant linguistic content can be plugged in as inputs of any downstream speech tasks that aim to improve accent robustness”].
[2. AIPNET “Our approach consists of two stages: pre-training and fine-tuning. In the pre-training stage, the model is built through adversarial training with the goal of learning accent-invariant representations” where the generator is adversarially trained];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Shao with Chen because Chen teaches jointly training the generator architecture with the speech recognition model, while Shao teaches optimizing the speech recognition model using CTC loss. Combing them would optimize the generator using a well-established ASR loss function.
However, Shao in view of Chen do not teach
However, Liu (Songxiang) teaches
from the prompt concatenation input
[4.3. Multi-task accented ASR model “Since the training data of the ASR model includes accented utterances, following [15], we concatenated an accent embedding with acoustic features at each frame as inputs to the ASR model” where the accent embedding corresponds to claimed prompt, the acoustic features correspond to audio feature, and concatenating the accent embedding with the acoustic features corresponds to the prompt concatenation input].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Shao and Chen with Liu (Songxiang) because Liu (Songxiang) teaches providing the speech recognition model with a concatenated input comprising accent-related information and acoustic features. Combining them would jointly optimize the generator and speech recognition model using accent-conditioned speech representation.
Regarding claim 9, the rejection of claim 8 is incorporated. Shao, Chen, and Liu (Songxiang) teach all limitations of the current invention as stated above.
Shao further teaches
obtaining a hidden state by inputting the audio feature into the speech recognition model;
[Page 1 lines 33-39 "Recent studies have shown that leveraging linguistic information from ASR can effectively mitigate the overfitting issue in AR tasks [4, 7, 8, 19].
Initializing the AR encoder with a pre-trained ASR encoder [4] or jointly training a multi-task ASR-AR network [8, 20] are the commonly used approaches, both of which have demonstrated their effectiveness on various datasets"];
[Page 2, Column 1, lines 11-13 "The hidden states of the DNN-based accent classifier [3] provide rich and stable utterance-level accent information" where the hidden states correspond to hidden states]; [Page 5, lines 17-20 “In a multi-task ASR-AR, accent embeddings can be chosen from the hidden embedding before the last linear layer x𝑑𝑛𝑛, the posterior probabilities of classification x𝑝𝑝, or the accent shifts s.”]
[Page 5, Column 2, lines 17- 39 "In a multi-task ASR-AR, accent embeddings can be chosen from the hidden embedding before the last linear layer x𝑑𝑛𝑛, the posterior probabilities of classification x𝑝𝑝, or the accent shifts s. These three choices can be denoted as:
emb𝑑𝑛𝑛 = x𝑑𝑛𝑛,
emb𝑝𝑝 = UpProject(x𝑝𝑝),
emb𝑠𝑖𝑚 = UpProject(s), where UpProject represents a dimension expanding of the embeddings by a linear layer. Since the dimension of x𝑑𝑛𝑛 is significantly larger than that of x𝑝𝑝 and s, for a fair comparison, we use the up-project operation. The emb𝑑𝑛𝑛 provides rich and stable accent information, the emb𝑝𝑝 is intuitive and concise, and using the emb𝑠𝑖𝑚 can provide text related frame-level accent variations” where the x𝑑𝑛𝑛 corresponds to the hidden state and the accent embeddings can correspond to the prompt. In this case it is understood that the accent embeddings may be selected from hidden embeddings.]
inputting the
[Page 2, lines 7- "In the multi-task ASR-AR framework, accent embeddings from the AR branch can be leveraged….. The hidden states of the DNN-based accent classifier [3] provide rich and stable utterance-level accent information, while the AR posterior probabilities are more straightforward and interpretable. ….Moreover, incorporating accent embeddings into either the encoder [3] or decoder [22] of the ASR model allows for the model to adapt to variations in pronunciation or linguistics, leading to varying effects on ASR performance”]
[Page 3 and 4 Section A, lines 42- 52, lines 1-3 "By stacking additional encoders after the shared encoder, our approach can alleviate the sequence-length inconsistency of the two-granularity unit modeling. And then, the computation of the losses for the CTC and attention decoders are denoted as:
L𝑐𝑡𝑐 = CTC(x𝑐𝑒, y𝑓)…..The total loss of our multi-task ASR-AR consists of the ASR loss L𝑎𝑡𝑡, the CTC loss L𝑐𝑡𝑐, and the AR loss L𝑎𝑟,
which can be formulated as: L𝐴𝑆𝑅−𝐴𝑅 = L𝑎𝑡𝑡 + 𝜆1L𝑐𝑡𝑐 + 𝜆2L𝑎𝑟," where x𝑐𝑒 corresponds to the CTC encoder outputs generated by the speech recogniton model (ASR-AR) and y𝑓 corresponds to transcription text used to compute the CTC loss.];
[Page 6, lines 3-6 "According to [23], the CTC forward algorithm computes the negative logarithm of the conditional probabilities of the CTC encoder outputs x𝑐𝑒 and a given text. If the given text is the transcription label y𝑓 , the CTC loss is computed" where the output text is y𝑓].
[Page 3 and 4 Section A, lines 42- 52, lines 1-3 "By stacking additional encoders after the shared encoder, our approach can alleviate the sequence-length inconsistency of the two-granularity unit modeling. And then, the computation of the losses for the CTC and attention decoders are denoted as:
L𝑐𝑡𝑐 = CTC(x𝑐𝑒, y𝑓)…..The total loss of our multi-task ASR-AR consists of the ASR loss L𝑎𝑡𝑡, the CTC loss L𝑐𝑡𝑐, and the AR loss L𝑎𝑟,
which can be formulated as: L𝐴𝑆𝑅−𝐴𝑅 = L𝑎𝑡𝑡 + 𝜆1L𝑐𝑡𝑐 + 𝜆2L𝑎𝑟," where x𝑐𝑒 corresponds to the CTC encoder outputs generated by the speech recogniton model (ASR-AR) and y𝑓 corresponds to transcription text used to compute the CTC loss];
[Page 6, lines 3-6 "According to [23], the CTC forward algorithm computes the negative logarithm of the conditional probabilities of the CTC encoder outputs x𝑐𝑒 and a given text. If the given text is the transcription label y𝑓 , the CTC loss is computed" where the output text is y𝑓];
[Page 4, lines 16-20 “To enhance the ASR’s assistance to AR, in this study, we first introduce LASAS into the multi-task framework by feeding it with the aligned text output from the CTC decoder and acoustic features output from the shared encoder” where this model minimizes a combined loss function].
minimize the CTC loss
[Page 6, lines 3-6 "According to [23], the CTC forward algorithm computes the negative logarithm of the conditional probabilities of the CTC encoder outputs x𝑐𝑒 and a given text. If the given text is the transcription label y𝑓 , the CTC loss is computed" where the output text is y𝑓].
[Page 4, lines 16-20 “To enhance the ASR’s assistance to AR, in this study, we first introduce LASAS into the multi-task framework by feeding it with the aligned text output from the CTC decoder and acoustic features output from the shared encoder” where this model minimizes a combined loss function].
Shao does not teach
training the prompt generator and obtaining a prompt
However Chen teaches
training the prompt generator and obtaining a prompt
[2.1.1. Adversarial Loss “To learn accent-invariant representations, we define two mappings from speech data: accent-invariant generator GAI(xt) and accent specific generator GAS(xt)” where generator architecture comprising GAI and GAS correspond to the prompt generator because it generates an representations from the acoustic features and p would be the conditioning variable that directs the generator that is obtained to direct. The acoustic feature xt corresponds to the audio feature and the accent-invariant representation generated by GAS corresponds to the claimed first accent feature];
[2.2. Fine-tuning for End-to-End Speech Recognition “In the fine-tuning stage, the outputs of GAI which encode accent invariant linguistic content can be plugged in as inputs of any downstream speech tasks that aim to improve accent robustness”].
[2. AIPNET “Our approach consists of two stages: pre-training and fine-tuning. In the pre-training stage, the model is built through adversarial training with the goal of learning accent-invariant representations” where the generator is adversarially trained];
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Shao and Chen because Shao teaches extracting accent embeddings from hidden representations of the speech recognition model as well as optimizing the model by minimizing the CTC loss using these hidden states and Chen teaches generator architecture for producing accent-related conditioning representations. Combining them would generate a more optimized prompt generator for the speech recognition performance.
But Liu (Songxiang) teaches
generating the prompt concatenation input by concatenating the audio feature and the prompt;
inputting the prompt concatenation
[4.3. Multi-task accented ASR model “Since the training data of the ASR model includes accented utterances, following [15], we concatenated an accent embedding with acoustic features at each frame as inputs to the ASR model” where the accent embedding corresponds to claimed prompt, the acoustic features correspond to audio feature, and concatenating the accent embedding with the acoustic features corresponds to the prompt concatenation input].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further combine Shao and Chen with Liu (Songxiang) because Liu (Songxiang) teaches concatenating an accent embedding with acoustic features before speech recognition processing, thereby providing the speech recognition model with both acoustic information and accent related information. This would provide richer accent-conditioned inputs for generating accent related representations and improving recognition of accented speech.
Regarding claim 13, Shao teaches
An automatic speech recognition apparatus operated by at least one processor, the automatic speech recognition apparatus comprising:
a prompt generator for generating a prompt from an accent feature, which is a state hidden in an audio feature of an utterance speech;
[Page 1, Line 7- 11 "The shared encoder is responsible for extracting acoustic features from the input speech for both branches, and backpropagation from these two branches enables the shared encoder to learn how to extract both linguistic and accent representations. " where acoustic features corresponds to audio features . The shared encoder is utilized by both
the ASR branch and AR branch, the ASR branch corresponding to the claimed speech recognition model and the hidden states of the DNN based accent classifier corresponds to hidden states of audio features generated by the shared encoder]
[Page 5, Column 2, lines 17- 32 "In a multi-task ASR-AR, accent embeddings can be chosen from the hidden embedding before the last linear layer x𝑑𝑛𝑛, the posterior probabilities of classification x𝑝𝑝, or the accent shifts s. These three choices can be denoted as:
emb𝑑𝑛𝑛 = x𝑑𝑛𝑛,
emb𝑝𝑝 = UpProject(x𝑝𝑝),
emb𝑠𝑖𝑚 = UpProject(s),” where in this case UpProject corresponds to the claimed prompt generator because UpProject transforms the extracted accent related representation into an output embedding (embpp or embsim) that conditions the speech recognition model, correpsonding to generating the claimed prompt.The hidden states of the DNN based accent classifer corresponds to the claimed state hidden in an audio feature of an utterance speech. Since Shao teaches that accent embeddings maybe be selected from hidden embeddings x𝑑𝑛𝑛, posterier embeddings x𝑝𝑝 or accent shifts s, a POSITA would have understood that the prompt like embedding emb𝑠𝑖𝑚 is generated from the accent shift s and that is merely one implementation of an accent embedding and that could similarily be generated from the hidden embedding x𝑑𝑛𝑛, thereby yielding a prompt generated from an accent feature that is hidden state].
However, Shao does not teach a speech recognition model for generating text for the utterance speech from a prompt concatenation input that concatenates the audio feature and the prompt.
Liu (Songxiang) teaches
a speech recognition model for generating text for the utterance speech from a prompt concatenation input that concatenates the audio feature and the prompt.
[4.3. Multi-task accented ASR model “Since the training data of the ASR model includes accented utterances, following [15], we concatenated an accent embedding with acoustic features at each frame as inputs to the ASR model” where the accent embedding corresponds to claimed prompt, the acoustic features correspond to audio feature, and concatenating the accent embedding with the acoustic features corresponds to the prompt concatenation input].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Liu’s (Songxiang) prompt concatenation techniques into Shao’s speech recognition system so that the accent derived embeddings generated by Shao is provided to as a prompt and concatenated with the input to the speech recognition model. Because the generated prompt conditions the input representations used by the speech recognition model, the combined system enables recognition to leverage accent-specific information throughout the recognition process thereby, improving transcription accuracy and robustness for accented speech.
Regarding claim 14, the rejection of claim 12 is incorporated. Shao, Chen, and Liu (Songxiang ) teach all limitations of the current invention as stated above.
Shao further teaches
[Page 2, Column 2, lines 3- 5 "Typically, an AR model is first trained to extract accent features from the input speech, which is then utilized to assist the ASR model" where AR model corresponds to accent module].
[Page 2, Column 1, lines 11-13 "The hidden states of the DNN-based accent classifier [3] provide rich and stable utterance-level accent information" where the utterance level accent information is the first accent feature.];
[Page 7, Column 1, lines 1-3 "Our baseline model is based on a triple-branch structure, which utilizes the hidden embedding before the last linear layer in the accent branch to generate accent embeddings" where a triple branch structure generating accent embeddings implies multiple accents including a first and second accent];
However, Shao does not teach
the prompt generator is adversarially trained to minimize interdependence between a first accent feature
Chen further teaches
the prompt generator is adversarially trained to minimize interdependence between a first accent feature
[1.1. Adversarial Loss “To learn accent-invariant representations, we define two mappings from speech data: accent-invariant generator GAI(xt) and accent specific generator GAS(xt)” where generator architecture comprising GAI and GAS correspond to the prompt generator because it generates representations from the acoustic features and p would be the conditioning variable that directs the generator. The acoustic feature xt corresponds to the audio feature and the accent-invariant representation generated by GAI corresponds to the claimed first accent feature and the accent specific representation generated by GAS as the second accent feature];
[2.2. Fine-tuning for End-to-End Speech Recognition “In the fine-tuning stage, the outputs of GAI which encode accent invariant linguistic content can be plugged in as inputs of any downstream speech tasks that aim to improve accent robustness”].
[2. AIPNET “Our approach consists of two stages: pre-training and fine-tuning. In the pre-training stage, the model is built through adversarial training with the goal of learning accent-invariant representations” where the generator is adversarially trained];
[2.1.2. Reconstruction Loss The adversarial loss defined between DAI and GAI enforces that accent-specific information is disentangled from GAI but preserved in GAS” where disentangled refers to minimization of interdependence between the feature accents.]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Shao and Chen because Shao teaches extracting accent features from speech to improve automatic speech recognition, while Chen teaches generating accent-related representations through a generator architecture. This combination would produce a more robust accent-conditioned representation for subsequent speech recognition processing.
Liu (Songxiang) teaches
extracted from the prompt concatenation input that concatenates the prompt to the audio feature of the utterance speech
[4.3. Multi-task accented ASR model “Since the training data of the ASR model includes accented utterances, following [15], we concatenated an accent embedding with acoustic features at each frame as inputs to the ASR model” where the accent embedding corresponds to claimed prompt, the acoustic features correspond to audio feature, and concatenating the accent embedding with the acoustic features corresponds to the prompt concatenation input].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Liu’s (Songxiang) prompt concatenation techniques into Shao’s speech recognition system so that the accent derived embeddings generated by Shao is provided to as a prompt and concatenated with the input to the speech recognition model. Because the generated prompt conditions the input representations used by the speech recognition model, the combined system enables recognition to leverage accent-specific information throughout the recognition process thereby, improving transcription accuracy and robustness for accented speech.
Regarding claim 15, the rejection of claim 14 is incorporated. Shao, Chen, and Liu (Songxiang) teach all limitations of the current invention as stated above.
Shao teaches
the prompt generator is trained to minimize Connectionist Temporal Classification (CTC) loss of a speech recognition model that outputs text
[Page 5, Column 2, lines 17- 32 "In a multi-task ASR-AR, accent embeddings can be chosen from the hidden embedding before the last linear layer x𝑑𝑛𝑛, the posterior probabilities of classification x𝑝𝑝, or the accent shifts s. These three choices can be denoted as:
emb𝑑𝑛𝑛 = x𝑑𝑛𝑛,
emb𝑝𝑝 = UpProject(x𝑝𝑝),
emb𝑠𝑖𝑚 = UpProject(s), where UpProject represents a dimension expanding of the embeddings by a linear layer. Since the dimension of x𝑑𝑛𝑛 is significantly larger than that of x𝑝𝑝 and s, for a fair comparison, we use the up-project operation. The emb𝑑𝑛𝑛 provides rich and stable accent information, the emb𝑝𝑝 is intuitive and concise, and using the emb𝑠𝑖𝑚 can provide text related frame-level accent variations. In this paper, we compare these three accent embeddings in experiments to determine which one is better" where in this case UpProject corresponds to the claimed prompt generator because UpProject transforms the extracted accent related representation into an output embedding (embpp or embsim) that conditions the speech recognition model, correpsonding to generating the claimed prompt.];
[Page 3 and 4 Section A, lines 42- 52, lines 1-3 "By stacking additional encoders after the shared encoder, our approach can alleviate the sequence-length inconsistency of the two-granularity unit modeling. And then, the computation of the losses for the CTC and attention decoders are denoted as:
L𝑐𝑡𝑐 = CTC(x𝑐𝑒, y𝑓)…..The total loss of our multi-task ASR-AR consists of the ASR loss L𝑎𝑡𝑡, the CTC loss L𝑐𝑡𝑐, and the AR loss L𝑎𝑟,
which can be formulated as: L𝐴𝑆𝑅−𝐴𝑅 = L𝑎𝑡𝑡 + 𝜆1L𝑐𝑡𝑐 + 𝜆2L𝑎𝑟," where x𝑐𝑒 corresponds to the CTC encoder outputs generated by the speech recogniton model (ASR-AR) and y𝑓 corresponds to transcription text used to compute the CTC loss. Upproject corresponds to the claimed prompt generator, and because Upproject is a trainable component of Shao’s jointly optimized network whose training includes the CTC loss L𝑐𝑡𝑐, Upproject is trained as part of minimizing the CTC loss ];
[Page 6, lines 3-6 "According to [23], the CTC forward algorithm computes the negative logarithm of the conditional probabilities of the CTC encoder outputs x𝑐𝑒 and a given text. If the given text is the transcription label y𝑓 , the CTC loss is computed" where the output text is y𝑓].
Liu (Songxiang) teaches
from the prompt concatenation input
[4.3. Multi-task accented ASR model “Since the training data of the ASR model includes accented utterances, following [15], we concatenated an accent embedding with acoustic features at each frame as inputs to the ASR model” where the accent embedding corresponds to claimed prompt, the acoustic features correspond to audio feature, and concatenating the accent embedding with the acoustic features corresponds to the prompt concatenation input].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Liu’s (Songxiang) prompt concatenation techniques into Shao’s speech recognition system so that the accent derived embeddings generated by Shao is provided to as a prompt and concatenated with the input to the speech recognition model. Because the generated prompt conditions the input representations used by the speech recognition model, the combined system enables recognition to leverage accent-specific information throughout the recognition process thereby, improving transcription accuracy and robustness for accented speech.
Claim [ 3, 12 ] are rejected under 35 U.S.C. 103 as being unpatentable over Shao, Chen, and Liu (Songxiang) as applied to claim 1 above, and in further view of Liu (Liu, Rui, et al. "Controllable accented text-to-speech synthesis." arXiv preprint arXiv:2209.10804 (2022, hereinafter Liu).
Regarding claim 3, the rejection of claim 2 is incorporated. Shao, Chen, and Liu (Songxiang disclose all of the elements of the current invention as state above.
Shao further discloses
the accent module includes: an accent feature extractor that is trained with an accent classification head that isolates an accent feature of a given speech and extracts the accent feature from the audio feature; and
[Page 2, Column 2, lines 3- 5 "Typically, an AR model is first trained to extract accent features from the input speech, which is then utilized to assist the ASR model" where AR model corresponds to accent module].
[Page 2, lines 17 - 20 "Moreover, incorporating accent embeddings into either the encoder [3] or decoder [22] of the ASR model allows for the model to adapt to variations in pronunciation or linguistics, leading to varying effects on ASR performance. Specifically, AR and ASR are first decoupled by separated branches and two-granular modeling units to learn task-specific representations”];
[Page 2, lines 32-39 The AR branch is from our previously proposed LASAS [21] AR model and the ASR branch is an encoder-decoder-based Conformer [24] model. Then, for the task interaction, the CTC branch is optimized with the same modeling units as the AR branch to provide linguistic features for the AR task, while latent accent embeddings extracted from our AR model are used to improve the ASR branch" where the Accent recognition branch (AR) takes the acoustic features and maps them to fine grained phonetic units. At the end of this sits the classification head because the AR model must have an accent classifier to generate these embeddings-the extracted embeddings are what the classification head isolates.]
predicting the CTC loss
[Page 3 and 4 Section A, lines 42- 52, lines 1-3 "By stacking additional encoders after the shared encoder, our approach can alleviate the sequence-length inconsistency of the two-granularity unit modeling. And then, the computation of the losses for the CTC and attention decoders are denoted as:
L𝑐𝑡𝑐 = CTC(x𝑐𝑒, y𝑓)…..The total loss of our multi-task ASR-AR consists of the ASR loss L𝑎𝑡𝑡, the CTC loss L𝑐𝑡𝑐, and the AR loss L𝑎𝑟,
which can be formulated as: L𝐴𝑆𝑅−𝐴𝑅 = L𝑎𝑡𝑡 + 𝜆1L𝑐𝑡𝑐 + 𝜆2L𝑎𝑟," where x𝑐𝑒 corresponds to the CTC encoder outputs generated by the speech recogniton model (ASR-AR) and y𝑓 corresponds to transcription text used to compute the CTC loss];
[Page 6, lines 3-6 "According to [23], the CTC forward algorithm computes the negative logarithm of the conditional probabilities of the CTC encoder outputs x𝑐𝑒 and a given text. If the given text is the transcription label y𝑓 , the CTC loss is computed" where the output text is y𝑓].
Shao, Chen, and Liu (Songxiang) do not expressly recite
an accent intensity regression head for
However, Liu teaches
an accent intensity regression head for
[Page 4, lines 3-9 The accent intensity predictor consists of a bi-directional recurrent neural network layer with Gated Recurrent Unit (GRU), that is followed by an FC layer.
It is trained in a supervised manner together with the CAI-TTS model subject to the total loss, Lfinal = Lmel+Ldur+Lp pitch+Lp energy+ Lcc, where Lmel and Ldur are the MSE loss for mel-spectrum and duration loss as in FastSpeech2 [1]" where the accent intensity predictor corresponds to the accent intensity regression head];
[Page 5, section 2.1 “Accent Feature Extraction: Studies show that pitch and energy [50] related features are the descriptors of accents. We use opensmile [31] to extract 36-dimensional features fL1 and fL2 for each utterance,…. We expect that the accent intensity of the L2-accented sample Fh is higher than that of the L1 sample Fm. For the similar set S, we pick up two samples Fm and Fh from fL1 (or fL2). We assume that two samples from the same domain (L1 or L2) have similar accent intensities. each of which contains both pitch and energy features” where accent features are extracted to capture accent intensity]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated Liu’s accent intensity regression head into the combined system of Shao and Chen because Liu teaches predicting CTC loss from extracted accent features to capture accept strength. Integrating Liu’s regressions head into the adversarially trained prompt framework would enable the system to quantify accent intensity while providing additional supervision for prompt generation, thereby improving adaptation and recognition performance across different accent strengths.
Regarding claim 12, the rejection of claim 10 is incorporated. Shao, Chen, and Liu (Songxiang) disclose all of the elements of the current invention as state above.
Shao further discloses
The method of claim 10, wherein: the training of the accent module includes:
training the accent module to extract an accent feature from the hidden state of the audio feature acquired through the speech recognition model,
and to predict the CTC loss using the extracted accent feature
[Page 2, lines 6-19 "In the multi-task ASR-AR framework, accent embeddings from the AR branch can be leveraged….The hidden states of the DNN-based accent classifier [3] provide rich and stable utterance-level accent information, while the AR posterior probabilities are more straightforward and interpretable….Moreover, incorporating accent embeddings into either the encoder [3] or decoder [22] of the ASR model allows for the model to adapt to variations in pronunciation or linguistics, leading to varying effects on ASR performance]; [Page 2, lines 34- 38 "Then, for the task interaction, the CTC branch is optimized with the same modeling units as the AR branch to provide linguistic features for the AR task, while latent accent embeddings extracted from our AR model are used to improve the ASR branch" where hidden states are formed and then DNN based accent classifier takes these hidden states and then extracts the accent embedding. The ASR-AR model corresponds to the speech recognition model mainly the ASR branch and the utterance level accent corresponds to accent feature].
[Page 7, lines 1-3 "Our baseline model is based on a triple-branch structure, which utilizes the hidden embedding before the last linear layer in the accent branch to generate accent embeddings" ]
[Page 3 and 4 Section A, lines 42- 52, lines 1-3 "By stacking additional encoders after the shared encoder, our approach can alleviate the sequence-length inconsistency of the two-granularity unit modeling. And then, the computation of the losses for the CTC and attention decoders are denoted as:
L𝑐𝑡𝑐 = CTC(x𝑐𝑒, y𝑓)…..The total loss of our multi-task ASR-AR consists of the ASR loss L𝑎𝑡𝑡, the CTC loss L𝑐𝑡𝑐, and the AR loss L𝑎𝑟,
which can be formulated as: L𝐴𝑆𝑅−𝐴𝑅 = L𝑎𝑡𝑡 + 𝜆1L𝑐𝑡𝑐 + 𝜆2L𝑎𝑟," where x𝑐𝑒 corresponds to the CTC encoder outputs generated by the speech recogniton model (ASR-AR) and y𝑓 corresponds to transcription text used to compute the CTC loss];
[Page 6, lines 3-6 "According to [23], the CTC forward algorithm computes the negative logarithm of the conditional probabilities of the CTC encoder outputs x𝑐𝑒 and a given text. If the given text is the transcription label y𝑓 , the CTC loss is computed" where the output text is y𝑓].
train the accent module to extract the accent feature from the audio feature by being trained with an accent classification head that isolates the accent feature of a given speech
[Page 2, Column 2, lines 3- 5 "Typically, an AR model is first trained to extract accent features from the input speech, which is then utilized to assist the ASR model" where AR model corresponds to accent module].
[Page 2, lines 17 - 20 "Moreover, incorporating accent embeddings into either the encoder [3] or decoder [22] of the ASR model allows for the model to adapt to variations in pronunciation or linguistics, leading to varying effects on ASR performance. Specifically, AR and ASR are first decoupled by separated branches and two-granular modeling units to learn task-specific representations”];
[Page 2, lines 32-39 The AR branch is from our previously proposed LASAS [21] AR model and the ASR branch is an encoder-decoder-based Conformer [24] model. Then, for the task interaction, the CTC branch is optimized with the same modeling units as the AR branch to provide linguistic features for the AR task, while latent accent embeddings extracted from our AR model are used to improve the ASR branch" where the Accent recognition branch (AR) takes the acoustic features and maps them to fine grained phonetic units. At the end of this sits the classification head because the AR model must have an accent classifier to generate these embeddings-the extracted embeddings are what the classification head isolates.]
However, Shao, Chen, and Liu (Songxiang) do not teach
But Liu teaches
capture the accent strength
[Page 4, lines 3-9 The accent intensity predictor consists of a bi-directional recurrent neural network layer with Gated Recurrent Unit (GRU), that is followed by an FC layer.
It is trained in a supervised manner together with the CAI-TTS model subject to the total loss, Lfinal = Lmel+Ldur+Lp pitch+Lp energy+ Lcc, where Lmel and Ldur are the MSE loss for mel-spectrum and duration loss as in FastSpeech2 [1]" where the accent intensity predictor corresponds to the accent strength];
[Page 5, section 2.1 “Accent Feature Extraction: Studies show that pitch and energy [50] related features are the descriptors of accents. We use opensmile [31] to extract 36-dimensional features fL1 and fL2 for each utterance,…. We expect that the accent intensity of the L2-accented sample Fh is higher than that of the L1 sample Fm. For the similar set S, we pick up two samples Fm and Fh from fL1 (or fL2). We assume that two samples from the same domain (L1 or L2) have similar accent intensities. each of which contains both pitch and energy features” where accent features are extracted to capture accent strength]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated Liu’s accent intensity regression head into the combined system of Shao, Chen, and Liu (Songxiang) because Liu teaches predicting CTC loss from extracted accent features to capture accept strength. Integrating Liu’s regressions head into the adversarially trained prompt framework would enable the system to quantify accent intensity while providing additional supervision for prompt generation, thereby improving adaptation and recognition performance across different accent strengths.
Claim [ 5, 11 ] are rejected under 35 U.S.C. 103 as being unpatentable over Shao, Chen, and Liu(Songxiang) as applied to claim 1 above, and in further view of Belghazi ( Belghazi, Mohamed Ishmael, et al. "Mutual information neural estimation." International conference on machine learning. PMLR, 2018.hereinafter Belghazi).
Regarding claim 5, the rejection of claim 4 is incorporated. Shao, Chen, and Liu(Songxiang) disclose all of the elements of the current invention as state above.
However, Shao, Chen and Liu(Songxiang) fail to expressly recite a mutual information neural estimator that estimates the interdependence by using a neural network.
But Belghazi teaches
a mutual information neural estimator that estimates the interdependence by using a neural network.
[Page 2, lines 1-4 "We introduce the Mutual Information Neural Estimator (MINE), which is scalable, flexible, and completely trainable via back-prop, as well as provide a thorough theoretical analysis."]
[Page 2, Section 2.1 lines 19-20 "Mutual information is a Shannon entropy-based measure of dependence between random variables" where mutual information between random variables corresponds to interdependence.]
[Page 3, Section 3.1, lines 6 "Using both Eqn. 3 for the mutual information and the dual representation of the KL-divergence, the idea is to choose Fto be the family of functions Tθ : X ×Z → R parametrized by a deep neural network with parameters θ ∈ Θ." where neural network T0 corresponds neural network used to estimate the interdependence].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the mutual information neural estimator of Belghazi into the combined system of Shao, Chen, and Liu(Songxiang) because Belghazi teaches a neural network based estimator for accurately measuring mutual information between learned representations. Incorporating Belghazi’s estimator into Chen’s adversarial training framework enables direct estimation of interdependence between the first and second accent features. This thereby improves optimization of the prompt generator and reduces accent dependent information while maintaining speech recognition performance.
Regarding claim 11, the rejection of claim 10 is incorporated. Shao, Chen, and Liu(Songxiang) disclose all of the elements of the current invention as stated above.
Chen further teaches
wherein: the adversarially training
[2. AIPNET “Our approach consists of two stages: pre-training and fine-tuning. In the pre-training stage, the model is built through adversarial training with the goal of learning accent-invariant representations” where the generator is adversarially trained];
[2.1.2. Reconstruction Loss The adversarial loss defined between DAI and GAI enforces that accent-specific information is disentangled from GAI but preserved in GAS” where disentangled refers to minimization of interdependence between the feature accents. ]
However, Shao, Chen and Liu(Songxiang) fail to expressly recite
measuring the interdependence by using a mutual information neural estimator based on a neural network model.
But Belghazi teaches measuring the interdependence by using a mutual information neural estimator based on a neural network model.
[Page 2, lines 1-4 "We introduce the Mutual Information Neural Estimator (MINE), which is scalable, flexible, and completely trainable via back-prop, as well as provide a thorough theoretical analysis."]
[Page 2, Section 2.1 lines 19-20 "Mutual information is a Shannon entropy-based measure of dependence between random variables" where mutual information between random variables corresponds to interdependence.]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the mutual information neural estimator of Belghazi into Chen’s adversarial framework because Chen teaches adversarially minimizing interdependence between learned representations, while Belghazi provides a neural network based mutual information estimator for directly measuring such interdependence. Applying Belghazi’s estimator enables more accurate estimation of the dependence being minimized during the adversarial training. This in turn improves optimization of the prompt generator and reducing accent dependent information while maintaining speech recognition performance.
Claim [ 10 ] is rejected under 35 U.S.C. 103 as being unpatentable over Shao, Chen, and Liu(Songxiang) as applied to claim 9 above, and in further view of Zhang (US20240274123 (hereinafter Zhang).
Regarding claim 10, the rejection of claim 9 is incorporated. Shao, Chen and Liu(Songxiang) disclose all of the elements of the current invention as state above
Shao further teaches
the extracting includes:… extracting the first accent feature and extracting the second accent features
[Page 2, Column 2, lines 3- 5 "Typically, an AR model is first trained to extract accent features from the input speech, which is then utilized to assist the ASR model" where AR model corresponds to accent module];
[Page 7, Column 1, lines 1-3 "Our baseline model is based on a triple-branch structure, which utilizes the hidden embedding before the last linear layer in the accent branch to generate accent embeddings" where a triple branch structure generating accent embeddings implies multiple accents including a first and second accent];
Liu (Songxiang) further teaches
[4.3. Multi-task accented ASR model “Since the training data of the ASR model includes accented utterances, following [15], we concatenated an accent embedding with acoustic features at each frame as inputs to the ASR model” where the accent embedding corresponds to claimed prompt, the acoustic features correspond to audio feature, and concatenating the accent embedding with the acoustic features corresponds to the prompt concatenation input].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Liu’s (Songxiang) prompt techniques into Shao’s accent speech recognition system so that accent derived embeddings generated by Shao is concatenated with the input feature prior to speech recognition. Because both produce speech representations, combining them would improve accent adaptation and speech recognition performance.
Shao in view of Chen and in further view of Liu (Songxiang) do no teach
the extracting includes: obtaining a first hidden state
and obtaining a second hidden state by inputting the audio feature to the speech recognition model; and
However, Zhang teaches
obtaining a first hidden state by inputting
and obtaining a second hidden state by inputting the audio feature to the speech recognition model; and
[0066 “The ML model 145 processes a sequence of inputs (e.g., audio frames) in order, so that the output corresponding to each input factors in both the inputs and outputs that preceded it. The encoder 210 receives acoustic feature data 205. The acoustic feature data 205 may be a raw acoustic feature vector corresponding to an audio frame from the audio data 411/511. For example, the encoder 210 may receive acoustic feature vectors x=x1, x2, . . . xT for an audio frame of length T. The encoder 210 converts the acoustic feature data 205 to a sequence of hidden states hEt=
PNG
media_image1.png
42
38
media_image1.png
Greyscale
(xt), where t is the time/frame index” "where hidden states correspond to either the first hidden state or the second hidden state, and DNN accent classifier from Shao above corresponds to the accent module capable of extracting accent information from hidden states].
[0066 “The ML model 145 processes a sequence of inputs (e.g., audio frames) in order, so that the output corresponding to each input factors in both the inputs and outputs that preceded it. The encoder 210 receives acoustic feature data 205. The acoustic feature data 205 may be a raw acoustic feature vector corresponding to an audio frame from the audio data 411/511. For example, the encoder 210 may receive acoustic feature vectors x=x1, x2, . . . xT for an audio frame of length T. The encoder 210 converts the acoustic feature data 205 to a sequence of hidden states hEt=
PNG
media_image1.png
42
38
media_image1.png
Greyscale
(xt), where t is the time/frame index” "where hidden states correspond to either the first hidden state or the second hidden state, and DNN accent classifier from Shao above corresponds to the accent module capable of extracting accent information from hidden states that are inputted].
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Shao, Chen, and Liu (Songxiang) with Zhang because this combination would enable extraction and comparison of accent features from both prompt-conditioned and original speech representations and this would thereby improve accent adaptive speech recognition.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHEZA ABDUL AZIZ whose telephone number is (571)272-9610. The examiner can normally be reached Monday-Friday 7:30am-5pm Alternate Fridays off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHEZA ABDUL AZIZ/Examiner, Art Unit 2657
/Sean E Serraguard/Primary Examiner, Art Unit 2657
Prosecution Insights