DETAILED ACTION
This communication is in response to the Application filed on 09/03/2024. Claims 1-20 are pending and have been examined. Claims 1, 10 and 19 are independent. This Application was published as U.S. Pub. No. 2025/009,5636.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 09/09/2024 was filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority
Applicant’s claim for benefit of a provisional applications 63/583,227 and 63/583,214 submitted on 9/15/2023 is acknowledged.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-5, 10, 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Lu et al., (US Pub No. 2024/0013777, hereinafter, Lu) in view of Biadsy et al., (US Pub No. 2022/0068257, hereinafter, Biadsy).
Regarding Claim 1,
Lu discloses A computer-implemented method comprising (Lu, Abstract, paras [002-004], "…method includes training an automatic speech recognition (ASR) model on the subset of utterances..."):
generating a set of test examples, the set of test examples comprising subsets of test examples, each respective subset of test examples of the subsets of test examples corresponding to a particular test category of a plurality of test categories (Lu, Fig. 3A, par [030], "…the corpus of unlabeled training data 358 may include a plurality of spoken utterances 360, 360a-n…");
for each respective subset of test examples of the subsets of test examples:
selecting a test category of the plurality of test categories (Lu, Fig.3B, paras [034-038], "…the training data selection process 300 selects the subset of unlabeled training data 359 from the available corpus of unlabeled training data 358 that best match the target domain 324…After determining the domain relevance scores 346, the process 300 selects the utterances 360 with the N-best scores 346 for inclusion in the subset of unlabeled training data 359....");
But, Lu does not explicitly discloses the limitations, "using a machine learning model to convert audio samples of the respective subset of test examples to text transcripts; determining a word error rate for the respective subset of test examples by comparing the text transcripts to text samples corresponding to the audio samples of the respective subset of test examples, wherein the word error rate for the respective subset of test examples is included in a set of word error rates for the set of test examples; selecting a test category of the plurality of test categories based on the word error rates for the set of test example; generating a set of training examples from a selected subset of test examples of the subsets of test examples, the selected subset of test examples corresponding to the test category."
Biadsy, in the analogous field of endeavor, discloses using a machine learning model to convert audio samples of the respective subset of test examples to text transcripts (Biadsy, Fig.2C, par [059], "…The decoder 214 may a neural network configured to receive..."; Fig.2D, paras [061-062], "…for each unspoken training text utterance 302b of the plurality of unspoken training text utterances...uses a text decoder 250 to generate a textual representation 318 for the corresponding audio waveform of synthesized canonical fluent speech 316 generated as output from the adapted S2S conversion model 301..."), and
determining a word error rate for the respective subset of test examples by comparing the text transcripts to text samples corresponding to the audio samples of the respective subset of test examples, wherein the word error rate for the respective subset of test examples is included in a set of word error rates for the set of test examples (Biadsy, par [062], "…the word error rate loss 342 is based on the textual representation 318 output from text decoder 250 for the synthesized canonical fluent speech 306 and the corresponding unspoken training text utterance 302b…");
selecting a test category of the plurality of test categories based on the word error rates for the set of test examples (Biadsy, paras [046-053], "…the data generation stage 200b selects the subset of the available unspoken training text utterances 302b from the corpus 402 that best match the specific domain…After determining the scores, the data generation process 200b selects the unspoken training text utterances 302b with the N-best scores S as these unspoken training text utterances 302b best match the specific domain..."; par [063], "…the validation and filtering stage 200d validates each synthetic speech representation 306 output from the adapted TTS model 210 by determining whether or not the corresponding word error rate loss 342 satisfies a word error rate loss threshold..."); and
generating a set of training examples from a selected subset of test examples of the subsets of test examples, the selected subset of test examples corresponding to the test category (Biadsy, Fig.2D, par [063], "…When the corresponding word error rate loss 342 satisfies the word error rate loss threshold, the corresponding synthetic speech representation 306 is stored in a filtered set of synthetic speech representations …").
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified a method of training an automatic speech recognition (ASR) model of Lu with the validation and filtering state corresponding word error loss of Biadsy with a reasonable expectation of success to achieve acceptable accuracy by the ASR models by including sufficient training data targeting specific domain that are spoke by speakers with atypical speech patterns (Biadsy, par [002]).
Regarding Claim 4,
Lu in view of Biadsy discloses the computer-implemented method of claim 1,
wherein the word error rate for the respective subset of test examples is determined by comparing a text transcript for a respective test example of the respective subset of test examples to a text sample corresponding to an audio sample for the respective test example, the text sample being included in the text samples and the audio sample being included in the audio samples (Biadsy, par [062], "…the word error rate loss 342 is based on the textual representation 318 output from text decoder 250 for the synthesized canonical fluent speech 306and the corresponding unspoken training text utterance 302b…").
Regarding Claim 5,
Lu in view of Biadsy discloses the computer-implemented method of claim 1, wherein selecting the test category of the plurality of test categories comprises:
identifying a candidate word error rate in the set of word error rates that is the greatest among word error rates in the set of word error rates (Biadsy, par [063], "…the validation and filtering stage 200d validates each synthetic speech representation 306 output from the adapted TTS model 210 by determining whether or not the corresponding word error rate loss 342 satisfies a word error rate loss threshold..."),
identifying a candidate subset of test examples of the set of test examples that is associated with the candidate word error rate (Biadsy, Fig.2D, par [063], "…When the corresponding word error rate loss 342 satisfies the word error rate loss threshold, the corresponding synthetic speech representation 306 is stored in a filtered set of synthetic speech representations …"), and
identifying a candidate test category that is associated with the candidate subset of test examples, the candidate test category being included in the plurality of test categories (Biadsy, paras [046-053], "…the data generation stage 200b selects the subset of the available unspoken training text utterances 302b from the corpus 402 that best match the specific domain…After determining the scores, the data generation process 200b selects the unspoken training text utterances 302b with the N-best scores S as these unspoken training text utterances 302b best match the specific domain...").
Claim 10 is a system claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale. Additionally,
Lu discloses a system comprising: one or more processing systems; and one or more computer-readable media storing instructions which, when executed by the one or more processing systems, cause the system to perform operations (Lu, Fig.6, paras [047-048], "…The computing device 600 includes a processor
610, memory 620, a storage device 630...The memory 620 may be a computer-readable medium, a volatile memory unit(s),or non-volatile memory unit(s). The non-transitory memory 620...")
Rationale for combination is similar to that provided for Claim 1.
Claim 13 is a system claim with limitations similar to the limitations of Claim 4 and is rejected under similar rationale.
Claim 14 is a system claim with limitations similar to the limitations of Claim 5 and is rejected under similar rationale.
Claims 2, 6, 11 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Lu in view of Biadsy further in view of Ye et al., (US Pub No. 2023/0186919, hereinafter, Ye).
Regarding Claim 2,
Lu in view of Biadsy discloses the computer-implemented method of claim 1, but does not explicitly teach the limitations of the claim.
Ye, in the analogous field of endeavor, discloses improved techniques for generating TTS data to modify speech recognition models (paras [009, 035-041], Fig.2), and further discloses wherein generating the set of test examples comprises: accessing a set of terms (Fig.4, paras [052-058], computing system receiving a custom keyword (act 420) (par [053]));
using a pre-trained language model to generate a set of sentences for the set of terms (par [054], "…the computing system obtains new TTS training data for the keyword (act 440)..."; Fig.7, paras [067-070], a neural language generator (NLG) 720 is used to generate the random text 732 and/or content relevant text 734);
extracting a subset of sentences from the set of sentences, each sentence of the subset of sentences comprising a term in the set of terms (par [069], Content relevant text 734 refers to text generated by the NLG 720 that pertains to a certain topic or type of speech and/or is personalized to a speaker's typical vocabulary and
topics of conversation or typical speech commands) ;
processing the subset of sentences to generate a set of processed sentences , wherein processing the subset of sentences comprises normalizing text in the subset of sentences and phonetically transcribing the text in the subset of sentences (paras [070,103], "…The TTS data is passed through data simulation and pre-processing steps...The front end includes many natural language processing (NLP) components including a sentence and word breaker, text normalization, POS tagger, pronunciation model (lexicon), prosody predictor, among other components...");
using a text-to-speech model to generate a plurality of audio samples for each respective processed sentence of the set of processed sentences and forming the set of test examples based on the plurality of audio samples and the subset of sentences (par [069], "…The output random text 732 or content relevant text 734 can then be used to generate synthesized speech data 760 from the Neural TTS System 750...").
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified a method of training an automatic speech recognition (ASR) mode in specific target domain taught by Lu in view of Biadsy with generating TTS data to modify speech recognition models of Ye with a reasonable expectation of success to overcome data sparsity for training models used for speech recognition, keyword spotting, and speaker adaptation (Ye, paras [002-008]).
Regarding Claim 6,
Lu in view of Biadsy discloses the computer-implemented method of claim 1, but does not explicitly discloses a data augmentation technique.
Ye, in the analogous field of endeavor, discloses wherein the set of training examples are generated from the selected subset of test examples by applying a data augmentation technique to the selected subset of test examples, wherein a total speech time that is associated with the set of training examples is greater than a total speech time associated with the selected subset of test examples (Ye, Fig.11, paras [094-100], "…the speaker adaptation task consists of six speakers (three native and three non-native), each with 10 minutes (or another amount) of adaptation data for training and 20 minutes ( or more or less) of data for testing...").
Rationale for combination is similar to that provided for Claim 2.
Claim 11 is a system claim with limitations similar to the limitations of Claim 2 and is rejected under similar rationale.
Claim 15 is a system claim with limitations similar to the limitations of Claim 6 and is rejected under similar rationale.
Claims 3 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Lu in view of Biadsy further in view of Liang et al., ("Improving code-switching and named entity recognition in ASR with speech editing based data augmentation." arXiv preprint arXiv:2306.08588 (2023), hereinafter, Liang).
Regarding Claim 3,
Lu in view of Biadsy discloses the computer-implemented method of claim 1, wherein generating the set of test examples comprises, but does not explicitly teach the limitations of the claim.
Liang, in the analogous field of endeavor, discloses accessing a template comprising a set of named entity classes (4.2. Named entity recognition in ASR, 4.2.2. Models and tasks,"…800 sentences containing person names are randomly sampled from the Gigaspeech-Xl set to build the data base of the templates...");
accessing lists of values for the set of named entity classes (4.2.2. Models and tasks, "…The target name list is formed based on all names appearing in the test sets…."); and
forming the set of test examples (4.2. Named entity recognition in ASR, 4.2.2. Models and tasks,"…The sentence templates with name tags and names from the name list can be flexibly combined to form complete sentences, which are then used as input for the three methods of speech generation...the generated speech is used to fine-tune the baseline system…") by:
(i) selecting a respective named entity class of the set of named entity classes;
(ii) selecting a value from a list of values of the lists of values, the list of values corresponding to the respective named entity class,
(iii) populating a portion of the template corresponding to the respective named entity class,
(iv) repeating steps (i)-(iii) for each respective named entity class of the set of named entity classes, and
(v) repeating steps (i)-(iv) a predetermined number of times.
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified a method of training an automatic speech recognition (ASR) mode in specific target domain taught by Lu in view of Biadsy with Named entity recognition model of Liang with a reasonable expectation of success to significantly improve the potential problem in named entity recognition of the audio splicing and neural TTS based data augmentation systems by applying the text-based speech editing model (Liang, Abstract).
Claim 12 is a system claim with limitations similar to the limitations of Claim 3 and is rejected under similar rationale.
Allowable Subject Matter
Claims 19 and 20 are allowed. The following is a statement of reasons for the indication of allowable subject matter: Claim 19 is allowed because prior art search could not locate the references teaching the limitations accessing the multiple subsets of training examples, assigning sampling weights to subsets of training examples, and sampling a set of candidate training examples from the subsets based on the sampling weights. Claim 20 is allowable as being dependent upon the base claim 19.
Claims 7-9 and 16-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Chen et al., (US Pub No. 2023/0197057, hereinafter, Chen) discloses updating parameters of the GAN-based TTS model based on the adversarial loss term determined at each of the plurality of output steps for each training text utterance of the plurality of training text utterances..
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JANGWOEN LEE whose telephone number is (703)756-5597. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, BHAVESH MEHTA can be reached at (571)272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JANGWOEN LEE/Examiner, Art Unit 2656
/BHAVESH M MEHTA/Supervisory Patent Examiner, Art Unit 2656
Prosecution Insights