DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 14 - 18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract without significantly more.
When considering subject matter eligibility under 35 USC 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter.
Specifically, claims 14 - 18 are directed to a method. They hereby fall under one of the four statutory classes of invention.
If the claim does not fall within one of the statutory categories, it must then be determined whether the claim is directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea).
Claims 14 - 18 steps of observation, evaluation, and judgement that can be practically performed by a human, either mentally or with the use of pen and paper.
The limitation of “generating a modified text label by converting the translated text label into phonemes and performing one or more of the following steps: inserting spacing characters into the phonemes to denote pauses in speech; inserting pacing characters into the phonemes to denote the pace of speech; inserting non-verbal characters into the phonemes to denote non-verbal speech elements;” in claims 14 - 18, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind, but for the recitation of generic computer components. That is, other than reciting “a machine learning model”, nothing in the claim element precludes the steps from practically being performed in a human mind.
The mere nominal recitation of a machine learning model does not take the claim limitations out of the mental processes grouping.
If a claim limitation, under its broadest reasonable interpretation, covers mental processes but for the recitation of generic computer components, then it falls within the "Mental Processes" grouping of abstract ideas (concepts performed in the human mind including an observation, evaluation, judgement, and opinion). Accordingly, the claims 14 - 18 recite an abstract idea.
This judicial exception is not integrated into a practical application. In particular, the claims recite the additional elements “acquiring an input text label in a first language; acquiring a language code corresponding to the first language; acquiring a speaker identification; providing the modified text label, the language code, and speaker identification to a machine learning model configured to output a synthetic mel spectrogram; comparing the synthetic mel spectrogram to an original mel spectrogram using a predetermined loss function to calculate a loss value; and repeating steps a) through f) until the loss value meets a predetermined threshold.”.
The limitation “acquiring an input text label in a first language; acquiring a language code corresponding to the first language; acquiring a speaker identification”, amount to data-gathering steps which is considered to be insignificant extra-solution activity, (See MPEP 2106.05(g)).
The limitation “providing the modified text label, the language code, and speaker identification to a machine learning model configured to output a synthetic mel spectrogram; comparing the synthetic mel spectrogram to an original mel spectrogram using a predetermined loss function to calculate a loss value; and repeating steps a) through f) until the loss value meets a predetermined threshold.”, represents an extra-solution activity because it is a mere nominal or tangential addition to the claim, a mere generic transmission and presentation of collected and analyzed data. (See MPEP 2106.05 (g)).
The claimed “a machine learning model” is recited at a high level of generality and are merely invoked as tool to perform an existing training machine learning model.
Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea, thus fail to integrate the abstract idea into a practical application. See MPEP 2106.05(g).
The insignificant extra-solution activities identified above, which include the data-gathering (acquiring), and providing steps, are recognized by the courts as well-understood, routine, and conventional activities when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity (See MPEP 2106.05(d)(II) (i) Receiving or transmitting data over a network, e.g., using the Internet to gather data, buySAPE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPO2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); (v) Presenting (displaying) offers and gathering statistics, OIP Techs., 788 F.3d at 1362-63, 115 USPO2d at 1092- 93). The claims are not patent eligible.
Claims 14 - 18 do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a machine learning model to perform the generating, acquiring, and inserting steps amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept.
Even when considered in combination, these additional elements (a machine learning model) represents mere instruction to apply an exception and insignificant extra-solution activity, which do not provide an inventive concept.
Claims 14 - 18 as a whole, do not amount to significantly more than the abstract idea itself. This is because the claims do not affect an improvement to the functioning of a computer itself; and the claims do not move beyond a general link of the use of an abstract idea to a particular technological environment.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1 – 18 are rejected under 35 U.S.C. 102(a)(2) as being anticipated Gupta et al. (US PAP 2022/0358905).
As per claim 1, Gupta et al. teach a method for synthesizing translated speech from original audio speech in a first language, comprising:
acquiring a translated text label, wherein the translated text label is in a second language and corresponds to the original audio speech (“translated audio is generated using the translated input transcription and meta information”; paragraph 16);
acquiring a speaker identification corresponding to a target speaker (“Each vocal segment in the plurality of vocal segments includes a speaker identification to identify the speaker of each vocal segment.”; paragraph 12);
acquiring a language code corresponding to the second language (“meta translation generator 132 is provided with specific output languages 112 for each input language (e.g., English to German and Spanish to German or English to German and Spanish to French).”; paragraph 75);
generating a modified text label by: converting the translated text label into phonemes; inserting spacing characters into the phonemes to denote pauses in speech; inserting pacing characters into the phonemes to denote the pace of speech; inserting non-verbal characters into the phonemes to denote non-verbal speech elements (“similar pacing includes less than or equal to 20% difference in hamming distance between phonetic characters and inclusion of pauses, breaths, and filler sounds in the proper locations… ability to modify translated transcription… AI SD processor 125 can further identify coughs, sneezes, pauses in speech and other non-verbal audio segments or non-verbal noises created by a speaker”; paragraphs 15, 67, 76, 107); and
providing the modified text label, the language code, and speaker identification to a machine learning model configured to output translated synthetic speech (“the generator can generate synthetic translated audio with roughly a 99% vocal match. If not trained on audio from the same speaker, the generator can generate synthetic translated audio having a vocal match around 80% or better.”; paragraph 84).
As per claim 2, Gupta et al. further disclose generating a mel spectrogram from the machine learning model based on the modified text label, the language code, and the speaker identification (“by incorporating digital representations of the corresponding audio signal (e.g., MelSpectogram and/or raw audio waveforms) with generative AI.”; paragraphs 42, 169).
As per claim 3, Gupta et al. further disclose acquiring an input text label in the first language that corresponds to the original audio speech and translating the input text label to create the translated text label (paragraphs 13, 78 – 80).
As per claim 4, Gupta et al. further disclose inputting the translated text label into a phoneme generator to convert the translated text label into phonemes, wherein the phoneme generator is a neural network trained on a dataset of phonetic representations of words (“text preprocessor 128 is configured to convert text into phoneme analysis and/or perform emotional/sentiment analysis.”; paragraphs 64- 66).
As per claim 5, Gupta et al. further disclose including inputting the original audio speech or a digital representation of the original audio speech into a spacing character generator to insert spacing characters into the phonemes, wherein the spacing character generator is configured to identify pauses by analyzing an amplitude of an audio signal of the original audio speech (“AI SD processor 125 can further identify coughs, sneezes, pauses in speech and other non-verbal audio segments or non-verbal noises created by a speaker.”; paragraphs 55, 107, 159).
As per claim 6, Gupta et al. further disclose inputting the original audio speech or a digital representation of the original audio speech into a pacing character generator to insert pacing characters into the phonemes, wherein the pacing character generator is configured to calculate a rate of speech (“For each vocal segment in the plurality of vocal segments pacing information is identified for each word or phoneme in each vocal segment…Pacing/prosody/rhythm (referred to herein after “pacing”) is the measurable time associated with each syllable, word, other phoneme, non-verbal speech (such as cough, laugh, gasp) or pauses in speech with 0.05 s resolution.”; paragraphs 12, 67).
As per claim 7, Gupta et al. further disclose inputting the original audio speech or a digital representation of the original audio speech into a non-verbal character generator to insert non-verbal characters into the phonemes, wherein the non-verbal character generator includes a neural network trained to recognize patterns in audio (“AI SD processor 125 can further identify coughs, sneezes, pauses in speech and other non-verbal audio segments or non-verbal noises created by a speaker”; paragraphs 15, 67, 76, 107).
As per claim 8, Gupta et al. teach a machine learning system for synthesizing translated speech having one or more processors to:
acquire an input text label in a first language(“translated audio is generated using the translated input transcription and meta information”; paragraph 16);
acquire a language code corresponding to a second language into which the input text label is to be translated(“meta translation generator 132 is provided with specific output languages 112 for each input language (e.g., English to German and Spanish to German or English to German and Spanish to French).”; paragraph 75);
acquire a translated text label, wherein the translated text label is in the second language and corresponds to input text label (“translated audio is generated using the translated input transcription and meta information”; paragraphs 12 - 16);
acquire a speaker identification corresponding to a target speaker(“Each vocal segment in the plurality of vocal segments includes a speaker identification to identify the speaker of each vocal segment.”; paragraph 12);
acquire a language code corresponding to the second language(“meta translation generator 132 is provided with specific output languages 112 for each input language (e.g., English to German and Spanish to German or English to German and Spanish to French).”; paragraph 75);
generate a modified text label by converting the translated text label into phonemes and performing one or more of the following steps: inserting spacing characters into the phonemes to denote pauses in speech; inserting pacing characters into the phonemes to denote the pace of speech; inserting non-verbal characters into the phonemes to denote non-verbal speech elements(“similar pacing includes less than or equal to 20% difference in hamming distance between phonetic characters and inclusion of pauses, breaths, and filler sounds in the proper locations… ability to modify translated transcription… AI SD processor 125 can further identify coughs, sneezes, pauses in speech and other non-verbal audio segments or non-verbal noises created by a speaker”; paragraphs 15, 67, 76, 107);
provide the modified text label, the language code, and speaker identification to a machine learning model configured to output translated synthetic speech; and generate translated speech from the machine learning model based on the modified text label, the language code, and the speaker identification(“the generator can generate synthetic translated audio with roughly a 99% vocal match. If not trained on audio from the same speaker, the generator can generate synthetic translated audio having a vocal match around 80% or better.”; paragraph 84).
As per claim 9, Gupta et al. further disclose acquiring an input text label in the first language that corresponds to the original audio speech and translate the input text label to create the translated text label (paragraphs 13, 78 – 80).
As per claim 10, Gupta et al. further disclose the one or more processors are further configured to input the translated text label into a phoneme generator to convert the translated text label into phonemes, wherein the phoneme generator is a neural network trained on a dataset of phonetic representations of words (“text preprocessor 128 is configured to convert text into phoneme analysis and/or perform emotional/sentiment analysis.”; paragraphs 64- 66).
As per claim 11, Gupta et al. further disclose the one or more processors are further configured to input the original audio speech or a digital representation of the original audio speech into a spacing character generator to insert spacing characters into the phonemes, wherein the spacing character generator is configured to identify pauses by analyzing an amplitude of an audio signal of the original audio speech (“AI SD processor 125 can further identify coughs, sneezes, pauses in speech and other non-verbal audio segments or non-verbal noises created by a speaker.”; paragraphs 55, 107, 159).
As per claim 12, Gupta et al. further disclose the one or more processors are further configured to input the original audio speech or a digital representation of the original audio speech into a pacing character generator to insert pacing characters into the phonemes, wherein the pacing character generator is configured to calculate a rate of speech (“For each vocal segment in the plurality of vocal segments pacing information is identified for each word or phoneme in each vocal segment…Pacing/prosody/rhythm (referred to herein after “pacing”) is the measurable time associated with each syllable, word, other phoneme, non-verbal speech (such as cough, laugh, gasp) or pauses in speech with 0.05 s resolution.”; paragraphs 12, 67).
As per claim 13, Gupta et al. further disclose the one or more processors are further configured to input the original audio speech or a digital representation of the original audio speech into a non-verbal character generator to insert non-verbal characters into the phonemes, wherein the non-verbal character generator is configured to recognize patterns in audio (“AI SD processor 125 can further identify coughs, sneezes, pauses in speech and other non-verbal audio segments or non-verbal noises created by a speaker”; paragraphs 15, 67, 76, 107).
As per claim 14, Gupta et al. teach a method for training a machine learning model, comprising:
a) acquiring an input text label in a first language(“translated audio is generated using the translated input transcription and meta information”; paragraph 16);
b) acquiring a language code corresponding to the first language(“meta translation generator 132 is provided with specific output languages 112 for each input language (e.g., English to German and Spanish to German or English to German and Spanish to French).”; paragraph 75);
c) acquiring a speaker identification(“Each vocal segment in the plurality of vocal segments includes a speaker identification to identify the speaker of each vocal segment.”; paragraph 12);
d) generating a modified text label by converting the translated text label into phonemes and performing one or more of the following steps: inserting spacing characters into the phonemes to denote pauses in speech; inserting pacing characters into the phonemes to denote the pace of speech; inserting non-verbal characters into the phonemes to denote non-verbal speech elements(“similar pacing includes less than or equal to 20% difference in hamming distance between phonetic characters and inclusion of pauses, breaths, and filler sounds in the proper locations… ability to modify translated transcription… AI SD processor 125 can further identify coughs, sneezes, pauses in speech and other non-verbal audio segments or non-verbal noises created by a speaker”; paragraphs 15, 67, 76, 107);
e) providing the modified text label, the language code, and speaker identification to a machine learning model configured to output a synthetic mel spectrogram (“by incorporating digital representations of the corresponding audio signal (e.g., MelSpectogram and/or raw audio waveforms) with generative AI.”; paragraphs 42, 84, 169).
f) comparing the synthetic mel spectrogram to an original mel spectrogram using a predetermined loss function to calculate a loss value; and g) repeating steps a) through f) until the loss value meets a predetermined threshold (“Training this AI audio translation generator 138 therefore further includes additional unique loss functions configured to detect loss or error between the generated emotion and meta characteristics and the training data…an additional second stage generator that was trained by custom GAN to convert MelSpectogram into raw audio waveform”; paragraphs 160, 169).
As per claim 15, Gupta et al. further disclose inputting the input text label into a phoneme generator to convert the input text label into phonemes, wherein the phoneme generator is a neural network trained on a dataset of phonetic representations of words (“text preprocessor 128 is configured to convert text into phoneme analysis and/or perform emotional/sentiment analysis.”; paragraphs 64- 66).
As per claim 16, Gupta et al. further disclose inserting spacing characters into the phonemes includes inputting the original mel spectrogram into a spacing character generator that is configured to identify pauses by analyzing an amplitude of an audio signal of the original mel spectrogram (“AI SD processor 125 can further identify coughs, sneezes, pauses in speech and other non-verbal audio segments or non-verbal noises created by a speaker.”; paragraphs 55, 107, 159).
As per claim 17, Gupta et al. further disclose inserting pacing characters into the phonemes includes inputting the original mel spectrogram into a pacing character generator that is configured to calculate a rate of speech (“For each vocal segment in the plurality of vocal segments pacing information is identified for each word or phoneme in each vocal segment…Pacing/prosody/rhythm (referred to herein after “pacing”) is the measurable time associated with each syllable, word, other phoneme, non-verbal speech (such as cough, laugh, gasp) or pauses in speech with 0.05 s resolution.”; paragraphs 12, 67).
As per claim 18, Gupta et al. further disclose inserting non-verbal characters into the phonemes includes inputting the original mel spectrogram into a non-verbal character generator that is configured to recognize patterns in audio (“AI SD processor 125 can further identify coughs, sneezes, pauses in speech and other non-verbal audio segments or non-verbal noises created by a speaker”; paragraphs 15, 67, 76, 107).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Rose et al. teach Automatic Voiceover Generation. Zhang et al. teach MULTILINGUAL SPEECH SYNTHESIS AND CROSS-LANGUAGE VOICE CLONING. Shimomura et al. teach performing a speech-synthesizing process based on outputs from the machine translation system.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEONARD SAINT-CYR whose telephone number is (571)272-4247. The examiner can normally be reached Monday- Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richemond Dorvil can be reached at (571)272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LEONARD SAINT-CYR/ Primary Examiner, Art Unit 2658
Prosecution Insights