DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments with respect to 35 U.S.C. 101 Abstract Idea in regards to claims 1-20 have been considered, however are not found to be persuasive due to the following reasons. In the first step of Alice/Mayo analysis, claims 1 and 11 fall within a statutory category. In the second step 2A, the claims are directed to an abstract idea because the focus of the claims are receiving language information, converting it from speech to text, translating or transliterating it into another language, and producing a speech representation of the translated phrase. These are language process and information conversion steps. A person could conceptually perform the core idea by listening to a phrase and instruction, writing down the phrase, translating it into another language, and speaking the translated phrase. The claims merely automates that language-conversion process using generic tools such as a speech recognizer, translator, language embedding, and TTS model. The claims do not improve how the speech recognizer, translator, embedding, or TTS model works. It only uses those components as tools to carry out the abstract idea and does not integrate it into a practical application. The Office explains that merely using a computer as a tool, or merely instructing a computer to apply an abstract idea, does not make the claims eligible.
Finally, the claims under Step 2B do not add an inventive concept. The additional elements, “data processing hardware,” “speech recognizer,” “translator”, “language embedding” and “TTS model”, are recited at a high-level of generality and perform their ordinary functions. The speech recognizer converts speech to text, the translator converts text into another language, the language embedding provides language information, and the TTS model generates an audio feature representation. The claims do not require a specific improved model architecture, a new training technique, a new embedding structure, or a technical improvement to speech synthesis. The elements amount to using conventional computer language processing components to implement the abstract idea. Thus, rejected under 101 Abstract Idea.
Applicant's arguments with respect to 35 U.S.C. 103 rejection in regards to claims 1 and 11 have been considered, however are not found to be persuasive due to the following reasons. Quidilig in view of Wang clearly teach the newly added limitations. See detailed rejection below.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101.
Claims 1 and 11 are directed to an abstract idea. The claims are directed to the abstract idea of translating spoken language from one language to another and converting it to speech, a fundamental linguistic process that humans perform naturally. The steps of receiving speech, recognizing it, translating it, and outputting synthesized speech in another language represent the kind of organizing human activity and mental process which are abstract. The recitation of generic computer components (a speech recognizer, translator, and TTS model) does not transform the nature of the claim, as these are simply computer-implemented analogs to conventional human translation tasks.
The claim limitations are further considered individually and as an ordered combination, do not integrate the abstract idea into a practical application in a meaningful way. The claim does not recite any particular improvement to the functioning of a computer or to speech processing technology itself.
There are no claims to a specific technical improvement in how speech recognition, translation accuracy, or TTS synthesis is performed. The claim simply puts together known technologies to achieve a result (translated synthesized speech) that is itself the abstract idea. The claim lacks an inventive concept sufficient to transform the abstract idea into patent-eligible subject matter. The use of a speech recognizer, a translator, and a TTS model are generic functional recitations that reflect conventional components well-known in the art.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims are (i) mere instructions to implement the idea on a computer, and/or (ii) recitation of generic computer structure that serves to perform generic computer functions that are well-understood, routine, and conventional activities previously known to the pertinent industry. Viewed as a whole, these additional claim element(s) do not provide meaningful limitation(s) to transform the abstract idea into a patent eligible application of the abstract idea such that the claim(s) amounts to significantly more than the abstract idea itself. Therefore, the claim(s) are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter. There is further no improvement to the computing device.
Dependent claims 2-10 and 12-20 further recite an abstract idea performable by a human and do not amount to significantly more than the abstract idea as they do not provide steps other than what is conventionally known in speech processing.
Claims 2 and 12: do not recite a specific technical improvement.
Claims 3 and 13: a mathematical model without a specific technological improvement.
Claims 4 and 14: not integrated into a practical application.
Claims 5 and 15: converting between data representations and outputting results without a particular application.
Claims 6 and 16: there is no inventive concept.
Claims 7 and 17: there is no inventive concept.
Claims 8 and 18: there is no inventive concept.
Claims 9 and 19: there is no inventive concept nor a technological improvement.
Claims 10 and 20: there is no inventive concept.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-4, 6-7, 10-11, 13-14, 16-17 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Quidilig et al. (WO 2010129056) in view of Wang et al. (“Tacotron: Towards End-to-End Speech Synthesis”; Apr. 6, 2017).
Claims 1 and 11,
Quidilig teaches a computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising ([0028] Quidilig teaches “The speech processing system 200 may be implemented as a dedicated processing hardware or as software executable on a general purpose processor”):
receiving a spoken input comprising an utterance spoken in a first language, the spoken input comprising a phrase and an instruction to synthesize the phrase into speech in a second language different than the first language ([0035] “Then, the user 10 is free to speak to the server 100 to effectuate his or her desired operation such as to send an email message merely by speaking to the server 100. The user's speech is obtained by the server 100 in step 300 and converted to text as the user input”; [0046] Quidilig teaches “In the present example, the user 10 then speaks (‘Sample Speech 1’) the following: ‘send email to John at domain dot com subject line test only email message hi John comma new line test only period question mark exclamation mark translate to Spanish send now’”; [0047] Quidilig further states: “the sound is converted into a stream of digitized electrical signals ("input audio stream") by the cellular telephone 14 and sent over the network 50 to the server 100”);
processing, using a speech recognizer, the spoken input to convert the spoken input into corresponding text in the first language ([0029] Quidilig states: “The function library 120 includes a number of facilities or functions such as, speech to text function 122…”; [0037] Quidilig further states: “At the server 100, the speech processing system 200 receives and processes the input audio stream by invoking the speech to text function 122. Step 310”; [0038] Quidilig also states: “The speech to text function 122 converts each audio segment into text that corresponds to the words spoken by the user during that audio segment using speech recognition techniques”);
processing, using a translator, the corresponding text in the first language to transliterate the corresponding text into translated text that recites the phrase in the second language ([0029] Quidilig states: “The function library 120 includes … text normalization function 128; and language translation functions 130”; [0063] Quidilig further states: “The input text base 156 is compared with a predetermined set of words or phrases: translate to; and match file … For example, phrase ‘translate to’ corresponds to the language translation operation 130”; [0064] Quidilig also states: “In the present example, the Optional Function is "translate" and its parameter, the Optional Function Parameter is "Spanish." Accordingly, in the present example, the Subject Line, the Message Text, or both are translated to Spanish, and the translated text, in Spanish, is then sent via email to the recipient”).
The difference between the prior art and the claimed inventio is that Quidilig does not explicitly teach obtaining a language embedding specifying language-dependent information associated with the second language; and processing, using a text-to-speech (TTS) model configured to receive the translated text that recites the phrase in the second language and the language embedding as input, the translated text that recites the phrase in the second language and the language embedding to generate an output audio feature representation as output from the TTS model, the output audio feature representation representing synthesized speech of the translated text that recites the phrase in the second language and having the language-dependent information specified by the language embedding.
Wang teaches obtaining a language embedding specifying language-dependent information associated with the second language ([1.] Wang states: “Second, it more easily allows for rich conditioning on various attributes, such as speaker or language, or high-level features like sentiment. This is because conditioning can occur at the very beginning of the model rather than only on certain components”; [Fig. 1] [Table 1] Fig. 1 labels the model input as “Character embeddings”; Table 1 lists “Character embedding 256-D”; [3.2] Wang also states: “The input to the encoder is a character sequence, where each character is represented as a one-hot vector and embedded into a continuous vector”); and
processing, using a text-to-speech (TTS) model configured to receive the translated text that recites the phrase in the second language and the language embedding as input, the translated text that recites the phrase in the second language and the language embedding to generate an output audio feature representation as output from the TTS model, the output audio feature representation representing synthesized speech of the translated text that recites the phrase in the second language and having the language-dependent information specified by the language embedding ([Abstract] Wang states: “In this paper, we present Tacotron, an end-to-end generative text-to-speech model that synthesizes speech directly from characters”; [Fig. 1 caption] Wang’s Fig. 1 states: “The model takes characters as input and outputs the corresponding raw spectrogram, which is then fed to the Griffin-Lim reconstruction algorithm to synthesize speech”; [3.] Wang states: “At a high-level, our model takes characters as input and produces spectrogram frames, which are then converted to waveforms”; [3.3] Wang further states “We use 80-band mel-scale spectrograms as the target …”; [3.4] “As mentioned above, the post-processing net’s task is to convert the seq2seq target to a target that can be synthesized into waveforms”; “The post-processing net learns to predict spectral magnitude sampled on a linear-frequency scale”; Wang also states: “We use the Griffin-Lim algorithm… to synthesize waveform from the predicted spectrogram”).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Quidilig with teachings of Wang by modifying the system and method for speech processing and speech to text as taught by Quidilig to include obtaining a language embedding specifying language-dependent information associated with the second language; and processing, using a text-to-speech (TTS) model configured to receive the translated text that recites the phrase in the second language and the language embedding as input, the translated text that recites the phrase in the second language and the language embedding to generate an output audio feature representation as output from the TTS model, the output audio feature representation representing synthesized speech of the translated text that recites the phrase in the second language and having the language-dependent information specified by the language embedding as taught by Wang for the benefit of generating speech at the frame level which is faster than sample-level autoregressive methods (Wang [Abstract]).
Claims 3 and 13,
Wang further teaches the computer-implemented method of claim 1, wherein processing the translated text that recites the phrase in the second language to generate the output audio feature representation as output from the TTS model comprises, for each of a plurality of time steps: processing, using an encoder neural network, a respective portion of translated text for the time step to generate a corresponding text encoding for the time step ([3.2 Encoder] the encoder is to extract robust sequential representations of text; the input to the encoder is a character sequence, where each character is represented as a one-hot vector and embedded into a continuous vector; A CBHG module transforms the pre-net outputs into the final encoder representation used by the attention module); and
processing, using a decoder neural network, the text encoding for the time step to generate a corresponding output audio feature representation for the time step ([3.3 Decoder] a stateful recurrent layer produces the attention query at each decoder time step; use 80-band mel-scale spectrogram as the target; the first decoder step is conditioned on an all-zero frame, which represents a <GO> frame, in inference, at decoder step t, the last frame of the r predictions is fed as input to the decoder at step t +1).
Claims 4 and 14,
Wang further teaches the computer-implemented method of claim 1, wherein the output audio feature representation comprises mel-frequency spectrograms ([3.3 Decoder] mel-scale spectrogram).
Claims 6 and 16,
Wang further teaches the computer-implemented method of claim 1, wherein the translated text corresponds to a character input representation ([3.2 Encoder] the input to the encoder is a character sequence).
Claims 7 and 17,
Wang further teaches the computer-implemented method of claim 1, wherein the translated text corresponds to a phoneme input representation ([2.] the model is trained on phoneme inputs).
Claims 10 and 20,
Quidilig further teaches the computer-implemented method of claim 1, wherein the first language comprises English and the second language comprises French ([0046] translate first language text (English) into a second language text (Spanish or another language); this purely a design choice).
Claim(s) 2 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Quidilig et al. (WO 2010129056) in view of Wang et al. (“Tacotron: Towards End-to-End Speech Synthesis”; Apr. 6, 2017) and further in view of Jia et al. (“Transfer Learning from Speaker Verification to Multispeaker Text-to-Speech Synthesis”; 2018).
Claims 2 and 12,
Quidilig and Wang teach all the limitations in claim 1. The difference between the prior art and the claimed invention is that Quidilig nor Wang explicitly teach obtaining a speaker embedding specifying specific voice characteristics of a target speaker for cloning a voice of the target speaker in synthesized speech, wherein processing, using the TTS model, the translated text further comprises processing, using the TTS model configured to receive the speaker embedding and the translated text that recites the phrase in the second language as input, the speaker embedding and the translated text to generate the output audio feature representation as output from the TTS model, the output feature representation representing the synthesized speech of the translated text that recites the phrase in the second language and that clones the voice of the target speaker.
Jia teaches obtaining a speaker embedding specifying specific voice characteristics of a target speaker for cloning a voice of the target speaker in synthesized speech ([Abstract] [Introduction] generate a fixed-dimensional embedding vector from only seconds of reference speech from a target speaker; few seconds of un-transcribed reference audio from a target speaker is used to synthesize new speech in that speaker’s voice; a speaker-discriminative embedding network that captures the space of speaker characteristics),
wherein processing, using the TTS model, the translated text further comprises processing, using the TTS model configured to receive the speaker embedding and the translated text that recites the phrase in the second language as input, the speaker embedding and the translated text to generate the output audio feature representation as output from the TTS model, the output feature representation representing the synthesized speech of the translated text that recites the phrase in the second language and that clones the voice of the target speaker ([Abstract] [2.] [2.2] generates a mel spectrogram from text, conditioned on the speaker embedding; an embedding vector for the target speaker is concatenated with the synthesizer encoder output at each time step; predicts a mel spectrogram from a sequence of grapheme or phoneme inputs, conditioned on the speaker embedding vector).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Quidilig and Wang with teachings of Jia by modifying the system and method for speech processing and speech to text as taught by Quidilig to include obtaining a speaker embedding specifying specific voice characteristics of a target speaker for cloning a voice of the target speaker in synthesized speech, wherein processing, using the TTS model, the translated text further comprises processing, using the TTS model configured to receive the speaker embedding and the translated text that recites the phrase in the second language as input, the speaker embedding and the translated text to generate the output audio feature representation as output from the TTS model, the output feature representation representing the synthesized speech of the translated text that recites the phrase in the second language and that clones the voice of the target speaker as taught by Jia for the benefit of showing that randomly sampled speaker embeddings can be used to synthesize speech in the voice of novel speakers dissimilar from those used in training, indicating that the model has learned a high quality speaker representation (Jia [Abstract]).
Claim(s) 5 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Quidilig et al. (WO 2010129056) in view of Wang et al. (“Tacotron: Towards End-to-End Speech Synthesis”; Apr. 6, 2017) and further in view of Tachibana et al. (US 2016/0012035).
Claims 5 and 15,
The computer-implemented method of claim 1, wherein the operations further comprise: inverting, using a waveform synthesizer, the output audio feature representation into a time-domain waveform ([3.4] we use the Griffin-Lim algorithm (Griffin & Lim, 1984) to synthesize waveform from the predicted spectrogram); and
The difference between the prior art and the claimed invention is that Quidilig nor Wang explicitly teach generating, using the time-domain waveform, a synthesized speech representation of the translated text that clones the voice of the target speaker in the second language.
Tachibana teaches generating, using the time-domain waveform, a synthesized speech representation of the translated text that clones the voice of the target speaker in the second language ([Abstract] [0012] [0033] carries out filtering using parameters of a spectral envelope representing vocal tract characteristics or the like to generate a speech waveform (Gaussian distribution at time T); second-language synthesis for a specific/target speaker and selecting output that sounds like the target speaker (select a speech synthesis dictionary of the specific speaker in the first language and a speech synthesis dictionary of the specific speaker in the second language).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Quidilig and Wang with teachings of Jia by modifying the system and method for speech processing and speech to text as taught by Quidilig to include generating, using the time-domain waveform, a synthesized speech representation of the translated text that clones the voice of the target speaker in the second language as taught by Tachibana for the benefit of improving the quality of synthetic speech (Tachibana [0004]).
Claim(s) 8 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Quidilig et al. (WO 2010129056) in view of Wang et al. (“Tacotron: Towards End-to-End Speech Synthesis”; Apr. 6, 2017) and further in view Luo et al. (CN 105702130).
Claims 8 and 18,
Quidilig and Wang teach all the limitations in claim 1. The difference between the prior art and the claimed invention is that Quidilig nor Wang explicitly teach wherein the translated text corresponds to an 8-bit Unicode Transformation Format (UTF-8) encoding sequence.
Luo teaches wherein the translated text corresponds to an 8-bit Unicode Transformation Format (UTF-8) encoding sequence ([Invention Contents] identity of the user character code is UTF-8, appointing the translation of source language and target language type).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Quidilig and Wang with teachings of Luo by modifying the system and method for speech processing and speech to text as taught by Quidilig to include wherein the translated text corresponds to an 8-bit Unicode Transformation Format (UTF-8) encoding sequence as taught by Luo for the benefit of realizing simple alternate for communication between deaf and normal person (Luo [Invention Contents]).
Claim(s) 9 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Quidilig et al. (WO 2010129056) in view of Wang et al. (“Tacotron: Towards End-to-End Speech Synthesis”; Apr. 6, 2017) and further in view Micheal et al. (JP 2007072927).
Claims 9 and 19,
Quidilig and Wang teach all the limitations in claim 1. The difference between the prior art and the claimed invention is that Quidilig nor Wang explicitly teach a first language training set comprising a plurality of utterances spoken in the first language and corresponding reference text; and a second language training set comprising a plurality of utterances spoken in the second language and corresponding reference text.
Micheal teaches a first language training set comprising a plurality of utterances spoken in the first language and corresponding reference text; and a second language training set comprising a plurality of utterances spoken in the second language and corresponding reference text ([Tech Solution] a first set of sentences in first language; a second set of sentences in the second language).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Quidilig and Wang with teachings of Michael by modifying the system and method for speech processing and speech to text as taught by Quidilig to include a first language training set comprising a plurality of utterances spoken in the first language and corresponding reference text; and a second language training set comprising a plurality of utterances spoken in the second language and corresponding reference text as taught by Micheal for the benefit of classifying a sentence into a good or bad translated sentence (Michael [Tech Solution]).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHREYANS A PATEL whose telephone number is (571)270-0689. The examiner can normally be reached Monday-Friday 8am-5pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SHREYANS A. PATEL
Primary Examiner
Art Unit 2653
/SHREYANS A PATEL/ Examiner, Art Unit 2659
Prosecution Insights