DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 Abstract Idea.
Claims 1, 11 and 12 are rejected under Abstract Idea, particularly a combination of mental processes and language translation, implemented using generic computer and speech processing components. Under step 2A, prong one, the claims recite receiving spoken information, converting speech into text, determining from a pause where a sentence ends, translating the sentence from one language to another, and converting the translated text back into speech. These steps describe the fundamental process performed by a human interpreter: listening to speech, recognizing words and sentence boundaries, translating the meaning into another language, and communicating the translation. The of ASR and TTS automates portion of this information processing activity but does not, by itself, remove the claim from the abstract idea category.
Under step 2A, prone two, the claim as a whole does not integrate the abstract idea into a practical application that improves the operation of the electronic device, ASR, translation, or TTS technology itself. The microphone merely receives speech, the ASR perform its ordinary function of converting speech to text, the translation operation performs its ordinary linguistic function, and TTS performs its ordinary function of converting text into synthetic speech. Identifying a sentence endpoint from a pause and generating translated synthetic speech from an earlier portion before the entire utterance ends may reduce the delay experienced by the listener, but the claim does not recite a particular technological mechanism, improved ASR, improved segmentation algorithm, or improved TTS technique that produces that result. Instead, the claimed components are used as tools to carry out the underlying translation process more quickly during a call.
Under step 2B, the additional elements do not provide an inventive concept sufficient to transform the abstract idea into patent eligible subject matter. The electronic device, microphone, ASR, language translation, and TTS components are recited according to their conventional functions, and the claim does not require a particular unconventional hardware arrangement or technological implementation. The requirement that synthetic speech corresponding to an earlier portion of the utterance be generated before the utterance ends principally specifies when the translated information is produced, rather than a technological improvement in how the computer performs speech recognition, translation or speech synthesis. Accordingly, considered individually and as an ordered combination, claims are characterized as using conventional computer technology to automate and accelerate the abstract process of interpreting spoken language.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims are (i) mere instructions to implement the idea on a computer, and/or (ii) recitation of generic computer structure that serves to perform generic computer functions that are well-understood, routine, and conventional activities previously known to the pertinent industry. Viewed as a whole, these additional claim element(s) do not provide meaningful limitation(s) to transform the abstract idea into a patent eligible application of the abstract idea such that the claim(s) amounts to significantly more than the abstract idea itself. Therefore, the claim(s) are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter. There is further no improvement to the computing device.
Dependent claims 2-10 and 13-20 further recite an abstract idea performable by a human and do not amount to significantly more than the abstract idea as they do not provide steps other than what is conventionally known.
Claims 2 and 13, abstract translation process using conventional communication technology.
Claims 3 and 14, does not improve the underlying computer or TTS technology.
Claims 4 and 15, does not recite a technological improvement to sentence detection or ASR.
Claims 5 and 16, abstract translation process using conventional audio processing functions.
Claims 6 and 17, provides a conventional UI.
Claims 7 and 18, does not improve the operation of the computer or TTS technology.
Claims 8 and 19, does not provide a technological improvement.
Claims 9, 10 and 20, does not improve the functioning of the computer or display technology.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4 and 11-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fantinuoli (US 11,704,507) in view of Waibel (US 2013/0144597).
Claims 1, 11 and 12,
Fantinuoli teaches a method performed by an electronic device during a call, the method comprising ([col. 1 line 65 to col. 2 line 17] Fantinuoli teaches a computer implemented automatic interpretation system usable during calls and video conferences; see Fig. 1):
receiving, via a microphone, an utterance from a user of the electronic device ([col. 4 lines 38-49] [col. 9 lines 12-19] [Figs. 1 and 4] [step 410] Fantinuoli states that computing system 100 receives audible speech from speech source 110 and that, where the source is an individual, “the audible speech may be received via microphone in communication with the computing device 100”; Fantinuoli states at step 410 that “the processor receives the audible speech via a microphone”);
based on a speech signal corresponding to a portion of the utterance, performing, by the electronic device, automatic speech recognition (ASR) to generate a first text in a first language ([col. 5 lines 3-22] [col. 9 lines 20-36] [Figs. 1 and 4] [step 420] Fantinuoli teaches that ASR model 120 “may receive the audible speech from the speech source 110 and generate a speech transcript of the audible speech,” that is “may generate speech transcript in real time,” and that is configured to generate the transcript from speech “in the source language”; step 420 likewise generates “in real-time, a speech transcript by applying an automatic speech recognition model on the audible speech,” where the transcript “may be in the source language”);
performing, by the electronic device, a text-to-speech (TTS) conversion on the second text ([col. 6 lines 39-46] [col. 10 lines 26-50] [Figs. 1 and 4] [steps 450-460] Fantinuoli teaches that “the text to speech model 170 may receive the translation from the machine translation model 160” and “may generate audible translated speech based on the translation”; step 460 similarly states that the processor “generates audible translated speech based on the translation of the speech by applying a text to speech model on the translation of the speech”); and
based on the TTS conversion, generating, by the electronic device, a synthetic speech corresponding to a portion of the utterance before an end of the utterance received from the user ([col. 3 lines 1-18] [col. 10 line 51 to col. 11 line 16] [claim 10] [fig. 4] Fantinuoli teaches generation of translated synthetic speech while the source speaker is still speaking; the method is performed “while receiving, by a processor, audible speech from a speech source” and generates “in real-time, an audible translated speech based on the received audible speech”; see claim 10; the detailed example is even more explicit: “as the doctor is speaking, the computer system 100” processes the speech and “generates a machine translation of the compressed speech segments and synthesis audible Swedish speech using the machine translation”; consequently, “as the doctor is explaining the injury in Spanish, the patient is hearing a low-latency Swedish translation of the explanation”; thus, translated synthetic speech corresponding to an earlier portion is generated while additional source speech is still being received – i.e., before the end of the continuing utterance).
The difference between the prior art and the claimed inventio is that Fantinuoli does not explicitly teach based on at least one pause section associated with the first text, identifying, by the electronic device, an end point of a sentence included in the first text; based on the identified end point of the sentence included in the first text, translating, by the electronic device, a portion of the first text corresponding to the sentence into a second text in a second language.
Waibel teaches based on at least one pause section associated with the first text, identifying, by the electronic device, an end point of a sentence included in the first text ([0029] [0061-0062] [Figs. 1, 5-7] Waibel teaches that ASR partial hypotheses are merged and resegmented after recognition into sentence like segments: “the partial hypotheses are merged together and resegmented to sentence-like segments,” and “segmentation is done at silence regions only”; Waibel states that the resegmentation unit may use “pause, prosodic cues, and/or language modeling” and “may construe a pause in the presentation as marking the end of a sentence by the speaker”; thus, Waibel uses a pause in connection with the recognized/resegmented text to identify the sentence endpoint);
based on the identified end point of the sentence included in the first text, translating, by the electronic device, a portion of the first text corresponding to the sentence into a second text in a second language ([0015] [0029] [0061] [Fig. 1] [elements 12, 22, 24 and 26] Waibel connects the detected sentence boundary to translation; speech recognition unit 12 produces partial hypotheses, which “are merged, filtered and resegmented by a resegmentation unit 22 using a boundary model 24,” after which “the processed hypotheses are transferred to a machine translation unit 26 for translation into another language; Waibel further teaches that because MT “awaits complete sentences as input,” the ASR hypotheses are resegmented into sentence-like segments. Para. [0061] establishes that a pause can mark the end of that sentence; Accordingly, the portion delimited by the pause-derived sentence endpoint is supplied for translation into the second language).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Fantinuoli with teachings of Waibel by modifying the systems and methods for automatic speech translation as taught by Fantinuoli to include based on at least one pause section associated with the first text, identifying, by the electronic device, an end point of a sentence included in the first text; based on the identified end point of the sentence included in the first text, translating, by the electronic device, a portion of the first text corresponding to the sentence into a second text in a second language as taught by Waibel for the benefit of providing translation service to a wide audience, making it possible to overcome language barriers and bring people closer together (Waibel [Background]).
Claims 2 and 13,
Fantinuoli further teaches the method of claim 1, further comprising: transmitting the synthetic speech toward a counterpart device before an end of a remaining portion of the utterance (Fantinuoli states that the system may “translate the received audio into a second language, generate a new audio in the second language, and transmit the translated audio (e.g. to the software solution, such as a particular participant of a video conferencing application)” [col. 2 lines 6-17]; Fantinuoli also teaches that listener 115 may itself be “an electronic device including one or more processors and non-transitory, processor-readable memory,” and that translated speech may be output by another computer/server associated with a video-conferencing platform connecting users [col. 2 lines 31-44]; Fantinuoli teaches simultaneous transmission/output of translated synthetic speech while the source speaker is still speaking… the method performs the translation “while receiving, by a processor, audible speech from a speech source” and generates “in real-time an audible translated speech based on the received audible speech [col. 3 lines 1-18]; see claim 10; see doctor/patient example in [col. 10 line 51 to col. 11 line 16]; translated synthetic speech for an earlier portion is being delivered to the counterpart while the source speaker continues speaking the remaining portion, i.e., before the remaining portion of the utterance has ended).
Claims 3 and 14,
Fantinuoli further teaches the method of claim 1, further comprising: determining a TTS conversion (Fantinuoli teaches performing TTS on each translated speech segment in its cascading simultaneous-translation architecture; the machine translation model 160 “may generate a translation of the compressed speech segments,” and immediately thereafter “the text to speech model 170 may receive the translation from the machine translation model 160” and “may generate audible translated speech based on the translation” [col. 6 lines 30-46] [Fig. 1]; the same sequence appears at steps 450-460 of Fig. 4)
Waibel further teaches at each time point at which an end point of a sentence is identified from the first text (Waibel teaches continuously producing ASR hypotheses and repeatedly identifying completed sentence segments from those hypotheses; to minimize latency, “the speech recognizer starts to decode while the speaker is talking and continuously returns partial back traces with first best hypotheses”; because MT requires complete sentences, “the partial hypotheses are merged together and resegmented to sentence-like segments,” with “segmentation… done at silence regions only” [0029]; Waibel further teaches that its resegmentation unit “may construe a pause in the presentation as marking the end of a sentence by the speaker” [0061]).
Fantinuoli teaches repeatedly processing the resulting speech segments through MT and TTS as part of continuous simultaneous translation. Thus, when Waibel’s pause-based endpoint determination is incorporated into Fantinuoli’s segmenter, each identified completed sentence becomes the translation unit that proceeds through Fantinuoli’s MT/TTS pipeline.
Claims 4 and 15,
Waibel further teaches the method of claim 1, wherein the identifying of the end point of the sentence included in the first text is based on a combination of at least one of information about the pause section, incoming token information after the pause section, or punctuation mark information (Waibel teaches using pauses to identify sentence endpoints; Waibel states that the resegmentation unit may use “pauses, prosodic cues, and/or language modeling” and specifically “may construe a pause in the presentation as marking the end of a sentence by the speaker”; Waibel further teaches that “segmentation is done at silence regions only” in certain embodiments [0029] [0061-0062]).
Claim(s) 5 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fantinuoli (US 11,704,507) in view of Waibel (US 2013/0144597) and further in view of Nagao (US 2008/0077390).
Claims 5 and 16,
Fantinuoli and Waibel teach all the limitations in claim 1. The difference between the prior art and the claimed invention is that Fantinuoli nor Waibel explicitly teach mixing a speech signal corresponding to the sentence in the speech signal and the synthetic speech; and outputting a result therefrom.
Nagao teaches mixing a speech signal corresponding to the sentence in the speech signal and the synthetic speech (Nagao teaches mixing source speech with translated synthetic speech; first Nagao defines translated speech as speech produced by “translating received speech and speech-synthesizing the resulting speech” and explains that the output speech can include both: “Therefore, the output speech includes the translated speech and the source speech” [0056] [0060]; Nagao’s translating unit generates the translated speech by performing “a speech recognition process, a machine translation process, and a speech synthesis process,” with speech synthesis generating translated speech from translated character string [0080-0083]; Nagao then provides that a speech storage unit stores “a mixed speech that includes the translated speeches or the source speeches,” and a saving unit “mixes the translated speech with the mixed speech … to generate a resulting mixed speech” [0014] [see claims 7-8 and 20]); and
outputting a result therefrom (Nagao outputs the mixed results; if speech already exists in storage, the output unit “generates the mixed speech, and stores the mixed speech as the speech corresponding to the speech duration”; “outputs the mixed speech” [0114-0015] [Fig. 7] [step S701]).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Fantinuoli and Waibel with teachings of Nagao by modifying the systems and methods for automatic speech translation as taught by Fantinuoli to include mixing a speech signal corresponding to the sentence in the speech signal and the synthetic speech; and outputting a result therefrom as taught by Nagao for the benefit of providing real time language translation so participants do not have to wait for speech to be translated (Nagao [0010]).
Claim(s) 6-8 and 17-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fantinuoli (US 11,704,507) in view of Waibel (US 2013/0144597) in view of Nagao (US 2008/0077390) and further in view of Cuthbert et al. (US 2015/0134322).
Claims 6 and 17,
Fantinuoli, Waibel and Nagao teach all the limitations of claim 5. The difference between the prior art and the claimed invention is that Fantinuoli, Waibel nor Nagao explicitly teach displaying an indicator for controlling an outputting of the synthetic speech.
Cuthbert teaches displaying an indicator for controlling an outputting of the synthetic speech (Cuthbert displays a graphical indictor associated with synthetic translated-speech output; when preparing the translation output, “a speaker icon 50 has replaced the microphone icon 30,” and the icon is animated to indicate that the translation application is preparing the output spoken translation [0038] [Fig. 5]; during actual output, “the speaker icon 50 has been highlighted to create a visual indication that the language translation application is outputting a spoken translation” [0041] [Fig. 6] [0052-0054] [Figs. 10-11).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Fantinuoli, Waibel and Nagao with teachings of Cuthbert by modifying the systems and methods for automatic speech translation as taught by Fantinuoli to include displaying an indicator for controlling an outputting of the synthetic speech as taught by Cuthbert for the benefit of sequencing the various inputs and outputs through the user interface (Cuthbert [0003]).
Claims 7 and 18,
Cuthbert further teaches the method of claim 6, wherein the indicator includes a user interface (UI) for controlling a combination of at least one of speed, volume, play, or stop of the synthetic speech (Cuthbert states “a language translation application on a user device may include a user interface that provides relevant textual and graphical feedback mechanisms associated with various states of voice input and translated speech” [0017]; Cuthbert further states that the UI provides “input/output controls” for the translation experience [0018]; speaker icon 50 is displayed as part of that UI [Figs. 5-6 and 10-11]; Cuthbert teaches the claimed stop control; “tapping the speaker icon… may stop any pending spoken translations from being output” [0057]; [0091] similarly refers to “manually tapping the speaker icon to stop the text from being spoken”; thus, the displayed UI indicator controls stopping synthetic translated speech).
Claims 8 and 19,
Cuthbert further teaches the method of claim 6, further comprising: automatically outputting the synthetic speech when the synthetic speech is generated; or outputting the synthetic speech according to a user input made through the indicator (Cuthbert’s state machine expressly links availability/generation of translated audio to output; [0074] states: “upon receiving a translation ready signal (e.g., the language translation application receives or generates an audio signal corresponding to a translation..), the user interface transitions to the output translation state 1712 (state E), in which the language translation application is outputting a spoken translation [Fig. 17]; the same process for the reverse language direction is disclosed in [0083]).
Claim(s) 9-10 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fantinuoli (US 11,704,507) in view of Waibel (US 2013/0144597) and further in view of Kivimaki (US 7,174,295).
Claim 9,
Fantinuoli and Waibel teach all the limitations in claim 1. The difference between the prior art and the claimed invention is that Fantinuoli nor Waibel explicitly teach differently displaying a portion of a text displayed on a display based on a time point related to the synthetic speech.
Kivimaki teaches differently displaying a portion of a text displayed on a display based on a time point related to the synthetic speech (Kivimaki teaches a controller that “controls the display to highlight text”; importantly, “highlighting” includes anything that visually differentiates a portion of text from the remaining body of text and may include contrast changes, color changes, underscoring, bold, italic.. or enclosing the text in shapes [0017]; thus, Kivimaki teaches differently displaying a portion of displayed text; Kivimaki further states that “the highlighting of the text is synchronized to the conversion of the text to audio output” [0018]; Kivimaki’s timed example is even more direct: “at time t1, a portion of text… is converted to audio output”; then “after a delay, at time t2, the controller 14 highlights the portion of text”; the process repeats at t3/t4, t5/t6, and t7/t8 for successive portions [0037-0044] [Fig. 3]; thus, which portion of the text is differently displayed is based upon a time point related to the corresponding synthetic speech).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Fantinuoli and Waibel with teachings of Kivimaki by modifying the systems and methods for automatic speech translation as taught by Fantinuoli to include differently displaying a portion of a text displayed on a display based on a time point related to the synthetic speech as taught by Kivimaki for the benefit of allowing the user to be directed to the relevant portion of text with the minimum of distraction (Kivimaki [col. 1 lines 53-59]).
Claim 10,
Kivimaki further teaches the method of claim 9, wherein the displaying comprises: changing a combination of at least one of color, font thickness, slant, size, or font type of the portion of the text displayed on the display (Kivimaki states that its controller “controls the display to highlight text” and defines highlighting as “anything which visually differentiates a portion of text from the remaining body of text on the display”; thus, Kivimaki changes the visual appearance of the particular displayed text portion [col. 3 line 64 to col. 4 line 28]; Kivimaki teaches “color variation of the text and/or display background” as a technique for visually differentiating/highlighting the text portion [col. 4 lines 11-20]).
Claim 20,
The electronic device of claim 12, wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to: based on a time point related to the synthetic speech, differently display a combination of at least one of color, font thickness, slant, size, or font type of a portion of a text displayed on a display. (Claim 20 contains subject matter similar to claims 9 and 10, and thus is rejected under similar rationale)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kivimaki (DE 60033122) – According to one Appearance of the present invention is a portable electronic device provided comprising a speech synthesizer having a Has speakers for that is set, an input depending on a highlighted Converting text into an audio output that personifies a person who reproduces the highlighted text orally; a user input device for entering commands to reverse or forward through highlighted text between positions highlighted by highlighting markers of the highlighted text are defined to a desired one Navigate highlighting markers; and a control unit, which one for that is set up, the navigation to the desired highlighting mark to regulate and input the speech synthesizer in response to provide navigational commands with an input that corresponds to a section corresponds to the highlighted text, which at the desired Highlighting mark begins.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHREYANS A PATEL whose telephone number is (571)270-0689. The examiner can normally be reached Monday-Friday 8am-5pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SHREYANS A. PATEL
Primary Examiner
Art Unit 2653
/SHREYANS A PATEL/ Examiner, Art Unit 2659
Prosecution Insights