DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments with respect to 35 U.S.C. 102 in regard to claims 1, 10 and 15 have been considered but are moot due to new grounds of rejection necessitated by amendments. See detailed rejection below.
Applicant's arguments with respect to 35 U.S.C. 101 in regard to claims 1-20 have been considered but are moot due to new grounds of rejection necessitated by amendments. See detailed rejection below.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 Abstract Idea.
Claims 1, 10 and 15 are directed to an abstract idea because the claims are geared toward the mental and linguistic process of language translation. The claims describe receiving spoken words, converting them chunk by chunk into an intermediate language text, and then converting that into a target language text. Translating speech and manipulating text are fundamental information processing activities that humans can perform mentally or on paper, placing these core steps squarely in the category of non-patentable abstract ideas.
The claims fail to integrate this abstract idea into a practical application that improves computer functionality. Although it mentions translating small audio “tokens” in real time “without buffering an entire sentence” to reduce latency, this merely states a desirable goal rather than describing a specific technical implementation. Reciting a desired outcome (faster processing by not waiting for full sentence) without specifying particular software architecture, unique algorithms, or specialized hardware keeps the claim focused on the abstract concept itself rather than a technical improvement.
Finally, the claims lack an inventive concept that adds significantly more to the abstract idea. Performing conventional language translation steps sequentially on a standard computing device and processing incoming streaming data without buffering, uses routine, well understood computing techniques. Simply instructing generic computer hardware to execute familiar translation tasks on data as it arrives piece by piece does not elevate the claims to patent eligible subject matter.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims are (i) mere instructions to implement the idea on a computer, and/or (ii) recitation of generic computer structure that serves to perform generic computer functions that are well-understood, routine, and conventional activities previously known to the pertinent industry. Viewed as a whole, these additional claim element(s) do not provide meaningful limitation(s) to transform the abstract idea into a patent eligible application of the abstract idea such that the claim(s) amounts to significantly more than the abstract idea itself. Therefore, the claim(s) are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter. There is no further improvement to the computing device.
Dependent claims 2-9, 11-14 and 16-20 are further recite an abstract idea performable by a human and do not amount to significantly more than the abstract idea as they do not provide steps other than what is conventionally known.
Claims 2, 14 and 20, no technical structure or computer improvements.
Claim 3, well-known output step to the abstract idea.
Claims 4 and 19, off the shelf machine learning tools.
Claim 5, NN techniques without providing a specific, patent eligible software hardware improvement.
Claims 6 and 11, merely categorizes the data input format without altering the fundamental abstract process.
Claim 7, 8 and 12, without providing inventive hardware or technical solution.
Claims 9, 13 and 16, conventional way organizing data.
Claims 17 and 18, conventional way of translating language.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2 and 4-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Arkhangorodsky et al. (US2023/0089902, hereinafter Arkhan) in view of Rao et al. (“Exploring Architectures, Data and Units for Streaming End-to-End Speech Recognition with RNN-Transducer”; IEEE 2017).
Claim 1,
Arkhan teaches a computer-implemented method, executed on a computing device (Arkhan identifies the server and end-user terminal as “computing devices” [0036] [Fig. 1]), comprising:
receiving, as source language speech, speech in a source language (Arkhan teaches that the system may “receive audio input from speaker A speaking Spanish” [0037] [Fig. 1]; Arkhan also states that the verbal description “from the first user in the first language, denoted as a first audio sequence, may be received by the live translation captioning system 340” [0052] [Fig. 3]);
generating intermediate language text (Arkhan teaches that the system may “receive audio input from speaker A speaking Spanish, transcribe the audio input into text in Spanish” [0037]; Arkhan further explains that the ASR subsystem “transcribes the incoming audio in the speaker’s language (e.g., first language) to be used as input to the MT 116”; the Spanish transcript is the claimed intermediate text under stated construction),
wherein the first token-based translation comprises translating tokens corresponding to successive segments of speech audio as the speech audio is streamed to the computing device (Arkhan teaches that “the input to the ASR subsystem may include a plurality of audio segments” and that, “for each of the plurality audio segments, the ASR subsystem may generate an output including a revised or an extended string”; Arkhan further states that contiguous ASR outputs may include “new words appended or changed” [0054] [Fig. 3]; the generated words are the claimed tokens corresponding to the successively processed audio segments; Arkhan teaches that “a participant’s speech in a first language is fed in a streaming fashion to the ASR subsystem”; Arkhan further states that the plurality of audio segments “is being streamed into the ASR subsystem” [0054]; also see [0036] [0040-0041] [Figs. 1 and 3]) and
(Arkhan states that its ASR to MT pipeline has a “tight latency requirement” for live captions [0044]; Arkhan defines “translation lag” as the difference between “when a word was spoken and when its translation is finalized on the screen” and states that it is valuable to generate captions “immediately after a speaker begins” [0046]; this measures latency across the speech input to translated text pipeline); and
generating target language text by performing a second token-based translation that translates the intermediate language text into text of a target language (Arkhan teaches that the system may “transcribe the audio input text in Spanish, and translate the Spanish text into English and display the English text” [0037]; Arkhan further teaches that the MT subsystem translates the ASR text “into the caption language selected by the reader,” and the system may perform “translate-K (translate every K new words)” [0043] [0054]; the MT output is “a first translated text in the second language selected by a second user” [0056]).
The difference between the prior art and the claimed invention is that Arkhan does not explicitly teach by performing a first token-based translation, without buffering an entire sentence or utterance of the source language speech prior to the first token-based translation.
Rao teaches by performing a first token-based translation (Rao teaches an end-to-end speech recognition model that “directly outputs the recognized transcript as a sequence of graphemes”; Rao additionally teaches using “sub-words units (‘wordpieces’) as output targets [Abstract] [pg. 193]; graphemes and wordpieces are discrete textual tokens),
without buffering an entire sentence or utterance of the source language speech prior to the first token-based translation (Rao explains that streaming transducer architectures “allow the output to be decoded as soon as the first input is encoded, without introducing additional latency incurred when processing the entire utterance at once” [1.] [pg. 194]; thus, token decoding begins from an initial portion of the speech rather than after the complete utterance is received and processed).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Arkhan with teachings of Rao by modifying the method and system for evaluating and improving live translation captioning system as taught by Arkhan to include by performing a first token-based translation, without buffering an entire sentence or utterance of the source language speech prior to the first token-based translation as taught by Rao for the benefit of improving the speech recognition performance through the use of sub-word units and thereby significantly reducing substitution errors (Rao [Abstract]).
Claim 2,
Arkhan further teaches the computer-implemented method of claim 1 wherein the intermediate language is English (Arkhan states “receive another audio input from speaker B responding in English, transcribe the audio input into text in English, and translate the English text into Spanish” [0037] [Fig. 1]).
Claim 4,
Rao further teaches the computer-implemented method of claim 1 wherein performing a first token-based translation of the source language speech into text of an intermediate language includes: utilizing a first neural network model to effectuate the first token-based translation of the source language speech into the intermediate language text (Rao states “we investigate training end-to-end SR models with recurrent neural network transducer (RNN-T): a streaming, all-neural, sequence-to-sequence architecture”; Rao further states that “the entire neural network.. directly outputs the recognized transcript as a sequence of graphemes” [Abstract] [pg. 193]; Rao teaches that the RNN-T encoder receives acoustic input frames, while the prediction and joint networks generate output-symbol probabilities; during inference, the model is fed “the next acoustic frame xt” and produces “the next output label probabilities,” thereby streaming recognition results [2.] [Fig. 1] [pgs. 194-195]).
Claim 5,
Rao further teaches the computer-implemented method of claim 4 wherein the first neural network model is a Recurrent Neural Network Transducer (RNN-T) model (Rao states, “we investigate training end-to-end SR models with recurrent neural network transducer (RNN-T)” [Abstract] [pg. 193]; Rao further states “in this work we only consider streaming recognition architectures, specifically the RNN-T model” [1.] [pg. 194]).
Claim 6,
Rao further teaches the computer-implemented method of claim 4 wherein the first token-based translation utilizes time-based tokens to translate the source language speech into the intermediate language text (Rao teaches that the SR operates on an acoustic input sequence x1…xt, where T is the number of frames in the utterance; Rao’s encoder converts “the input acoustic frame xt” into a representation conditioned on the preceding frames x0…xt; the joint network then produces a distribution over the next output symbol [1.] [eq. 1] [2.] [eqs. 4-6] [pgs. 193-194]).
Claim 7,
Arkhan further teaches the computer-implemented method of claim 1 wherein performing a second token-based translation from the intermediate language text into text of a target language includes: utilizing a second neural network model to effectuate the second token-based translation from the intermediate language text into the target language text (Arkhan teaches that “the MT subsystem may include a single multilingual neural machine translation model trained based on a joint set of corpora from a plurality of languages or a plurality of MR models” [0055]; the summary similarly states that the “MT subsystem comprises a multilingual neural machine translation model” [0012]; Arkhan states that “each of the transcribed text returned by the ASR subsystem may be fed to the MT subsystem to translate it into the caption language selected by the reader”; the MT output is “a first translated text in the second language selected by a second user” [0054-0056]).
Claim 8,
Rao further teaches the computer-implemented method of claim 7 wherein the second neural network model is a Recurrent Neural Network Transducer (RNN-T) model (Rao teaches “end-to-end speech recognition models with the recurrent neural network transducer (RNN-T)” [Abstract] [pg. 193]).
Claim 9,
Arkhan further teaches the computer-implemented method of claim 7 wherein the second token-based translation utilizes content-based tokens to translate the intermediate language text into the target language text (Arkhan teaches that the system includes the setting “translate-K (translate every K new words)”; the ASR output supplied to MT contains “new words appended or changed,” and the MT model translates those words into the reader’s selected language [0043-0044] [0054-0056]).
Claim 10,
A computer program product residing on a computer readable medium having a plurality of instructions stored thereon which, when executed by a processor, cause the processor to perform operations (Arkhan states “the software product may be stored in a storage medium, comprising a number of instructions to cause a computer device … to execute all or some steps of the methods” [0097-0098] [0024]) comprising:
receiving, as source language speech, speech in a source language (Arkhan states that the system may “receive audio input from speaker A speaking Spanish”; Arkhan also receives a “first audio sequence” containing a verbal description in a first language [0037] [0052] [Figs. 1 and 3]).
generating intermediate language text (Arkhan states that the system may “transcribe the audio input into text in Spanish”; the ASR subsystem “processes the incoming audio sequences in a streaming mode” and transcribes the audio “in the speaker’s language … to be used as input to the MT” [0037] [0041] [Fig. 1]); and
generating target language text by performing a second token-based translation that translates the intermediate language text into text of a target language (Arkhan states that the system may “transcribe the audio input into text in Spanish, and translate the Spanish text into English and display the English text”; Arkhan also teaches a translate-K setting that will “translate every K new words” [0037] [0043]);
as the speech audio is streamed to a computing device (Arkhan states “a participant’s speech in a first language is fed in a streaming fashion to the ASR subsystem”; the ASR input includes “a plurality of audio segments that is being steamed into the ASR subsystem”, and an output is generated for each segment [0054] [Fig. 3]; the ASR subsystem is implemented by the disclosed server or terminal computer devices [0036] [0040-0041]) and
reducing end-to-end latency (Arkhan states that its ASR to MT pipeline has a “tight latency requirement” for live captions [0044]; Arkhan defines “translation lag” as the difference between “when a word was spoken and when its translation is finalized on the screen” and states that it is valuable to generate captions “immediately after a speaker begins” [0044-0046]; this measures latency across the speech input to translated text pipeline).
The difference between the prior art and the claimed invention is that Arkhan does not explicitly teach by performing a first token-based translation, wherein performing a first token-based translation of the source language speech into text of an intermediate language includes utilizing a first neural network model to translate tokens corresponding to successive segments of speech audio and without buffering an entire sentence or utterance of the source language speech prior to the first token-based translation and wherein the first neural network model is a Recurrent Neural Network Transducer (RNN-T) model.
Rao teaches by performing a first token-based translation (Rao teaches an end-to-end speech recognition model that “directly outputs the recognized transcript as a sequence of graphemes”; Rao additionally teaches using “sub-words units (‘wordpieces’) as output targets [Abstract] [pg. 193]; graphemes and wordpieces are discrete textual tokens),
wherein performing a first token-based translation of the source language speech into text of an intermediate language includes utilizing a first neural network model (Rao states that “we investigate training end-to-end speech recognition models with the RNN-T: a streaming, all neural, sequence to sequence architecture” [Abstract] [pg. 193]),
to translate tokens corresponding to successive segments of speech audio (Rao further teaches, during inference, “the RRN-T model is fed the next acoustic frame xt” and “produces the next output label probabilities”; if a blank label is produced, “the next acoustic frame, xt+1, is used to update the encoder” [2.] [pg. 195] [Fig. 1]) and
without buffering an entire sentence or utterance of the source language speech prior to the first token-based translation (Rao further teaches streaming transducer architectures “allow the output to be decoded as soon as the first input is encoded, without introducing additional latency incurred when processing the entire utterance at once” [1.] [pg. 194]) and
wherein the first neural network model is a Recurrent Neural Network Transducer (RNN-T) model (Rao states, “we investigate training end-to-end speech recognition models with the RRN-T” and “in this work we only consider streaming recognition architectures, specifically the RNN-T model” [Abstract] [1.0] [pgs. 193-194]).
Claim 11,
The computer program product of claim 10 wherein the first token-based translation utilizes time-based tokens to translate the source language speech into the intermediate language text. (Claim 11 contains subject matter similar to claim 6, and thus is rejected under similar rationale)
Claim 12,
The computer program product of claim 10 wherein performing a second token-based translation from the intermediate language text into text of a target language includes: utilizing a second neural network model to effectuate the second token-based translation from the intermediate language text into the target language text, wherein the second neural network model is a Recurrent Neural Network Transducer (RNN-T) model. (Claim 12 contains subject matter similar to claims 7 and 8, and thus is rejected under similar rationale)
Claim 13,
The computer program product of claim 12 wherein the second token-based translation utilizes content-based tokens to translate the intermediate language text into the target language text. (Claim 13 contains subject matter similar to claim 9, and thus is rejected under similar rationale)
Claim 14,
The computer program product of claim 10 wherein the intermediate language is English. (Claim 14 contains subject matter similar to claim 2, and thus is rejected under similar rationale)
Claim 15,
Claim 15 contains subject matter similar to claim 1 and thus is rejected under similar rationale.
Claim 16,
Claim 16 contains subject matter similar to claim 13 and thus is rejected under similar rationale.
Claim 17,
Claim 17 contains subject matter similar to claim 10 and thus is rejected under similar rationale.
Claim 18,
Claim 18 contains subject matter similar to claim 10 and thus is rejected under similar rationale.
Claim 19,
Claim 19 contains subject matter similar to claim 4 and thus is rejected under similar rationale.
Claim 20,
Claim 20 contains subject matter similar to claim 14 and thus is rejected under similar rationale.
Claim 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Arkhangorodsky et al. (US2023/0089902, hereinafter Arkhan) in view of Rao et al. (“Exploring Architectures, Data and Units for Streaming End-to-End Speech Recognition with RNN-Transducer”; IEEE 2017) and further in view of Fantinuoli (US 11,704,507).
Claim 3,
Arkhan and Rao teach all the limitations in claim 1. The difference between the prior art and the claimed invention is that Arkhan nor Rao explicitly teach converting the target language text into speech of the target language to define target language speech.
Fantinuoli teaches converting the target language text into speech of the target language to define target language speech (Fantinuoli states “the text to speech model 170 may receive the translation from the machine translation model 160” and “may generate audible translated speech based on the translation” [col. 6 lines 39-46] [Fig. 1]; Fantinuoli further states that the TTS model “may be trained on a corpus of text in the target language mapped to audible speech in the target language” [col. 10 lines 26-50] [Fig. 4]).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the teachings of Arkhan with teachings of Fantinuoli by modifying the method and system for evaluating and improving live translation captioning system as taught by Arkhan to include converting the target language text into speech of the target language to define target language speech as taught by Fantinuoli for the benefit of engaging in a natural conversation without waiting for the translation to occur (real-time language translation) (Rao [Abstract]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHREYANS A PATEL whose telephone number is (571)270-0689. The examiner can normally be reached Monday-Friday 8am-5pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SHREYANS A. PATEL
Primary Examiner
Art Unit 2653
/SHREYANS A PATEL/Examiner, Art Unit 2659
