CTNF 18/589,358 CTNF 98731 Detailed Action This communication is in response to the Arguments and Amendments filed on 2/17/2026. Claims 1-12 and 21-28 are pending and have been examined. Claims 13-20 are cancelled. Claims 1-12 and 21-28 are rejected. Apparent priority: 2/28/2023. Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Information Disclosure Statement The information disclosure statement (IDS) submitted on 7/11/2024, 5/28/2024 and 5/17/2024 have been considered by the examiner. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-12 and 21-28 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Regarding Independent Claim 1, Claim 1 recites, “1. (Original) A method implemented by one or more processors, the method comprising: identifying a first language spectrogram that corresponds to ground truth first language audio data that captures a first language utterance that is spoken in a first language; [This relates to a human identifying a spectrogram that corresponds to an audio using auditory processes and processes of observation.] processing the first language spectrogram, using a multilingual encoder, to generate a multilingual embedding; [This relates to a mathematical process a human can perform using pen and paper.] processing the multilingual embedding, using a second language decoder, to generate a predicted second language spectrogram; [This relates to a mathematical process a human can perform using pen and paper.] processing the predicted second language spectrogram, using the multilingual encoder, to generate an additional multilingual embedding; [This relates to a mathematical process a human can perform using pen and paper.] processing the additional multilingual embedding, using a first language decoder, to generate a predicted first language spectrogram; [This relates to a mathematical process a human can perform using pen and paper.] generating a back-translation loss based on comparing the predicted first language spectrogram to the first language spectrogram; and [This relates to a mathematical process a human can perform using pen and paper.] updating the first language decoder and the second language decoder based on the back-translation loss. [This relates to a mathematical process a human can perform using pen and paper.] Regarding Independent Claim 21 , claim 21 is a system claim with limitations similar to that of claim 1 and is rejected under the same rationale. The Dependent Claim does not include additional limitations that could incorporate the abstract idea into a practical application or cause the Claim as a whole to amount to significantly more than the underlying abstract idea. This judicial exception is not integrated into a practical application. In particular, claim 1 and 21 recites additional elements of “memory storing instructions; and one or more processors” For example, in [0098] there is the description of software module executed by processors. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of using a processor and memory is noted as a general computer. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Further, the additional limitation in the claims noted above are directed towards insignificant solution activity. The claims are not patent eligible. Dependent claim 2 recites, “2. (Original) The method of claim 1, further comprising: identifying a second language spectrogram that corresponds to ground truth second language audio data that captures a second language utterance that is spoken in a second language; [This relates to a human identifying a spectrogram that corresponds to audio data using auditory processes and perception.] processing the second language spectrogram, using the multilingual encoder, to generate a further multilingual embedding; [this relates to a mathematical process a human can perform.] processing the further multilingual embedding, using the first language decoder, to generate an additional predicted first language spectrogram; [this relates to a mathematical process a human can perform.] processing the additional predicted first language spectrogram, using the multilingual encoder, to generate a further additional multilingual embedding; [this relates to a mathematical process a human can perform.] processing the further additional multilingual embedding, using the second language decoder, to generate an additional predicted second language spectrogram; [this relates to a mathematical process a human can perform.] wherein generating the back-translation loss is further based on comparing the additional predicted second language spectrogram to the second language spectrograms. [this relates to a mathematical process a human can perform.]No additional limitations present. Dependent claim 3 recites, “3. (Original) The method of claim 2, wherein the generating the back-translation loss comprises: generating a first-to-second language back-translation loss based on comparing the predicted first language spectrogram to the first language spectrogram; [this relates to a mathematical process a human can perform.] generating a second-to-first language back-translation loss based on comparing the additional predicted second language spectrogram to the second language spectrogram; and [this relates to a mathematical process a human can perform.] generating the back-translation loss based on a sum of the first-to-second language back-translation loss and the second-to-first language back-translation loss. [this relates to a mathematical process a human can perform.] No additional limitations present. Dependent claim 4 recites, “4. (Original) The method of claim 1, further comprising updating the multilingual encoder based on the back-translation loss. [This relates to a human updating an encoder using physical processes and logic and reasoning.] No additional limitations present. Dependent claim 5 recites, “5. (Original) The method of claim 4, further comprising: generating a multilingual unsupervised and supervised embeddings (MUSE) loss based on the multilingual embedding; wherein updating the multilingual encoder is further based on the MUSE loss. [this relates to a mathematical process a human can perform.] No additional limitations present. Dependent claim 6 recites, “6. (Original) The method of claim 1, further comprising: prior to updating the first language decoder and the second language decoder based on the back-translation loss: training the multilingual decoder, the first language decoder, and the second language decoder during an auto-encoding, reconstruction training phase [this relates to a mathematical process a human can perform.] No additional limitations present. Dependent claim 7 recites, “7. (Original) The method of claim 1, further comprising: identifying an additional instance first language spectrogram that corresponds to additional ground truth first language audio data that captures an additional first language utterance that is spoken in the first language; [This relates to a human using auditory processes to capture audio] processing the additional instance first language spectrogram, using the multilingual encoder, to generate an additional instance multilingual embedding; [this relates to a mathematical process a human can perform.] processing the additional instance multilingual embedding, using a third language decoder, to generate a predicted third language spectrogram; [this relates to a mathematical process a human can perform.] processing the predicted third language spectrogram, using the multilingual encoder, to generate an additional additional instance multilingual embedding; [this relates to a mathematical process a human can perform.] processing the additional additional instance multilingual embedding, using the first language decoder, to generate an additional instance predicted first language spectrogram; [this relates to a mathematical process a human can perform.] generating an additional instance back-translation loss based on comparing the additional instance predicted first language spectrogram to the additional instance first language spectrogram; and [this relates to a mathematical process a human can perform.] further updating the first language decoder and the third language decoder based on the back-translation loss. [this relates to a mathematical process a human can perform.] No additional limitations present. Dependent claim 8 recites, “8. (Original) The method of claim 7, further comprising: identifying an additional instance third language spectrogram that corresponds to additional ground truth third language audio data that captures a third language utterance that is spoken in a third language; [this relates to observation that a human can perform] processing the third language spectrogram, using the multilingual encoder, to generate a further additional instance multilingual embedding; [this relates to a mathematical process a human can perform.] processing the further additional instance multilingual embedding, using the first language decoder, to generate an additional instance predicted first language spectrogram; [this relates to a mathematical process a human can perform.] processing the additional instance predicted first language spectrogram, using the multilingual encoder, to generate a further additional instance multilingual embedding; [this relates to a mathematical process a human can perform.] processing the further additional instance multilingual embedding, using the third language decoder, to generate an additional instance predicted third language spectrogram; [this relates to a mathematical process a human can perform.] wherein generating the additional back-translation loss is further based on comparing the additional instance predicted third language spectrogram to the additional instance third language spectrogram. [this relates to a mathematical process a human can perform.] No additional limitations present. Dependent claim 9 recites, “9. (Original) The method of claim 1, further comprising: subsequent to updating the first language decoder and the second language decoder based on the back-translation loss: using the multilingual encoder and the second language decoder in processing a new spectrogram, that corresponds to audio data that captures a new spoken utterance in the first language, to automatically generate a new predicted second language spectrogram that corresponds to generated audio data that includes a synthetic spoken utterance that is spoken in the second language and that corresponds, both linguistically and para-linguistically, to the new spoken utterance in the first language. [this relates to a mathematical and auditory processes a human can perform.] No additional limitations present. Dependent claim 10 recites, 10. (Original) The method of claim 9, further comprising causing the generated audio data to be rendered via one or more speakers of a client device. [This relates to a human speaking.] device speakers are noted as additional limitations. Dependent claim 11 recites, “11. (Original) The method of claim 9, wherein using the multilingual encoder and the second language decoder in processing the new spectrogram to automatically generate the new predicted second language spectrogram comprises: processing the new spectrogram, using the multilingual encoder, to generate a new multilingual embedding; and processing new multilingual embedding, using the second language decoder, to generate the new predicted second language spectrogram. [this relates to a mathematical process a human can perform.] No additional limitations present. Dependent claim 12 recites, “12. (Original) The method of claim 11, further comprising: determining that the new spoken utterance is to be automatically translated to the second language; [This relates to a human making a determination in the human mind.] in response to determining that the new spoken utterance is to be automatically translated to the second language: selecting the second language decoder, from multiple candidate language decoders including the second language decoder and the first language decoder, [this relates to a human making a selection in the human mind.] and using the selected second language decoder in processing the new multilingual embedding. [this relates to a mathematical process a human can perform.] No additional limitations present. As to dependent claim 22 , Claim 22 is a system claim with limitations similar to that of claim 2 and is rejected under the same rationale. As to dependent claim 23 , Claim 23 is a system claim with limitations similar to that of claim 3 and is rejected under the same rationale. As to dependent claim 24 , Claim 24 is a system claim with limitations similar to that of claim 4 and is rejected under the same rationale. As to dependent claim 25 , Claim 25 is a system claim with limitations similar to that of claim 5 and is rejected under the same rationale. As to dependent claim 26 , Claim 26 is a system claim with limitations similar to that of claim 6 and is rejected under the same rationale. As to dependent claim 27 , Claim 27 is a system claim with limitations similar to that of claim 7 and is rejected under the same rationale. As to dependent claim 28 , Claim 28 is a system claim with limitations similar to that of claim 9 and is rejected under the same rationale. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim s 1-5, 7-12, 21-25 and 27-28 are rejected under 35 U.S.C. 103 as being unpatentable over Jia (U.S. Patent Number US 20230013777 A1 ) Hereafter Jia (A), in view of Jia (U.S. Patent Number US 20210209315 A1 ) Hearafter Jia (B) . Regarding independent Claim 1, Jia (A) teaches 1. (Original) A method implemented by one or more processors, the method comprising: identifying a first language spectrogram that corresponds to ground truth first language audio data that captures a first language utterance that is spoken in a first language; processing the first language spectrogram, using a multilingual encoder, to generate a multilingual embedding; processing the multilingual embedding, using a second language decoder, to generate a predicted second language spectrogram; (see Jia (A) [0022] In the example shown, the direct S2ST model 200 is configured to convert input audio data 102 corresponding to an utterance 108 spoken in a first/source language (e.g., Spanish) by a source speaker 104 into output audio data (e.g., mel-spectrogram) 106 corresponding to a translated synthesized speech representation of a translated utterance 114 spoken in a different second language (e.g., English) by the source speaker 104. The direct S2ST model 200 may convert an input spectrogram corresponding to the input audio data 102 directly into an output spectrogram corresponding to the output audio data 102 without performing speech recognition and text-to-text machine translation, or otherwise without requiring the generation of any intermediate discrete representations (e.g., text or phonemes) from the input data 102. While described in greater detail with reference to FIGS. 2 and 3, the direct S2ST model 200 includes a spectrogram encoder 210, an attention module 220, a decoder 230, and a synthesizer (e.g., a spectrogram decoder) 300.”) processing the additional multilingual embedding, using a first language decoder, to generate a predicted first language spectrogram; (see Jia (A) [0023] “A vocoder 375 may synthesize the output audio data 106 output from the direct S2ST model 200 into a time-domain waveform for audible output as the translated utterance 114 spoken in the second language and in the voice of the source speaker. A time-domain audio waveform includes an audio waveform that defines an amplitude of an audio signal over time. The lieu of a vocoder 375, a unit selection module or a WaveNet module may instead synthesize the output audio data 106 into time-domain waveforms of synthesized speech in the translated second language and in the voice of the source speaker 104. In some implementations, the vocoder 375 includes a vocoder network, i.e., neural vocoder, which is separately trained and conditioned on mel-frequency spectrograms for conversion into time-domain audio waveforms.”) generating a back-translation loss based on comparing the predicted first language spectrogram to the first language spectrogram; and updating the first language decoder and the second language decoder based on the back-translation loss. [0025-0026] Notably, the direct S2ST model 200 is trained to retain voice characteristics of the source speaker such that the output audio data 106 corresponding to the synthesized speech representation and resulting translated utterance 114 conveys the voice of the source speaker, but in the different second language. Put another way, the translated utterance 114 conveys the voice characteristics of the source speaker 104 (e.g., speaking style/prosody) as the source speaker 104 would actually speak the different second language. In some examples, and described in greater detail below, the direct S2ST model 200 is trained to not only retain the voice characteristics of the source speaker in output audio data 106, but also prevent the ability to generate speech in a voice different from the source speaker to mitigate misuse of the model 200 for creating spoofing audio artifacts. [0026] A computing device associated with the source speaker 104 may capture the utterance 108 spoken by the source speaker 104 in the source/first language (e.g., Spanish) and transmit the corresponding input audio data 102 to the direct S2ST model 200 for conversion into the output audio data 106. Thereafter, the direct S2ST model 200 may transmit the output audio data 106 corresponding to the translated synthesized speech representation of the translated utterance 114 to another computing device 116 associated with recipient user 118, whereby the other computing device 116 audibly outputs the translated synthesized speech representation as the translated utterance 114 in the different second language (e.g., English). In this example, the source speaker 104 and the user 118 are speaking with each other through their respective computing devices 110, 116, such as over an audio/video call (e.g., video meeting/chat) telephone call or other type of voice communication protocol, for example, voice over internet protocol.”) Jia (A) does not specifically teach processing the predicted second language spectrogram, using the multilingual encoder, to generate an additional multilingual embedding; (However, Jia (B) does teach this limitation (see Jia (B) [0046] “…Any different pair of languages can be used..”) (see Jia (B) “[0080] The speaker encoder model 502 can obtain reference speech 504 from a target speaker and can generate a speaker embedding 506 descriptive of the target speaker's speech. The speaker embedding 506 can be input into the attention model 306 alongside the series of hidden state representations 304. For example, a concatenation operation 508 can be performed to concatenate the speaker embedding 506 to (e.g., each of) the series of hidden state representations 304 prior to input into the attention model 306. The reference speech 504 can be in the first language or the second language.”) Jia (A) and Jia (B) are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Jia (A) to incorporate processing the predicted second language spectrogram, using the multilingual encoder, to generate an additional multilingual embedding; of Jia (B). This allows for direct translation of speech from one language into speech in another language without relying on an intermediate text representation as recognized by Jia (B) Abstract. As to Independent claim 21 , claim 21 is a system claim with limitations similar to that of claim 1 and is rejected under the same rationale. Furthermore Jia (A) teaches A system comprising: memory storing instructions; and one or more processors operable to execute the instructions to: (see Jia (A) [0040] “The non-transitory memory may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. The non-transitory memory may be volatile and/or non-volatile addressable semiconductor memory.”) As to Claim 2, Jia (A) in view of Jia (B) teaches . 2. (Original) The method of claim 1, Furthermore, Jia (B) teaches further comprising: identifying a second language spectrogram that corresponds to ground truth second language audio data that captures a second language utterance that is spoken in a second language; processing the second language spectrogram, using the multilingual encoder, to generate a further multilingual embedding; processing the further multilingual embedding, using the first language decoder, to generate an additional predicted first language spectrogram; processing the additional predicted first language spectrogram, using the multilingual encoder, to generate a further additional multilingual embedding; processing the further additional multilingual embedding, using the second language decoder, to generate an additional predicted second language spectrogram; wherein generating the back-translation loss is further based on comparing the additional predicted second language spectrogram to the second language spectrogram. (see Jia (B) [0028] To train the machine-learned translation model, a computing system can obtain a set of training data that includes a number of pairs of training examples. Each training example can include data descriptive of first speech in a first language and data descriptive of second speech in a second language, wherein the second speech includes the same content (e.g., underlying expression or statement) as the first speech, but in a different language. As an example, the data descriptive of the first speech and second speech can be digital recordings of the first speech and second speech which may take the form of digitized speech waveforms. The computing system can generate a respective series of acoustic feature representations for both the first speech and the second speech. Example acoustic feature representations include various forms of spectrograms such as, as examples, linear frequency spectrograms or logarithmic frequency spectrograms (e.g., log-mel spectrograms). In particular, the series of acoustic feature representations generated based on the data descriptive of the second speech can be treated as ground truth acoustic feature representations which the translation model will attempt to predict.”) Jia (A) and Jia (B) are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of combination of Jia (A) and Jia (B) to incorporate further comprising: identifying a second language spectrogram that corresponds to ground truth second language audio data that captures a second language utterance that is spoken in a second language; processing the second language spectrogram, using the multilingual encoder, to generate a further multilingual embedding; processing the further multilingual embedding, using the first language decoder, to generate an additional predicted first language spectrogram; processing the additional predicted first language spectrogram, using the multilingual encoder, to generate a further additional multilingual embedding; processing the further additional multilingual embedding, using the second language decoder, to generate an additional predicted second language spectrogram; wherein generating the back-translation loss is further based on comparing the additional predicted second language spectrogram to the second language spectrogram of Jia (B). This allows for direct translation of speech from one language into speech in another language without relying on an intermediate text representation as recognized by Jia (B) Abstract. As to Claim 3, Jia (A) in view of Jia (B) teaches 3. (Original) The method of claim 2, Furthermore Jia (B) teaches wherein the generating the back-translation loss comprises: generating a first-to-second language back-translation loss based on comparing the predicted first language spectrogram to the first language spectrogram; generating a second-to-first language back-translation loss based on comparing the additional predicted second language spectrogram to the second language spectrogram; and generating the back-translation loss based on a sum of the first-to-second language back-translation loss and the second-to-first language back-translation loss. (see Jia (B) [0007] The operations may further comprise backpropagating a loss function through the machine-learned translation model to train the machine-learned translation model. The loss function may evaluate a respective difference between each of the series of output acoustic feature representations and a corresponding ground truth acoustic feature representation derived from the second speech in the second language. One or more auxiliary speech recognition models may be configure to receive and process information from the encoder model to predict one or more items of textual content associated with the first speech in the first language or the second speech in the second language. The operations may further comprise backpropagating one or more auxiliary loss functions respectively associated with the one or more auxiliary speech recognition models through at least a portion of the encoder model to train at least the portion of the encoder model. Each auxiliary loss function evaluates a respective difference between the one or more items of textual content output by the corresponding auxiliary speech recognition model and a corresponding ground truth item of textual content associated with the first speech in the first language or the second speech in the second language. The one or more auxiliary speech recognition models may comprise one or more first speech recognition models configured to receive and process the information from the encoder model to predict textual representations of one or more of phonemes, graphemes, words, or n-grams included in the first speech in the first language. The one or more auxiliary speech recognition models may comprise one or more second speech recognition models configured to receive and process the information from the encoder model to predict textual representations of one or more of phonemes, graphemes, words, or n-grams included in the second speech in the second language.”) Jia (A) and Jia (B) are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of combination of Jia (A) and Jia (B) to incorporate wherein the generating the back-translation loss comprises: generating a first-to-second language back-translation loss based on comparing the predicted first language spectrogram to the first language spectrogram; generating a second-to-first language back-translation loss based on comparing the additional predicted second language spectrogram to the second language spectrogram; and generating the back-translation loss based on a sum of the first-to-second language back-translation loss and the second-to-first language back-translation loss of Jia (B). This allows for direct translation of speech from one language into speech in another language without relying on an intermediate text representation as recognized by Jia (B) Abstract. As to Claim 4, Jia (A) in view of Jia (B) teaches 4. (Original) The method of claim 1, Furthermore, Jia (B) teaches further comprising updating the multilingual encoder based on the back-translation loss. (see Jia (B) [0007] The operations may further comprise backpropagating a loss function through the machine-learned translation model to train the machine-learned translation model. The loss function may evaluate a respective difference between each of the series of output acoustic feature representations and a corresponding ground truth acoustic feature representation derived from the second speech in the second language. One or more auxiliary speech recognition models may be configure to receive and process information from the encoder model to predict one or more items of textual content associated with the first speech in the first language or the second speech in the second language. The operations may further comprise backpropagating one or more auxiliary loss functions respectively associated with the one or more auxiliary speech recognition models through at least a portion of the encoder model to train at least the portion of the encoder model. Each auxiliary loss function evaluates a respective difference between the one or more items of textual content output by the corresponding auxiliary speech recognition model and a corresponding ground truth item of textual content associated with the first speech in the first language or the second speech in the second language. The one or more auxiliary speech recognition models may comprise one or more first speech recognition models configured to receive and process the information from the encoder model to predict textual representations of one or more of phonemes, graphemes, words, or n-grams included in the first speech in the first language. The one or more auxiliary speech recognition models may comprise one or more second speech recognition models configured to receive and process the information from the encoder model to predict textual representations of one or more of phonemes, graphemes, words, or n-grams included in the second speech in the second language.”) Jia (A) and Jia (B) are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of combination of Jia (A) and Jia (B) to incorporate further comprising updating the multilingual encoder based on the back-translation loss of Jia (B). This allows for direct translation of speech from one language into speech in another language without relying on an intermediate text representation as recognized by Jia (B) Abstract. As to Claim 5, Jia (A) in view of Jia (B) teaches 5. (Original) The method of claim 4, Furthermore, Jia (A) teaches further comprising: generating a multilingual unsupervised and supervised embeddings (MUSE) loss based on the multilingual embedding; wherein updating the multilingual encoder is further based on the MUSE loss. (see Jia (A) [0034] FIG. 3 provides an example of the synthesizer 300 of FIG. 1. Here, the synthesizer 300 may include a phoneme duration modeling network (i.e., duration predictor) 310, and upsampler module 320, a recurrent neural network (RNN) 330, and a convolutional layer 340. The duration modeling network receives the phoneme representation 235 from the decoder 230 and the context vector 224 from the attention module 220 as input. Moreover, the duration modeling network 310 is tasked with predicting a duration 315 for each phoneme in the phoneme representation 235 corresponding to the output audio data 106 that represents the translated synthesized speech representation in the target/second language. During training, an individual target duration 315 for each phoneme is unknown, thus, the duration model network 310 determines a target average duration based on a proportion of the T total frame duration of an entire reference mel-frequency spectrogram sequence and K total number of phonemes (e.g., tokens) in a reference phoneme sequence corresponding to the reference mel-frequency spectrogram sequence. That is, the target average duration is the average duration for all phonemes using the reference mel-frequency spectrogram sequence and the reference phoneme sequence used during training. During training, a loss term (e.g., L2 loss term) is then determined between the predicted phoneme durations and the target average duration. As such, the duration model network 310 learns to predict phoneme durations in an unsupervised manner without the use of supervised phoneme duration labels provided from an external aligner. While external aligners are capable of providing reasonable alignments between phonemes and mel-spectral frames, phoneme duration rounding is required by a length regulator to upsample phonemes in the reference phoneme sequence according to their duration which leads to rounding errors that may persist. In some instances, using supervised duration labels from the external aligner during training and using predicted durations during inference creates phoneme duration discrepancies between training the S2ST model 200 and inference of the S2ST model 200. Moreover, such rounding operations are not differentiable, and thus, an error gradient is unable to propagate through the duration model network.”) As to Claim 7, Jia (A) in view of Jia (B) teaches 7. (Original) The method of claim 1, Furthermore, Jia (A) teaches further comprising: identifying an additional instance first language spectrogram that corresponds to additional ground truth first language audio data that captures an additional first language utterance that is spoken in the first language; processing the additional instance first language spectrogram, using the multilingual encoder, to generate an additional instance multilingual embedding; processing the additional instance multilingual embedding, using a third language decoder, to generate a predicted third language spectrogram; processing the predicted third language spectrogram, using the multilingual encoder, to generate an additional additional instance multilingual embedding; processing the additional additional instance multilingual embedding, using the first language decoder, to generate an additional instance predicted first language spectrogram; (see Jia (A) [0024] “In the example shown, the source speaker 104 is a native speaker of the first/source language of Spanish. The direct S2ST 200 is accordingly trained to covert the input audio data 102 corresponding to utterances 108 spoken by the source speaker 104 in Spanish directly into the output audio data 106 corresponding to the translated synthesized speech representation corresponding to a translated utterance 114 in English (e.g., second/target language). That is, the translated utterance 114 in English (e.g., “Hi, what are your plans this afternoon?) includes synthesized audio for a translated version of the input utterance 108 that was spoken by the source speaker 104 in Spanish (e.g., “Hola, cuales son tus planes esta tarde?”). Thus, the translated synthesized representation provided by the output audio data 106 in English allows the native Spanish speaker to convey the utterance 108 spoken in Spanish to a recipient user 118 that natively speaks English. In some examples, the source speaker 104 does not speak English and the recipient speaker 118 does not speak/understand Spanish. In some implementations, the direct S2ST model 200 is a multilingual and trained to also convert input utterances spoken in English by speaker 118 into translated utterances in Spanish. In these implementations, the direct S2ST model 200 may be configured to convert speech between one or more other pairs of languages in addition to, or in lieu of, Spanish and English.”) generating an additional instance back-translation loss based on comparing the additional instance predicted first language spectrogram to the additional instance first language spectrogram; and further updating the first language decoder and the third language decoder based on the back-translation loss. (see Jia (A) [0034] “FIG. 3 provides an example of the synthesizer 300 of FIG. 1. Here, the synthesizer 300 may include a phoneme duration modeling network (i.e., duration predictor) 310, and upsampler module 320, a recurrent neural network (RNN) 330, and a convolutional layer 340. The duration modeling network receives the phoneme representation 235 from the decoder 230 and the context vector 224 from the attention module 220 as input. Moreover, the duration modeling network 310 is tasked with predicting a duration 315 for each phoneme in the phoneme representation 235 corresponding to the output audio data 106 that represents the translated synthesized speech representation in the target/second language. During training, an individual target duration 315 for each phoneme is unknown, thus, the duration model network 310 determines a target average duration based on a proportion of the T total frame duration of an entire reference mel-frequency spectrogram sequence and K total number of phonemes (e.g., tokens) in a reference phoneme sequence corresponding to the reference mel-frequency spectrogram sequence. That is, the target average duration is the average duration for all phonemes using the reference mel-frequency spectrogram sequence and the reference phoneme sequence used during training. During training, a loss term (e.g., L2 loss term) is then determined between the predicted phoneme durations and the target average duration. As such, the duration model network 310 learns to predict phoneme durations in an unsupervised manner without the use of supervised phoneme duration labels provided from an external aligner. While external aligners are capable of providing reasonable alignments between phonemes and mel-spectral frames, phoneme duration rounding is required by a length regulator to upsample phonemes in the reference phoneme sequence according to their duration which leads to rounding errors that may persist. In some instances, using supervised duration labels from the external aligner during training and using predicted durations during inference creates phoneme duration discrepancies between training the S2ST model 200 and inference of the S2ST model 200. Moreover, such rounding operations are not differentiable, and thus, an error gradient is unable to propagate through the duration model network.”) As to Claim 8, Jia (A) in view of Jia (B) teaches 8. (Original) The method of claim 7, Furthermore, Jia (A) teaches 8. (Original) The method of claim 7, further comprising: identifying an additional instance third language spectrogram that corresponds to additional ground truth third language audio data that captures a third language utterance that is spoken in a third language; processing the third language spectrogram, using the multilingual encoder, to generate a further additional instance multilingual embedding; processing the further additional instance multilingual embedding, using the first language decoder, to generate an additional instance predicted first language spectrogram; (see Jia (A) [0024] “In the example shown, the source speaker 104 is a native speaker of the first/source language of Spanish. The direct S2ST 200 is accordingly trained to covert the input audio data 102 corresponding to utterances 108 spoken by the source speaker 104 in Spanish directly into the output audio data 106 corresponding to the translated synthesized speech representation corresponding to a translated utterance 114 in English (e.g., second/target language). That is, the translated utterance 114 in English (e.g., “Hi, what are your plans this afternoon?) includes synthesized audio for a translated version of the input utterance 108 that was spoken by the source speaker 104 in Spanish (e.g., “Hola, cuales son tus planes esta tarde?”). Thus, the translated synthesized representation provided by the output audio data 106 in English allows the native Spanish speaker to convey the utterance 108 spoken in Spanish to a recipient user 118 that natively speaks English. In some examples, the source speaker 104 does not speak English and the recipient speaker 118 does not speak/understand Spanish. In some implementations, the direct S2ST model 200 is a multilingual and trained to also convert input utterances spoken in English by speaker 118 into translated utterances in Spanish. In these implementations, the direct S2ST model 200 may be configured to convert speech between one or more other pairs of languages in addition to, or in lieu of, Spanish and English.”) processing the additional instance predicted first language spectrogram, using the multilingual encoder, to generate a further additional instance multilingual embedding; processing the further additional instance multilingual embedding, using the third language decoder, to generate an additional instance predicted third language spectrogram; wherein generating the additional back-translation loss is further based on comparing the additional instance predicted third language spectrogram to the additional instance third language spectrogram. (see Jia (A) [0034] “FIG. 3 provides an example of the synthesizer 300 of FIG. 1. Here, the synthesizer 300 may include a phoneme duration modeling network (i.e., duration predictor) 310, and upsampler module 320, a recurrent neural network (RNN) 330, and a convolutional layer 340. The duration modeling network receives the phoneme representation 235 from the decoder 230 and the context vector 224 from the attention module 220 as input. Moreover, the duration modeling network 310 is tasked with predicting a duration 315 for each phoneme in the phoneme representation 235 corresponding to the output audio data 106 that represents the translated synthesized speech representation in the target/second language. During training, an individual target duration 315 for each phoneme is unknown, thus, the duration model network 310 determines a target average duration based on a proportion of the T total frame duration of an entire reference mel-frequency spectrogram sequence and K total number of phonemes (e.g., tokens) in a reference phoneme sequence corresponding to the reference mel-frequency spectrogram sequence. That is, the target average duration is the average duration for all phonemes using the reference mel-frequency spectrogram sequence and the reference phoneme sequence used during training. During training, a loss term (e.g., L2 loss term) is then determined between the predicted phoneme durations and the target average duration. As such, the duration model network 310 learns to predict phoneme durations in an unsupervised manner without the use of supervised phoneme duration labels provided from an external aligner. While external aligners are capable of providing reasonable alignments between phonemes and mel-spectral frames, phoneme duration rounding is required by a length regulator to upsample phonemes in the reference phoneme sequence according to their duration which leads to rounding errors that may persist. In some instances, using supervised duration labels from the external aligner during training and using predicted durations during inference creates phoneme duration discrepancies between training the S2ST model 200 and inference of the S2ST model 200. Moreover, such rounding operations are not differentiable, and thus, an error gradient is unable to propagate through the duration model network.”) As to Claim 9, Jia (A) in view of Jia (B) teaches 9. (Original) The method of claim 1, Furthermore, Jia (A) teaches further comprising: subsequent to updating the first language decoder and the second language decoder based on the back-translation loss: using the multilingual encoder and the second language decoder in processing a new spectrogram, that corresponds to audio data that captures a new spoken utterance in the first language, to automatically generate a new predicted second language spectrogram that corresponds to generated audio data that includes a synthetic spoken utterance that is spoken in the second language and that corresponds, both linguistically and para-linguistically, to the new spoken utterance in the first language. (see Jia (A) [0036] Implementations herein are further directed toward voice retaining techniques that restrict the trained S2ST model 200 to retain only the source speaker's voice, without the ability to generate synthesized speech in a different speaker's voice. This technique includes training on parallel utterances with the same speaker's voice on both the input utterance in a first language and the output utterance in a second language. Since fluent bilingual speakers are not prevalent, a cross-lingual TTS model may be employed to synthesize training utterances in the target second language that include the voice of the source speaker. Thus, the S2ST model 200 may train using utterances from the source speaker 104 in the first language and the synthesized training utterances of the source speaker 104 in the target second language. The S2ST model 200 can be further be trained to retain the source speakers voice in translated synthesized speech for each source speaker during speaker turns.”) As to Claim 10, Jia (A) in view of Jia (B) teaches 10. (Original) The method of claim 9, Furthermore, Jia (A) teaches further comprising causing the generated audio data to be rendered via one or more speakers of a client device. (see Jia (A) [0016] “The translated synthesize speech representation may be configured to a speaking style/prosody of the source speaker. In some implementations, the S2ST model is trained on pairs of parallel source language and target language utterances each including a voice spoken in the source utterance. In these implementations, at least one of the source language utterance or the target language utterance may include speech synthesized by a text-to-speech model trained to generate synthesized speech in the voice of the source utterance. In some examples, the operations further include receiving the translated synthesized speech representation at a vocoder of the S2ST model and synthesizing, by the vocoder, the translated synthesized speech representation into an audible output of the translated synthesized speech representation. Optionally, the phoneme representation may include a probability distribution of possible phonemes in a phoneme sequence corresponding to the translated synthesized speech representation.”) As to Claim 11, Jia (A) in view of Jia (B) teaches 11. (Original) The method of claim 9, Furthermore, Jia (A) teaches wherein using the multilingual encoder and the second language decoder in processing the new spectrogram to automatically generate the new predicted second language spectrogram comprises: processing the new spectrogram, using the multilingual encoder, to generate a new multilingual embedding; and processing new multilingual embedding, using the second language decoder, to generate the new predicted second language spectrogram. (see Jia (A) [0033] “The synthesizer 300 receives, as input during each of a plurality of output steps, a concatenation of the phoneme representation 235 (or phoneme sequence 245) and the context vector 225 at the corresponding output step and generates, as output at each of the plurality of output steps, the output audio data 106 corresponding to the translated synthesized speech representation in the target/second language and in the voice of the source speaker 104. Alternatively, the synthesizer 300 may receive the phoneme representation 235 and the context vector 225 (e.g., without any concatenation). The synthesizer 300 may also be referred to as a spectrogram decoder. In some examples, the synthesizer is autoregressive where each output spectrogram predicted is based on the sequence of previously predicted spectrograms. In other examples, the synthesizer 300 is parallel and non-autoregressive.”) As to Claim 12, Jia (A) in view of Jia (B) teaches 12. (Original) The method of claim 11, Furthermore, Jia (B) teaches further comprising: determining that the new spoken utterance is to be automatically translated to the second language; in response to determining that the new spoken utterance is to be automatically translated to the second language: selecting the second language decoder, from multiple candidate language decoders including the second language decoder and the first language decoder, and using the selected second language decoder in processing the new multilingual embedding. (see Jia (B) [0024] “Generally, the present disclosure is directed to systems and methods that train and use machine-learned models such as, for example, sequence-to-sequence models, to perform direct and text-free speech-to-speech translation. In particular, aspects of the present disclosure provide an attention-based sequence-to-sequence neural network which can directly translate speech from one language into speech in another language, without relying on an intermediate text representation. According to one aspect of the present disclosure, the machine-learned models described herein can be trained end-to-end, learning to map acoustic feature representations (e.g., spectrograms) of speech in a first language (e.g., Spanish) directly into acoustic feature representations (e.g., spectrograms) of speech in a second language (e.g., English). For example, the speech in the second language can correspond to translated content of the speech in the first language (e.g., which may also be in a different voice). According to another aspect of the present disclosure, additional techniques are provided which enable the machine-learned models to synthesize the translated speech using the voice of a target speaker (e.g., which may, in some instances, be the source speaker that is uttering the speech to be translated). According to yet another aspect of the present disclosure, multitask training can be performed by incorporating auxiliary decoder networks to predict the source or target transcripts, thereby improving performance of the machine-learned translation models. Finally, U.S. Provisional Patent Application No. 62/826,258, which is incorporated into and forms a portion of this disclosure, describes example experiments performed on example implementations of the systems and methods described herein which demonstrate the feasibility of the proposed approaches.”) Jia (A) and Jia (B) are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of combination of Jia (A) and Jia (B) to incorporate further comprising: determining that the new spoken utterance is to be automatically translated to the second language; in response to determining that the new spoken utterance is to be automatically translated to the second language: selecting the second language decoder, from multiple candidate language decoders including the second language decoder and the first language decoder, and using the selected second language decoder in processing the new multilingual embedding of Jia (B). This allows for direct translation of speech from one language into speech in another language without relying on an intermediate text representation as recognized by Jia (B) Abstract. As to dependent claim 22 , Claim 22 is a system claim with limitations similar to that of claim 2 and is rejected under the same rationale. As to dependent claim 23 , Claim 23 is a system claim with limitations similar to that of claim 3 and is rejected under the same rationale. As to dependent claim 24 , Claim 24 is a system claim with limitations similar to that of claim 4 and is rejected under the same rationale. As to dependent claim 25 , Claim 25 is a system claim with limitations similar to that of claim 5 and is rejected under the same rationale. As to dependent claim 27 , Claim 27 is a system claim with limitations similar to that of claim 7 and is rejected under the same rationale. As to dependent claim 28 , Claim 28 is a system claim with limitations similar to that of claim 9 and is rejected under the same rationale . 07-21-aia AIA Claims 6 and 26 are r ejected under 35 U.S.C. 103 as being unpatentable over J ia (A) (U.S. Patent Number US 20230013777 A1 ) in view of Jia (B) (U.S. Patent Number US 20210209315 A1 ), and further in view of Zheng (U.S. Patent Number US 20240119862 A1 ), A s to dependent Claim 6, Jia (A) in view of Jia (B) teaches 6. (Original) The method of claim 1, Jia (A) in view of Jia (B) do not specifically teach further comprising: prior to updating the first language decoder and the second language decoder based on the back-translation loss: training the multilingual decoder, the first language decoder, and the second language decoder during an auto-encoding, reconstruction training phase. However, Zheng does teach this limitation (see Zheng [0080] In step 314 of the converter training preparation process 300 shown in FIG. 3, the divided textual segments are embedded into one-hot encoding embeddings. These divided textual segments are those segments that were produced in step 310. For languages which inherently have one syllable per word so that step 310 was skipped in the converter training preparation process 300, the words themselves represent the divided textual segments for step 314. Embeddings are dense numerical representations of real-world objects and relationships, expressed as a vector. The divided textual segments may as a part of step 314 be input into an autoencoder or predictor in order to generate these one-hot encoding embeddings. The autoencoder or predictor may as an output produce the one-hot encoding embeddings in response to receiving the divided textual segments as input. The divided textual segments are input as a sequence of syllables. The sequence refers back to the original input text—with the various pieces (text syllables) together being arranged in the correct order to create the original word, phrase, sentence, and/or paragraph that was provided as input text. Similar to natural language processing of textual tokens to one-hot encoding embeddings, each divided textual segment is assigned a unique index of a vocabulary group that includes all distinct textual segments of different languages. A textual segment is converted to the corresponding one-hot encoding embedding based on the assigned index. That means that in a one-hot encoding embedding the dimension index is same as the assigned index of a textual segment, the corresponding dimension value is one, and the other dimension values are all zero.“) Jia (A) in view of Jia (B) and Zheng are in the same field of endeavor of signal processing, therefore, it would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method combination of Jia (A) and of Jia (B) to include prior to updating the first language decoder and the second language decoder based on the back-translation loss: training the multilingual decoder, the first language decoder, and the second language decoder during an auto-encoding, reconstruction training phase of Zheng This allows for improved pronounciation in a more precise manner as recognized by Zheng [0011]. As to dependent claim 26 , Claim 26 is a system claim with limitations similar to that of claim 6 and is rejected under the same rationale. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to KRISTEN MICHELLE MASTERS whose telephone number is (703)756-1274. The examiner can normally be reached M-F 8:30 AM - 5:00 PM. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Louis Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /KRISTEN MICHELLE MASTERS/Examiner, Art Unit 2659 /PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659 Application/Control Number: 18/589,358 Page 2 Art Unit: 2659 Application/Control Number: 18/589,358 Page 3 Art Unit: 2659 Application/Control Number: 18/589,358 Page 4 Art Unit: 2659 Application/Control Number: 18/589,358 Page 5 Art Unit: 2659 Application/Control Number: 18/589,358 Page 6 Art Unit: 2659 Application/Control Number: 18/589,358 Page 7 Art Unit: 2659 Application/Control Number: 18/589,358 Page 8 Art Unit: 2659 Application/Control Number: 18/589,358 Page 9 Art Unit: 2659 Application/Control Number: 18/589,358 Page 10 Art Unit: 2659 Application/Control Number: 18/589,358 Page 11 Art Unit: 2659 Application/Control Number: 18/589,358 Page 12 Art Unit: 2659 Application/Control Number: 18/589,358 Page 13 Art Unit: 2659 Application/Control Number: 18/589,358 Page 14 Art Unit: 2659 Application/Control Number: 18/589,358 Page 15 Art Unit: 2659 Application/Control Number: 18/589,358 Page 16 Art Unit: 2659 Application/Control Number: 18/589,358 Page 17 Art Unit: 2659 Application/Control Number: 18/589,358 Page 18 Art Unit: 2659 Application/Control Number: 18/589,358 Page 19 Art Unit: 2659 Application/Control Number: 18/589,358 Page 20 Art Unit: 2659 Application/Control Number: 18/589,358 Page 21 Art Unit: 2659 Application/Control Number: 18/589,358 Page 22 Art Unit: 2659 Application/Control Number: 18/589,358 Page 23 Art Unit: 2659 Application/Control Number: 18/589,358 Page 24 Art Unit: 2659 Application/Control Number: 18/589,358 Page 25 Art Unit: 2659 Application/Control Number: 18/589,358 Page 26 Art Unit: 2659 Application/Control Number: 18/589,358 Page 27 Art Unit: 2659 Application/Control Number: 18/589,358 Page 29 Art Unit: 2659 Application/Control Number: 18/589,358 Page 30 Art Unit: 2659
Prosecution Insights