DETAILED ACTION
This office action is in response to Applicant’s Request for Continued Examination (RCE), received on 01/29/2026. Claims 1, 9, and 16 have been amended. Claims 1-9, 11-16, 18-20 are pending and have been considered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/29/2026 has been entered.
Response to Arguments
Applicant’s arguments, see pgs. 9-14, filed 01/29/2026, with respect to the rejection(s) of independent claim(s) 1, 9, and 16 under 35 U.S.C. 102(a)(1) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Zheng et al. (“Duplex Sequence-to-Sequence Learning for Reversible Machine Translation”). Zheng has been previously cited for mapping to dependent claims 5 and 18. Zheng discloses a duplex translational model featuring a mirrored architecture with respect to the two directions of translation (see Fig. 2). See updated rejections below.
Applicant’s arguments with respect to dependent claims (see pgs. 11-14) have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (US-20200387676-A1), hereinafter Kim, in view of Zheng et al. (“Duplex Sequence-to-Sequence Learning for Reversible Machine Translation”), hereainfter Zheng.
Regarding claim 1, Kim discloses: a method comprising:
processing, using a duplex neural network (NN) ([0098] bi-directional translation by an electronic device, [0097] the electronic device 200 may use neural machine translation (NMT) [Bi-directional translation tracks to duplex translation, see [0003] of instant application. Further, applying a neural machine translation to a bi-directional translation indicates a bi-directional, i.e. duplex, neural network]), a first representation of a first speech utterance in a first language to obtain a second representation of a second speech utterance in a second language ([Fig. 1, First Directional Translation 130], [Converting text, i.e. a representation of a speech utterance, into a different language indicates obtaining a second representation of a second speech utterance in a second language, i.e. the translation]),
the second speech utterance comprising a translation of the first speech utterance to the second language ([0049] The electronic device 200-1 may translate the received utterance into the second language and output the translated result to the second speaker 120 [A received utterance tracks to a first speech in view of the first directional translation of Fig. 1]),
wherein the duplex NN comprises a first subnetwork and a second subnetwork ([Fig. 7, 701, 703]).
Kim is not relied upon to disclose:
the first subnetwork comprising a first plurality of NN layers arranged in a first configuration that includes residual connections for performing addition operations, and
the second subnetwork comprising a second plurality of NN layers arranged in a second configuration that includes residual connections for performing subtraction operations,
the second configuration being a mirror image of the first configuration.
Zheng is relied upon to disclose:
the first subnetwork comprising a first plurality of NN layers arranged in a first configuration that includes residual connections for performing addition operations ([Fig. 2, Regular Form, see addition operation performed based on output from SAN layer and FFN layer]), and
the second subnetwork comprising a second plurality of NN layers arranged in a second configuration that includes residual connections for performing subtraction operations ([Fig. 2, Reverse Form, see subtraction operation performed based on output from FFN layer and SAN layer]),
the second configuration being a mirror image of the first configuration ([In view of the operations and ordering of layers of the previously disclosed Regular and Reverse forms of translation, the examiner asserts that the second configuration, i.e. reverse, is a mirror image of the first configuration, i.e. regular. See Fig. 2 in its entirety]).
Kim and Zheng are considered analogous art within reversible machine translation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim to incorporate the teachings of Zheng, because of the novel way to use one model for duplex (Zheng, [Introduction, Par. 3-4]).
Regarding claim 2, Kim in view of Zheng discloses: the method of claim 1.
Kim further discloses:
wherein the duplex NN is deployed in a first direction ([Fig. 7, see “Language1 > Language2 TRANSLATION”]), and the duplex NN is further trained to process data in a reverse direction ([Fig. 7, see “Language2 > Language1”], [Wherein a conversion to a new language represents one direction, and a conversion back into the original language represents a second direction]).
Regarding claim 16, Kim discloses: a system comprising:
one or more processing units ([0064] a processor 320) to:
process, using a duplex neural network (NN) ([0098] bi-directional translation by an electronic device, [0097] the electronic device 200 may use neural machine translation (NMT) [Bi-directional translation tracks to duplex translation, see [0003] of instant application. Further, applying a neural machine translation to a bi-directional translation indicates a bi-directional, i.e. duplex, neural network]) trained to perform translation in a first direction from a first language to a second language ([Fig. 1, translation from Korean into English represents one direction]) and in a second direction from the second language to the first language ([Fig. 1, translation from English back into Korean represents translation in a second direction], [Wherein Korean represents a first language, English represents the second language]),
a first representation of a first speech utterance in a first language to obtain a second representation of a second speech utterance in a second language ([Fig. 1, First Directional Translation 130], [Converting text, i.e. a representation of a speech utterance, into a different language indicates obtaining a second representation of a second speech utterance in a second language, i.e. the translation]),
the second speech utterance comprising a translation of the first speech utterance to the second language ([0049] The electronic device 200-1 may translate the received utterance into the second language and output the translated result to the second speaker 120 [A received utterance tracks to a first speech in view of the first directional translation of Fig. 1]),
wherein the duplex NN comprises a first subnetwork and a second subnetwork ([Fig. 7, 701, 703]).
Kim is not relied upon to disclose:
the first subnetwork comprising a first plurality of NN layers arranged in a first configuration that includes residual connections for performing addition operations, and
the second subnetwork comprising a second plurality of NN layers arranged in a second configuration that includes residual connections for performing subtraction operations,
the second configuration being a mirror image of the first configuration.
Zheng is relied upon to disclose:
the first subnetwork comprising a first plurality of NN layers arranged in a first configuration that includes residual connections for performing addition operations ([Fig. 2, Regular Form, see addition operation performed based on output from SAN layer and FFN layer]), and
the second subnetwork comprising a second plurality of NN layers arranged in a second configuration that includes residual connections for performing subtraction operations ([Fig. 2, Reverse Form, see subtraction operation performed based on output from FFN layer and SAN layer]),
the second configuration being a mirror image of the first configuration ([In view of the operations and ordering of layers of the previously disclosed Regular and Reverse forms of translation, the examiner asserts that the second configuration, i.e. reverse, is a mirror image of the first configuration, i.e. regular. See Fig. 2 in its entirety]).
Kim and Zheng are considered analogous art within reversible machine translation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim to incorporate the teachings of Zheng, because of the novel way to use one model for duplex (Zheng, [Introduction, Par. 3-4]).
Claim(s) 3-6, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Zheng, further in view of Kumar et al. ("Spoken Language Translation using Conformer Model"), disclosed in the "2023 Internation Conference on Intelligent Systems for Communication, IoT and Security", dating from 2/9/23 - 2/11/2023, published to IEEE Xplore on 4/19/23, hereinafter Kumar.
Regarding claim 3, Kim in view of Zheng discloses: the method of claim 1.
Kim in view of Zheng does not disclose:
wherein the first subnetwork comprises one or more neuron blocks comprising two or more of:
a fully-connected layer, a convolutional layer, a self-attention layer, or a normalization layer.
Kumar discloses:
wherein the first subnetwork comprises one or more neuron blocks comprising two or more of:
a fully-connected layer, a convolutional layer ([Fig. 2, “Convolution Module”]), a self-attention layer ([Fig. 2, “Multi-head Self Attention Module”]), or a normalization layer ([Fig. 2, “Layernorm”] [In view of the neural machine translator of Kim, which commonly are comprised of encoder/decoder pairs, i.e. transformers, see 3.2.4 of Kumar disclosing replacement of traditional transformers with conformers, for machine translation, indicating the conformer of Kumar could be substituted for the transformer, i.e. within the machine translator, of Kim without a change in functionality]).
Kim are considered analogous art within language translation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim in view of Zheng to incorporate the teachings of Kumar, because of the novel way to arrange a transformer architecture to convolve in between feed-forward modules, improving translation efficiency over previous transformer transducers (generally consisting of encoder/decoder networks) (Kumar, [Section 2.1, Par. 4]).
Regarding claim 4, Kim in view of Zheng discloses: the method of claim 1.
Kim in view of Zheng does not disclose:
wherein the first subnetwork comprises a neuron block performing operations comprising:
splitting a block input into a first portion and a second portion;
processing, using a first neuron layer of the neuron block, the second portion; and,
aggregating the first portion and the processed second portion to obtain a first block output.
Kumar discloses:
wherein the first subnetwork comprises a neuron block ([Fig. 2, “Conformer Blocks”], [In view of the subnetworks 701, 703 of Kim which perform machine translation, indicating the transformers to perform this translation could be replaced with the conformer of Kumar]) performing operations comprising:
splitting a block input into a first portion and a second portion ([Fig. 2, input into “Feed Forward Module”], [Splitting input into that which enters a feed-forward module and that which does not enter the feed-forward module indicates a splitting of a block input, in view of the conformer blocks, into first and second portions]);
processing, using a first neuron layer of the neuron block ([Fig. 2, “Feed Forward Module], [A feed forward module tracks to a neuron block, see [0010] of instant app]), the second portion ([Fig. 2, output from “Feed Forward Module”], [Sending portions of data into feed forward modules resulting in output indicates processing occurs within the module, wherein the second portion of text represents that sent into the feed forward module]); and,
aggregating the first portion and the processed second portion to obtain a first block output ([Fig. 2, summation block following feed forward module], [Combining input data with that processors through a feed forward module indicates aggregation of first and second processed portions of data to obtain a block, i.e. feed forward module, output]).
Kim, Zheng, and Kumar are considered analogous art within language translation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim in view of Zheng to incorporate the teachings of Kumar, because of the novel way to arrange a transformer architecture to convolve in between feed-forward modules, improving translation efficiency over previous transformer transducer (generally consisting of encoder/decoder networks) (Kumar, [Section 2.1, Par. 4]).
Regarding claim 5, Kim in view of Zheng, further in view of Kumar discloses: the method of claim 4.
Kumar further discloses:
wherein the operations performed by the neuron block further comprise:
processing, using a second neuron layer of the neuron block ([Fig. 2, “Multi-head Self Attention Module”], [A self-attention module tracks to a neuron block, see [0010] of instant app]), a copy of the first block output ([Fig. 2, Input into a multi-head attention layer, wherein that input comes from a first block output, see claim 4 rejection], [Sending a signal through a second module indicates a second processing step, wherein that data received as input could be the audio just determined, i.e. output from the first neuron block, or a copy of the output without a change in functionality to what occurs at the multi-head attention module. Further, Kumar discloses a transformer with copying, see section 3.3.1 Par. 2, indicating the copying with the transformer could be applied to the conformer as the conformer can replace the transformer]).
Zheng further discloses:
aggregating a copy of the second portion and the processed copy of the first block output to obtain a second block output ([Fig. 2, “Regular Form”], [In view of the processed copy of the first block output of Kumar, represented through the signal sent through FFN module to be added to x(2) in Zheng, further in view of the addition block aggregating a second portion, i.e. x(2), and a processed copy of the first block output, i.e. x(2) + (FFN(x(1) + SAN(x(2)))), represented as output from the feed forward module of Kumar, sent out as y(2) indicating a second block output in view of y(1) of Zheng]).
Regarding claim 6, Kim in view of Zheng discloses: the method of claim 1.
Kim in view of Zheng does not disclose:
wherein the duplex NN comprises a conformer NN.
Kumar discloses:
wherein the duplex NN comprises a conformer NN ([Fig. 2, “Conformer Blocks”], [In view of the neural machine translation of Kim (indicating neural network dependence), indicating a conformer NN using the conformer of Kumar with the system of Kim]).
Kim, Zheng, and Kumar are considered analogous art within language translation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim in view of Zheng to incorporate the teachings of Kumar, because of the novel way to arrange a transformer architecture to convolve in between feed-forward modules, improving translation efficiency over previous transformer transducer (generally consisting of encoder/decoder networks) (Kumar, [Section 2.1, Par. 4]).
Regarding claim 18, Kim in view of Zheng discloses: the system of claim 16.
Kim in view of Zheng does not disclose:
wherein the first subnetwork comprises a neuron block configured to:
split a block input into a first portion and a second portion;
process, using a first neuron layer of the neuron block, the second portion; and,
aggregate the first portion and the processed second portion to obtain a first block output.
Kumar discloses:
wherein the first subnetwork comprises a neuron block ([Fig. 2, “Conformer Blocks”], [In view of the subnetworks 701, 703 of Kim which perform machine translation, indicating the transformers to perform this translation could be replaced with the conformer of Kumar]) configured to:
split a block input into a first portion and a second portion ([Fig. 2, input into “Feed Forward Module”], [Splitting input into that which enters a feed-forward module and that which does not enter the feed-forward module indicates a splitting of a block input, in view of the conformer blocks, into first and second portions]);
process, using a first neuron layer of the neuron block ([Fig. 2, “Feed Forward Module], [A feed forward module tracks to a neuron block, see [0010] of instant app]), the second portion ([Fig. 2, output from “Feed Forward Module”], [Sending portions of data into feed forward modules resulting in output indicates processing occurs within the module, wherein the second portion of text represents that sent into the feed forward module]); and,
aggregate the first portion and the processed second portion to obtain a first block output ([Fig. 2, summation block following feed forward module], [Combining input data with that processors through a feed forward module indicates aggregation of first and second processed portions of data to obtain a block, i.e. feed forward module, output]), and
process, using a second neuron layer of the neuron block ([Fig. 2, “Multi-head Self Attention Module”], [A self-attention module tracks to a neuron block, see [0010] of instant app]), a copy of the first block output ([Fig. 2, Input into a multi-head attention layer, wherein that input comes from a first block output, see claim 4 rejection], [Sending a signal through a second module indicates a second processing step, wherein that data received as input could be the audio just determined, i.e. output from the first neuron block, or a copy of the output without a change in functionality to what occurs at the multi-head attention module. Further, Kumar discloses a transformer with copying, see section 3.3.1 Par. 2, indicating the copying with the transformer could be applied to the conformer as the conformer can replace the transformer]).
Kim, Zheng, and Kumar are considered analogous art within language translation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim in view of Zheng to incorporate the teachings of Kumar, because of the novel way to arrange a transformer architecture to convolve in between feed-forward modules, improving translation efficiency over previous transformer transducer (generally consisting of encoder/decoder networks) (Kumar, [Section 2.1, Par. 4]).
Zheng further discloses:
aggregate a copy of the second portion and the processed copy of the first block output to obtain a second block output ([Fig. 2, “Regular Form”], [In view of the processed copy of the first block output of Kumar, represented through the signal sent through FFN module to be added to x(2) in Zheng, further in view of the addition block aggregating a second portion, i.e. x(2), and a processed copy of the first block output, i.e. x(2) + (FFN(x(1) + SAN(x(2)))), represented as output from the feed forward module of Kumar, sent out as y(2) indicating a second block output in view of y(1) of Zheng]).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Zheng, further in view of Finkelstein et al. (US-20220051654-A1), hereinafter Finkelstein.
Regarding claim 7, Kim in view of Zheng discloses: the method of claim 1.
Kim in view of Zheng further discloses:
wherein the first representation comprises a set of embeddings ([0038] The input vector and the context vector may be word embedding vectors corresponding to a certain word [Wherein an input vector represents input text, i.e. a first representation of a first speech]).
Kim in view of Zheng does not disclose:
embeddings obtained, at least in part, by processing, using an embeddings network, a first set of spectrograms for the first speech utterance in the first language.
Finkelstein discloses:
embeddings obtained, at least in part, by processing, using an embeddings network, a first set of spectrograms for the first speech utterance in the first language ([0060] The encoder portion 300a is configured to encode a plurality of fixed-length reference mel-frequency spectrogram frames 211 sampled/extracted from the intermediate synthesized speech representation 202 into the utterance embedding 204 [In view of the signals/speech representations of the speakers of Kim, applying the method of Finkelstein to a non-synthesized audio, as disclosed in Kim, would not change the functionality of Finkelstein’s embedding extraction using spectrogram processing. Further, consider the encoder 300 of Finkelstein located within a TTS system 220, wherein that TTS system can include a neural network architecture, see [0037] of Finkelstein, indicates the embeddings are obtained using an embedding network]).
Kim are considered analogous art within speech analysis/synthetic speech generation, i.e. generating a machine translation indicates that translation would be spoken synthetically. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim in view of Zheng to incorporate the teachings of Finkelstein, because of the novel way to improve the quality of synthesized speech output by conversation agents by considering features not conveyed in textual input, i.e. prosody, tone, intonation, etc. (Finkelstein, [0002]).
Claim(s) 8, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Zheng, further in view of Liu et al. (US-20210216727-A1), hereinafter Liu.
Regarding claim 8, Kim in view of Zheng discloses: the method of claim 1.
Kim in view of Zheng does not disclose:
wherein the duplex NN has been trained using one or more diffusion training techniques. Liu discloses:
wherein the duplex NN has been trained using one or more diffusion training techniques ([0061] Further varying techniques may be used to select features, including but not limited to, diffusion mapping [In view of the machine learning training system of Fig. 4, indicating diffusion training techniques, further in view of the duplex NN of Kim which could be substituted for the machine learning algorithm 440 of Liu without a change in functionality]).
Kim are considered analogous art within speech translation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim in view of Zheng to incorporate the teachings of Liu, because of the novel way to consider translation factors, (Liu, [0004]).
Regarding claim 20, Kim in view of Zheng discloses: the system of claim 16.
Kim further discloses:
wherein the system is comprised in at least one of:
a system for performing deep learning operations ([0073] artificial intelligence system based on deep learning); and,
a system for performing conversational Al operations ([0097] artificial intelligence (AI) system based on deep learning while performing the first directional translation 130, [In view of Kim outputting speech signal, [0050], indicating the artificial intelligence is for conversational operations, i.e. translating within a conversation]).
Kim does not disclose:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system implemented using an edge device;
a system for generating or presenting at least one of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using a robot;
a system implementing one or more large language models (LLMs);
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
Liu discloses:
a system incorporating one or more virtual machines (VMs) ([0079] virtual machines, and services)); and,
a system implemented at least partially using cloud computing resources ([0096] FIG. 6, illustrative cloud computing environment 50).
Kim, Zheng, and Liu are considered analogous art within speech translation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim in view of Zheng to incorporate the teachings of Liu, because of the novel way to consider translation factors, i.e. context information, selected through user feedback, to adjust machine translations, improving the quality of final translations (Liu, [0004]).
Claim(s) 9, 12-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Freitag et al. (US-11295092-B2), hereinafter Freitag, further in view of Zheng.
Regarding claim 9, Kim discloses a method comprising:
obtaining training data that comprises a first training input ([Fig. 5B, 525, “Obtain first input text in first language”]), and a second training input ([Fig. 5B, 545, “Obtain second input text in second language”] [Wherein the inputs of Kim could be used as “training inputs” without a change in functionality to Kim’s system]),
wherein the second training input comprises a second representation of a second speech utterance in a second language ([Fig. 1, Second Directional Translation 140], [Receiving a speaker input, i.e. representation of speech, in a different language than that used in the first directional translation 130 indicates the second speech is in a second language, wherein that speech could be used for training without a change in functionality, in view of the utterances of Kim [0042]]), and,
wherein the second speech utterance comprises a translation of the first speech utterance to the second language ([Fig. 1, “Hi Japson. Long time no see.”], [A translation of a first speech utterance, i.e. in Korean, into English indicates the translation is a second speech text, which can be output in the form of a speech signal, see [0050] of Kim, indicating it is an utterance]).
Kim does not disclose:
obtaining training data that comprises a target output;
wherein the target output comprises a first representation of a first speech utterance in a first language,
wherein the first training input comprises the target output distorted by a first noise, and,
training a neural network (NN) deployed in a first direction to identify, using the first training input and the second training input, at least one of: the target output, or the first noise.
Freitag discloses:
obtaining training data that comprises a target output ([Col. 7, Lines 3-5] Training text 310 in the first language and ground truth text 302 in the first language [A ground truth tracks to a target output, wherein the ground truth text is clearly used in training]);
wherein the target output comprises a first representation of a first speech utterance in a first language ([Col. 6, Lines 65-67] ground truth text 302 is in first language, [Wherein ground truth text, i.e. a first representation, tracks to a target output, in view of the first speech utterance of Kim, wherein both are in a first language]), and,
wherein the first training input comprises the target output distorted by a first noise ([Col. 16, Lines 50-60] the training text is generated by processing the ground truth text using random noise to generate the training text, wherein processing the ground truth text using random noise to generate the training text comprises inserting one or more words into the ground truth text [Processing a ground truth instance using random noise, i.e. adding “noise” words to a ground truth text, indicates distorting a target output, i.e. ground truth instance, with noise, i.e. additional words]); and,
training a neural network (NN) deployed in a first direction ([In view of the bi-directional neural network of Kim, indicating at least a first direction]) to identify, using the first training input and the second training input ([In view of the first training input of Freitag and the second training input of Kim]), at least one of: the target output, or the first noise ([Col. 14, Lines 63-67], [Col. 15, Lines 1-5] The method further includes processing the training text using an automatic post-editing model to generate predicted output, wherein the automatic post-editing model, when trained, is used in correcting one or more translation errors introduced by a neural machine translation model translating text from a source language into the target language. The method further includes determining a difference between the predicted output and the ground truth training text [Determining differences between predicted output, i.e. a first training input, and a ground truth, i.e. second training input, texts wherein that ground truth is a translation (a second representation of a second speech utterance in a second language), indicates the differences represent identified first noise, in view of the noise of Freitag which adds additional words to ground truth texts. Further, removing/correcting the errors introduced (see [Col. 11, Lines 1-10] of Freitag) indicates a corrected output will be a target output, i.e. matching the ground truth]).
Kim and Freitag are considered analogous art within speech translation processing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim to incorporate the teachings of Freitag, because of the novel way to use neural machine translation to predict sequences of words as opposed to individual word substitutions, improving the quality of machine-generated translations (Freitag, [Col. 1, Lines 5-30]).
Kim further discloses:
wherein the NN includes at least a duplex NN comprising a first subnetwork and a second subnetwork ([Fig. 7, 701, 703]).
Kim in view of Freitag is not relied upon to disclose:
the first subnetwork comprising a first plurality of NN layers arranged in a first configuration that includes residual connections for performing addition operations, and
the second subnetwork comprising a second plurality of NN layers arranged in a second configuration that includes residual connections for performing subtraction operations,
the second configuration being a mirror image of the first configuration.
Zheng is relied upon to disclose:
the first subnetwork comprising a first plurality of NN layers arranged in a first configuration that includes residual connections for performing addition operations ([Fig. 2, Regular Form, see addition operation performed based on output from SAN layer and FFN layer]), and
the second subnetwork comprising a second plurality of NN layers arranged in a second configuration that includes residual connections for performing subtraction operations ([Fig. 2, Reverse Form, see subtraction operation performed based on output from FFN layer and SAN layer]),
the second configuration being a mirror image of the first configuration ([In view of the operations and ordering of layers of the previously disclosed Regular and Reverse forms of translation, the examiner asserts that the second configuration, i.e. reverse, is a mirror image of the first configuration, i.e. regular. See Fig. 2 in its entirety]).
Kim are considered analogous art within reversible machine translation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim in view of Freitag to incorporate the teachings of Zheng, because of the novel way to use one model for duplex (Zheng, [Introduction, Par. 3-4]).
Regarding claim 12, Kim in view of Freitag, further in view of Zheng discloses: the method of claim 9.
Freitag further discloses:
wherein the first noise is sampled from a random distribution ([Col. 12, Lines 50-52] using random noise to generate training text).
Regarding claim 13, Kim in view of Freitag, further in view of Zheng discloses: the method of claim 9.
Kim further discloses:
obtaining additional training data that comprises a third training input ([Fig. 5B, 525, “Obtain first input text in first language”]), and a fourth training input ([Fig. 5B, 545, “Obtain second input text in second language”] [Wherein the inputs of Kim could be used as “training inputs” without a change in functionality to Kim’s system. The examiner would like to note that due to the bi-directionality of Kim, the steps for “third” and “fourth” inputs/representation/etc. track to the same values used for the rejection of claim 9, but in a reverse direction, i.e. from a second language to a first language as opposed to from first to second as disclosed in claim 9. Therefore, the first and second values from claim 9 could be applied to the bi-directional system of Kim in a reverse direction as fourth and third values respectively]),
wherein the fourth training input comprises a fourth representation of a fourth speech utterance in a first language ([Fig. 1, Second Directional Translation 140], [Receiving a speaker input, i.e. representation of speech, in a different language than that used in the first directional translation 130 indicates the fourth speech is in a first language, wherein that speech could be used for training without a change in functionality, in view of the utterances of Kim [0042]. The examiner would like to note that due to the bi-directionality of Kim, the steps for “third” and “fourth” inputs/representation/etc. track to the same values used for the rejection of claim 9, but in a reverse direction, i.e. from a second language to a first language as opposed to from first to second as disclosed in claim 9. Therefore, the first and second values from claim 9 could be applied to the bi-directional system of Kim in a reverse direction as fourth and third values respectively]), and,
wherein the fourth speech utterance comprises a translation of the third speech utterance to the second language ([Fig. 1, “Hi Japson. Long time no see.”], [A translation of a third speech utterance, i.e. in Korean, into English indicates the translation is a fourth speech text, which can be output in the form of a speech signal, see [0050] of Kim, indicating it is an utterance. The examiner would like to note that due to the bi-directionality of Kim, the steps for “third” and “fourth” inputs/representation/etc. track to the same values used for the rejection of claim 9, but in a reverse direction, i.e. from a second language to a first language as opposed to from first to second as disclosed in claim 9. Therefore, the first and second values from claim 9 could be applied to the bi-directional system of Kim in a reverse direction as fourth and third values respectively]).
Freitag further discloses:
obtaining training data that comprises an additional target output ([Col. 7, Lines 3-5] Training text 310 in the first language and ground truth text 302 in the first language [A ground truth tracks to a target output, wherein the ground truth text is clearly used in training. The examiner would like to note that due to the bi-directionality of Kim, the steps for “third” and “fourth” inputs/representation/etc. track to the same values used for the rejection of claim 9, but in a reverse direction, i.e. from a second language to a first language as opposed to from first to second as disclosed in claim 9. Therefore, the first and second values from claim 9 could be applied to the bi-directional system of Kim in a reverse direction as fourth and third values respectively]);
wherein the additional target output comprises a third representation of a third speech utterance in a second language ([Col. 6, Lines 65-67] ground truth text 302 is in first language, [Wherein ground truth text, i.e. a fourth representation, tracks to a target output, in view of the first/fourth speech utterance of Kim, wherein both are in a second language. The examiner would like to note that due to the bi-directionality of Kim, the steps for “third” and “fourth” inputs/representation/etc. track to the same values used for the rejection of claim 9, but in a reverse direction, i.e. from a second language to a first language as opposed to from first to second as disclosed in claim 9. Therefore, the first and second values from claim 9 could be applied to the bi-directional system of Kim in a reverse direction as fourth and third values respectively]),
wherein the third training input comprises the additional target output distorted by a second noise ([Col. 16, Lines 50-60] the training text is generated by processing the ground truth text using random noise to generate the training text, wherein processing the ground truth text using random noise to generate the training text comprises inserting one or more words into the ground truth text [Processing a ground truth instance using random noise, i.e. adding “noise” words to a ground truth text, indicates distorting a target output, i.e. ground truth instance, with noise, i.e. additional words. The examiner would like to note that due to the bi-directionality of Kim, the steps for “third” and “fourth” inputs/representation/etc. track to the same values used for the rejection of claim 9, but in a reverse direction, i.e. from a second language to a first language as opposed to from first to second as disclosed in claim 9. Therefore, the first and second values from claim 9 could be applied to the bi-directional system of Kim in a reverse direction as fourth and third values respectively. Further, a second noise could be the same noise as a first noise applied in the reverse direction]); and,
training a neural network (NN) deployed in a second direction ([In view of the bi-directional neural network of Kim, indicating at least a second direction]) to identify, using the third training input and the fourth training input ([In view of the fourth training input of Freitag and the third training input of Kim]), at least one of: the target output, or the first noise ([Col. 14, Lines 63-67], [Col. 15, Lines 1-5] The method further includes processing the training text using an automatic post-editing model to generate predicted output, wherein the automatic post-editing model, when trained, is used in correcting one or more translation errors introduced by a neural machine translation model translating text from a source language into the target language. The method further includes determining a difference between the predicted output and the ground truth training text [Determining differences between predicted output, i.e. a fourth training input, and a ground truth, i.e. third training input, texts wherein that ground truth is a translation (a third representation of a third speech utterance in a second language), indicates the differences represent identified first noise, in view of the noise of Freitag which adds additional words to ground truth texts. Further, removing/correcting the errors introduced (see [Col. 11, Lines 1-10] of Freitag) indicates a corrected output will be a target output, i.e. matching the ground truth. The examiner would like to note that due to the bi-directionality of Kim, the steps for “third” and “fourth” inputs/representation/etc. track to the same values used for the rejection of claim 9, but in a reverse direction, i.e. from a second language to a first language as opposed to from first to second as disclosed in claim 9. Therefore, the first and second values from claim 9 could be applied to the bi-directional system of Kim in a reverse direction as fourth and third values respectively]).
Regarding claim 14, Kim in view of Freitag, further in view of Zheng discloses: the method of claim 9.
Kim further discloses:
obtaining additional training data that comprises a third training input ([Fig. 5B, 525, “Obtain first input text in first language”]);and an additional target output,
wherein the third training input comprises a third representation of a third speech utterance in the first language ([Fig. 1, “First Directional Translation”], [The written speech in Korean displayed in Fig. 1 represents a third representation of a third speech in a first language, wherein the input could be training input, in view of the training inputs of Freitag]).
Freitag further discloses:
obtaining additional training data that comprises an additional target output ([Col. 7, Lines 3-5] Training text 310 in the first language and ground truth text 302 in the first language [A ground truth tracks to a target output, wherein the ground truth text is clearly used in training. The examiner would like to note that due to the bi-directionality of Kim, the steps for “third” and “fourth” inputs/representation/etc. track to the same values used for the rejection of claim 9, but in a reverse direction, i.e. from a second language to a first language as opposed to from first to second as disclosed in claim 9. Therefore, the first and second values from claim 9 could be applied to the bi-directional system of Kim in a reverse direction as fourth and third values respectively]);
wherein the additional target output comprises a fourth representation of a fourth speech utterance in the second language ([Col. 6, Lines 65-67] ground truth text 302 is in first language, [Wherein ground truth text, i.e. a fourth representation, tracks to a target output, in view of the first speech utterance of Kim, wherein both are in a first language. Further, the terms “first” and “fourth” are represent the same data in reverse directions of translation. Therefore, the ground truth text of Freitag could be applied to the bi-directional translation of Kim, resulting in a fourth speech utterance in a second language, i.e. translating back from English to Korean]),
training the NN deployed in the first direction ([In view of the bi-directional neural network of Kim, indicating at least a first direction]) to generate, using the third training input ([In view of the third training input of Kim]), an output that emulates the additional target output ([Col. 14, Lines 63-67], [Col. 15, Lines 1-5] The method further includes processing the training text using an automatic post-editing model to generate predicted output, wherein the automatic post-editing model, when trained, is used in correcting one or more translation errors introduced by a neural machine translation model translating text from a source language into the target language. The method further includes determining a difference between the predicted output and the ground truth training text [Determining differences between predicted output, i.e. a fourth training input, and a ground truth, i.e. third training input, texts wherein that ground truth is a translation (a third representation of a third speech utterance in a second language), indicates the differences represent identified first noise, in view of the noise of Freitag which adds additional words to ground truth texts. Further, removing/correcting the errors introduced (see [Col. 11, Lines 1-10] of Freitag) indicates a corrected output will be a target output, i.e. matching the ground truth (emulating a target output). The examiner would like to note that due to the bi-directionality of Kim, the steps for “third” and “fourth” inputs/representation/etc. track to the same values used for the rejection of claim 9, but in a reverse direction, i.e. from a second language to a first language as opposed to from first to second as disclosed in claim 9. Therefore, the first and second values from claim 9 could be applied to the bi-directional system of Kim in a reverse direction as fourth and third values respectively]).
Kim further discloses:
wherein the fourth speech utterance comprises a translation of the third speech utterance to the second language ([Fig. 1, “Hi Japson. Long time no see.”], [Wherein Kim discloses output of translations into speech signals ([0050]) indicating a fourth speech utterance, in view of the fourth speech utterance based on the target output of Freitag, in a second language, i.e. English]).
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Freitag, further in view of Zheng, further in view of Kumar.
Regarding claim 11, Kim in view of Freitag, further in view of Zheng discloses: the method of claim 9.
Kim in view of Freitag, further in view of Zheng does not disclose:
wherein the duplex NN comprises a conformer NN.
Kumar discloses:
wherein the duplex NN comprises a conformer NN ([Fig. 2, “Conformer Blocks”], [In view of the neural machine translation of Kim (indicating neural network dependence), indicating a conformer NN using the conformer of Kumar with the system of Kim]).
Kim, Freitag, Zheng, and Kumar are considered analogous art within language translation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim in view of Freitag, further in view of Zheng to incorporate the teachings of Kumar, because of the novel way to arrange a transformer architecture to convolve in between feed-forward modules, improving translation efficiency over previous transformer transducer (generally consisting of encoder/decoder networks) (Kumar, [Section 2.1, Par. 4]).
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Freitag, further in view of Zheng, further in view of Finkelstein.
Regarding claim 15, Kim in view of Freitag, further in view of Zheng discloses: the method of claim 9.
Kim further discloses:
wherein the first representation comprises a set of embeddings ([0038] The input vector and the context vector may be word embedding vectors corresponding to a certain word [Wherein an input vector represents input text, i.e. a first representation of a first speech]).
Kim in view of Freitag, further in view of Zheng does not disclose:
embeddings obtained, at least in part, by processing, using an embeddings network, a first set of spectrograms for the first speech utterance in the first language.
Finkelstein discloses:
embeddings obtained, at least in part, by processing, using an embeddings network, a first set of spectrograms for the first speech utterance in the first language ([0060] The encoder portion 300a is configured to encode a plurality of fixed-length reference mel-frequency spectrogram frames 211 sampled/extracted from the intermediate synthesized speech representation 202 into the utterance embedding 204 [In view of the signals/speech representations of the speakers of Kim, applying the method of Finkelstein to a non-synthesized audio, as disclosed in Kim, would not change the functionality of Finkelstein’s embedding extraction using spectrogram processing. Further, consider the encoder 300 of Finkelstein located within a TTS system 220, wherein that TTS system can include a neural network architecture, see [0037] of Finkelstein, indicates the embeddings are obtained using an embedding network]).
Kim, Freitag, Zheng, and Finkelstein are considered analogous art within speech analysis/synthetic speech generation, i.e. generating a machine translation indicates that translation would be spoken synthetically. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim in view of Freitag, further in view of Zheng to incorporate the teachings of Finkelstein, because of the novel way to improve the quality of synthesized speech output by conversation agents by considering features not conveyed in textual input, i.e. prosody, tone, intonation, etc. (Finkelstein, [0002]).
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Zheng, further in view of Freitag.
Regarding claim 19, Kim in view Zheng discloses: the system of claim 16.
Kim further discloses:
wherein to train the NN, the one or more processing units are to:
obtain training data that comprises a first training input ([Fig. 5B, 525, “Obtain first input text in first language”]), and a second training input ([Fig. 5B, 545, “Obtain second input text in second language”] [Wherein the inputs of Kim could be used as “training inputs” without a change in functionality to Kim’s system]),
wherein the second training input comprises a second representation of a second speech utterance in a second language ([Fig. 1, Second Directional Translation 140], [Receiving a speaker input, i.e. representation of speech, in a different language than that used in the first directional translation 130 indicates the second speech is in a second language, wherein that speech could be used for training without a change in functionality, in view of the utterances of Kim [0042]]), and,
wherein the second speech utterance comprises a translation of the first speech utterance to the second language ([Fig. 1, “Hi Japson. Long time no see.”], [A translation of a first speech utterance, i.e. in Korean, into English indicates the translation is a second speech text, which can be output in the form of a speech signal, see [0050] of Kim, indicating it is an utterance]).
Kim in view of Zheng does not disclose:
obtain training data that comprises a target output;
wherein the target output comprises a first representation of a first speech utterance in a first language,
wherein the first training input comprises the target output distorted by a first noise, and,
train a neural network (NN) deployed in a first direction to identify, using the first training input and the second training input, at least one of: the target output, or the first noise.
Freitag discloses:
wherein to train the NN, the one or more processing units are to:
obtain training data that comprises a target output ([Col. 7, Lines 3-5] Training text 310 in the first language and ground truth text 302 in the first language [A ground truth tracks to a target output, wherein the ground truth text is clearly used in training]);
wherein the target output comprises a first representation of a first speech utterance in a first language ([Col. 6, Lines 65-67] ground truth text 302 is in first language, [Wherein ground truth text, i.e. a first representation, tracks to a target output, in view of the first speech utterance of Kim, wherein both are in a first language]), and,
wherein the first training input comprises the target output distorted by a first noise ([Col. 16, Lines 50-60] the training text is generated by processing the ground truth text using random noise to generate the training text, wherein processing the ground truth text using random noise to generate the training text comprises inserting one or more words into the ground truth text [Processing a ground truth instance using random noise, i.e. adding “noise” words to a ground truth text, indicates distorting a target output, i.e. ground truth instance, with noise, i.e. additional words]); and,
train a neural network (NN) deployed in a first direction ([In view of the bi-directional neural network of Kim, indicating at least a first direction]) to identify, using the first training input and the second training input ([In view of the first training input of Freitag and the second training input of Kim]), at least one of: the target output, or the first noise ([Col. 14, Lines 63-67], [Col. 15, Lines 1-5] The method further includes processing the training text using an automatic post-editing model to generate predicted output, wherein the automatic post-editing model, when trained, is used in correcting one or more translation errors introduced by a neural machine translation model translating text from a source language into the target language. The method further includes determining a difference between the predicted output and the ground truth training text [Determining differences between predicted output, i.e. a first training input, and a ground truth, i.e. second training input, texts wherein that ground truth is a translation (a second representation of a second speech utterance in a second language), indicates the differences represent identified first noise, in view of the noise of Freitag which adds additional words to ground truth texts. Further, removing/correcting the errors introduced (see [Col. 11, Lines 1-10] of Freitag) indicates a corrected output will be a target output, i.e. matching the ground truth]).
Kim are considered analogous art within speech translation processing. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim in view of Zheng to incorporate the teachings of Freitag, because of the novel way to use neural machine translation to predict sequences of words as opposed to individual word substitutions, improving the quality of machine-generated translations (Freitag, [Col. 1, Lines 5-30]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THEODORE JOHN WITHEY whose telephone number is (703)756-1754. The examiner can normally be reached Monday - Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571) 272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THEODORE WITHEY/Examiner, Art Unit 2655
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655
Prosecution Insights