DETAILED ACTION
Claims 1-20 of the instant application are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/25/2025 and 04/08/2026 were filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 1 and 11 objected to because of the following informalities
The last limitation of the claims should end with a period instead of a comma (i.e., “…and processing, using a word piece decoder, the second hidden representations to output a textual representation that corresponds to a transcription of the utterance [[,]] .”).
Appropriate correction is required.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-18 of U.S. Patent No. US 12272348 B2 because the claims of the issued patent anticipate the claims of the instant application.
Table 1 shows the overall claim mapping comparing equivalence between claims from instant application and issued patent.
Table 2 show the limitations of the independent claim 1 the instant application when compared with issued patent, respectively, wherein the underlined portions indicate the main differences between instant application and issued patent.
Table 1: Overall claim mapping comparing Instant Application and Issued Patent.
Instant Application
Issued Patent
(US 12272348 B2)
1*, 8-10, 11*, 18-20
1*-2, 10*-11
2, 12
Has/have no equivalent
3-4, 13-14
4, 13
5, 15
Has/have no equivalent
6-7, 16-17
5, 14
Note: * denotes an independent claim
Table 2: Independent claim mapping (comparing each of the limitations)
Instant Application
Issued Patent
(US 12272348 B2)
Independent claim 11:
Independent claim 1:
11. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations comprising:
1. A system comprising
data processing hardware and
memory hardware in communication with the data processing hardware and storing instructions that when executed by the data processing hardware causes the data processing hardware to perform operations for executing a speech conversion system, the speech conversion system comprising:
receiving audio data corresponding to an utterance, the input audio data comprising a sequence of acoustic frames each having a length equal to a first value;
an encoder configured to encode an input spectrogram corresponding to an utterance, the input spectrogram extracted from input speech spoken by a source speaker with atypical speech and comprising a sequence of spectrogram frames having a length equal to a first value,
processing, by an encoder comprising a stack of self-attention blocks, the sequence of acoustic frames to output first hidden representations each having a length equal to a second value greater than the first value;
the encoder comprising:
a first subsampling layer configured to subsample the sequence of spectrogram frames to increase the length of the sequence of spectrogram frames to a second value greater than the first value;
a stack of self-attention blocks comprising an initial set of self-attention blocks and a final set of self-attention blocks, the initial set of self-attention blocks is configured to process the subsampled sequence of spectrogram frames output by the first subsampling layer to output first hidden representations having a length equal to the second value; and
after the first hidden representations are output from the encoder, processing the first hidden representations to provide second hidden representations each having a length equal to a third value greater than the first value; and
a second subsampling layer configured to subsample the first hidden representations output by the initial set of self-attention blocks to increase the length of the first hidden representations to a third value greater than the second value,
wherein the final set of self-attention blocks is configured to process the subsampled first hidden representations output by the initial set of self-attention blocks to output second hidden representations having a length equal to the third value; and
processing, using a word piece decoder, the second hidden representations to output a textual representation that corresponds to a transcription of the utterance,
a spectrogram decoder configured to:
receive, as input, the encoded spectrogram from the encoder; and
generate, as output, an output spectrogram corresponding to a synthesized speech representation of the utterance comprising a synthesized canonical fluent speech representation of the utterance in a voice of the source speaker.
Note: Main differences between instant application and issued patent are underlined.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1 and 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biadsy et al. "Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation." arXiv preprint arXiv:1904.04169 (2019), https://arxiv.org/abs/1904.04169 further in view of Makeev et al. (US 20210185320 A1).
As to independent claim 1, Biadsy et al. teaches:
1. A computer-implemented method that when executed on data processing hardware causes the data processing hardware (see Fig. 1 and ¶ 1 of 2. Model Architecture: “We use an end-to-end sequence-to-sequence model architecture that takes an input source speech and generates/synthesizes target speech as output…” and ¶ 2 of 1. Introduction: “In this paper, we combine attention-based speech recognition and synthesis models to build a direct end-to-end speech-to-speech sequence transducer. This model generates a speech spectrogram as a function of a different input spectrogram, with no intermediate discrete representation.”, footnote 1: https://google.github.io/tacotron/publications/parrotron, and 3.1.1 and 3.2.1. Experiments sections.) to perform operations comprising:
receiving audio data corresponding to an utterance, the input audio data comprising a sequence of acoustic frames each having a length equal to a first value (see ¶ 2.1. Spectrogram encoder: “The base encoder configuration is similar to the encoder in [24], and some variations are evaluated in Section 3.1. From the input speech signal, sampled at 16 kHz, we extract 80-dimensional log-mel spectrogram features over a range of 125-7600 Hz, calculated using a Hann window, 50 ms frame length, 12.5 ms frame shift, and 1024-point Short-Time Fourier Transform (STFT). The input features are passed into a stack of two convolutional layers…”);
processing, by an encoder comprising a stack of self-attention blocks (see ¶ 2.1. Spectrogram encoder citations as in limitations above and further: Fig. 1 and ¶ 2 of 2. Model Architecture: “As shown in Figure 1, the network is composed of an encoder and a decoder with attention, followed by a vocoder to synthesize a time-domain waveform. The encoder converts a sequence of acoustic frames into a hidden feature representation which the decoder consumes to predict a spectrogram. The core architecture is based on recent attention-based end-to-end ASR models [2, 22] and TTS models such as Tacotron [4, 23].”
and ¶ 2 of 2.1 Spectrogram encoder: “…The input features are passed into a stack of two convolutional layers…”),
processing, using a word piece decoder, the second hidden representations to output a textual representation that corresponds to a transcription of the utterance (see ¶ 2.3. Multitask training with an ASR decoder: “Since the goal of this work is to generate only speech sounds and not arbitrary audio, jointly training the encoder network to simultaneously learn a high level representation of the underlying language serves to bias the spectrogram decoder predictions toward a representation of the same underlying speech content. We accomplish this by adding an auxiliary ASR decoder to predict the (grapheme or phoneme) transcript of the output speech, conditioned on the encoder latent representation.”),
However, Biadsy et al. does not explicitly teach, but Makeev et al. does teach:
the sequence of acoustic frames to output first hidden representations each having a length equal to a second value greater than the first value (see ¶ [0022]: “upsampling the representation at the first level of quality using a first upsampler in the encoder device to derive a first preliminary representation of the signal at a second, higher level of quality;”);
after the first hidden representations are output from the encoder (see ¶ [0022] citation as in limitation above), processing the first hidden representations to provide second hidden representations each having a length equal to a third value greater than the first value (see ¶ [0022] citation as in limitation above: “[0027] upsampling the reference representation at the second level of quality using a second upsampler in the encoder device to derive a second preliminary representation of the signal at a third level of quality, higher than the second level of quality;”);
Biadsy et al. and Makeev et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in signal processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Biadsy et al. to incorporate the teachings of Makeev et al. of the sequence of acoustic frames to output first hidden representations each having a length equal to a second value greater than the first value; after the first hidden representations are output from the encoder, processing the first hidden representations to provide second hidden representations each having a length equal to a third value greater than the first value which provides the benefit of obtaining a higher level of quality of the signal ([0022 and 0027] of Makeev et al.).
As to independent claim 11, Makeev et al. further teaches:
11. A system (see [0115]: “… the apparatus 600 comprises one or more processors 601 configured to process information and/or instructions.…”) comprising:
data processing hardware (see [0115]: “one or more processors 601); and
memory hardware in communication with the data processing hardware (see ¶ [0116]: “…the apparatus 600 comprises computer-useable volatile memory 603 configured to store information and/or instructions for the one or more processors 601…”) and storing instructions that when executed on the data processing hardware causes the data processing hardware (see ¶ [0116] citations as in limitation above.) to perform operations comprising:
[limitations as in claim 1 and taught by Biadsy et al. in combination with Makeev et al., above.]
Biadsy et al. and Makeev et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in signal processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Biadsy et al. to incorporate the teachings of Makeev et al. of a system comprising: data processing hardware; and memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations which provides the benefit of obtaining a higher level of quality of the signal ([0022 and 0027] of Makeev et al.).
Claims 2 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biadsy et al. "Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation." arXiv preprint arXiv:1904.04169 (2019), https://arxiv.org/abs/1904.04169 further in view of Makeev et al. (US 20210185320 A1) as applied to claims 1 and 11 above, and further in view of Wang et al. (US 11557292 B1).
Regarding claims 2 and 12, Biadsy et al. in combination with Makeev et al. teaches the limitations as in claims 1 and 11, above.
However, Biadsy et al. in combination with Makeev et al. does not explicitly teach, but Wang et al. does teach:
2 and 12. The computer-implemented method/system of claims 1 and 11, wherein the first value comprises 10 milliseconds (see ¶ col. 6, lines 4-18: “(27) The neural network(s) described herein may receive, as input, a “window” of audio data; this window may include a number of units of audio data, such as frames of audio data, which represent a fixed length of time of received audio. For example, a frame of audio data may represent 10 milliseconds of received audio. The frame of audio data may be a time-domain representation of the magnitude and phase of the received audio. The frame of audio data may instead be or include a representation of processed received audio, which may be, for example, a frequency-domain representation of the received audio, a spectrogram (such as a Mel-cestrum), and/or an LFBE of the received audio. The window may include a plurality of frames, such as 80 frames, and may thus correspond to a certain duration of received audio, such as 800 milliseconds.”).
Biadsy et al., Makeev et al. and Wang et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in encoding/decoding speech/audio signals. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Biadsy et al. in combination with Makeev et al. to incorporate the teachings of Wang et al. of wherein the first value comprises 10 milliseconds which provides the benefit of possess more processing resources compared to the user device (e.g., a larger, faster neural network) that may more accurately determine if the audio data includes the representation of the speech command. (¶ col. 3, lines 15-20 of Wang et al.)
Claims 3-6 and 13-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biadsy et al. "Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation." arXiv preprint arXiv:1904.04169 (2019), https://arxiv.org/abs/1904.04169 further in view of Makeev et al. (US 20210185320 A1) as applied to claims 1 and 11 above, and further in view of Gulati et al. "Conformer: Convolution-augmented transformer for speech recognition." arXiv preprint arXiv:2005.08100 (2020). (https://arxiv.org/pdf/2005.08100.pdf)).
Regarding claims 3 and 13, Biadsy et al. in combination with Makeev et al. teaches the limitations as in claims 1 and 11, above.
However, Biadsy et al. in combination with Makeev et al. does not explicitly teach, but Gulati et al. does teach:
3 and 13. The computer-implemented method/system of claims 1 and 11, wherein the stack of self-attention blocks of the encoder comprises a stack of Conformer blocks (see Fig. 1 (Conformer encoder model architecture. Conformer comprises of two macaron-like feed-forward layers with half-step residual connections sandwiching the multi-headed self-attention and convolution modules. This is followed by a post layernorm. “Conformer blocks x N”; “Multi-head self attention module” in each of the conformer blocks.) and ¶ 2.4. Conformer Block: “Our proposed Conformer block contains two Feed Forward modules sandwiching the Multi-Headed Self-Attention module and the Convolution module, as shown in Figure 1.”).
Biadsy et al., Makeev et al. and Gulati et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in encoding/decoding speech/audio signals. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Biadsy et al. in combination with Makeev et al. to incorporate the teachings of Gulati et al. of wherein the stack of self-attention blocks of the encoder comprises a stack of Conformer blocks which provides the benefit of achieving state-of-the-art results on LibriSpeech, outperforming previous best published Transformer Transducer by 15% relative improvement on the test other dataset with an external language mode (¶ 5 of 1. Introduction of Gulati et al.)
Regarding claims 4 and 14, Biadsy et al. in combination with Makeev et al. teaches the limitations as in claims 3 and 13, above.
Gulati et al. further teaches:
4 and 14. The computer-implemented method/system of claims 3 and 13, wherein each Conformer block comprises a multi-headed self-attention mechanism (see Fig. 1 and ¶ 2.4. Conformer Block citations as in claim 3 and 3, above.).
Biadsy et al., Makeev et al. and Gulati et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in encoding/decoding speech/audio signals. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Biadsy et al. in combination with Makeev et al. to incorporate the teachings of Gulati et al. of wherein each Conformer block comprises a multi-headed self-attention mechanism which provides the benefit of achieving state-of-the-art results on LibriSpeech, outperforming previous best published Transformer Transducer by 15% relative improvement on the test other dataset with an external language mode (¶ 5 of 1. Introduction of Gulati et al.)
Regarding claims 5 and 15, Biadsy et al. in combination with Makeev et al. teaches the limitations as in claims 3 and 13, above.
Gulati et al. further teaches:
5 and 15. The computer-implemented method/system of claims 3 and 13, wherein each Conformer block (see Fig. 1 and ¶ 2.4. Conformer Block citations as in claim 3 and 3, above.) comprises:
a first half feed-forward layer (see Fig. 1 (“two macaron-like feed-forward layers with half step residual connections sandwiching the multi-headed self attention and convolution modules.”);
a second half feed-forward layer (see Fig. 1 (“two macaron-like feed-forward layers with half step residual connections sandwiching the multi-headed self attention and convolution modules.”);
a multi-head self-attention block (see Fig. 1 (“two macaron-like feed-forward layers with half step residual connections sandwiching the multi-headed self attention and convolution modules.”);
and a convolution layer disposed between the first and second half feed-forward layers (see Fig. 1 (“two macaron-like feed-forward layers with half step residual connections sandwiching the multi-headed self attention and convolution modules.”).
Biadsy et al., Makeev et al. and Gulati et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in encoding/decoding speech/audio signals. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Biadsy et al. in combination with Makeev et al. to incorporate the teachings of Gulati et al. of wherein each Conformer block comprises: a first half feed-forward layer; a second half feed-forward layer; a multi-head self-attention block; and a convolution layer disposed between the first and second half feed-forward layers which provides the benefit of achieving state-of-the-art results on LibriSpeech, outperforming previous best published Transformer Transducer by 15% relative improvement on the test other dataset with an external language mode (¶ 5 of 1. Introduction of Gulati et al.)
Regarding claim 6 and 16, Biadsy et al. in combination with Makeev et al. teaches the limitations as in claims 3 and 13, above.
Gulati et al. further teaches:
6 and 16. The computer-implemented method/system of claims 3 and 13, wherein the encoder further comprises a subsampling layer disposed before the stack of Conformer blocks (see Fig. 1 (“convolution subsampling”) and ¶ 1 of 2. Conformer Encoder: “Our audio encoder first processes the input with a convolution subsampling layer and then with a number of conformer blocks, as illustrated in Figure 1. The distinctive feature of our model is the use of Conformer blocks in the place of Transformer blocks as in [7, 19].”).
Biadsy et al., Makeev et al. and Gulati et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in encoding/decoding speech/audio signals. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Biadsy et al. in combination with Makeev et al. to incorporate the teachings of Gulati et al. of wherein the encoder further comprises a subsampling layer disposed before the stack of Conformer blocks which provides the benefit of achieving state-of-the-art results on LibriSpeech, outperforming previous best published Transformer Transducer by 15% relative improvement on the test other dataset with an external language mode (¶ 5 of 1. Introduction of Gulati et al.)
Claims 7 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biadsy et al. "Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation." arXiv preprint arXiv:1904.04169 (2019), https://arxiv.org/abs/1904.04169 further in view of Makeev et al. (US 20210185320 A1) and Gulati et al. "Conformer: Convolution-augmented transformer for speech recognition." arXiv preprint arXiv:2005.08100 (2020). (https://arxiv.org/pdf/2005.08100.pdf))as applied to claims 3 and 13 above, and further in view of Mathew et al. (US 20180181857 A1).
Regarding claims 7 and 17, Biadsy et al. in combination with Makeev et al. teaches the limitations as in claims 3 and 13, above.
Gulati et al. further teaches:
7 and 17. The computer-implemented method/system of claims 6 and 16, wherein the subsampling layer comprises Convolution Neural Network (CNN) layers (see Fig. 1 (“convolution subsampling”) and ¶ 1 of 2. Conformer Encoder: “Our audio encoder first processes the input with a convolution subsampling layer and then with a number of conformer blocks, as illustrated in Figure 1. The distinctive feature of our model is the use of Conformer blocks in the place of Transformer blocks as in [7, 19].”).
Biadsy et al., Makeev et al. and Gulati et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in encoding/decoding speech/audio signals. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Biadsy et al. in combination with Makeev et al. to incorporate the teachings of Gulati et al. of wherein the subsampling layer comprises Convolution Neural Network (CNN) layers which provides the benefit of achieving state-of-the-art results on LibriSpeech, outperforming previous best published Transformer Transducer by 15% relative improvement on the test other dataset with an external language mode (¶ 5 of 1. Introduction of Gulati et al.)
However, Biadsy et al. in combination with Makeev et al. and Gulati et al. do not explicitly teach, but Mathew et al. does teach:
Convolution Neural Network (CNN) layers followed by pooling in time to reduce a number of the sequence of acoustic frames being processed by an initial Conformer block in the stack of Conformer blocks (see ¶ [0004 and 0028-0029]: “[0004] Embodiments of the present disclosure relate to reducing the complexity of convolutions in convolutional neural networks (CNNs). In one aspect, a method for convolution in a convolutional neural network (CNN) is provided that includes accessing a coefficient value of a filter corresponding to an input feature map of a convolution layer of the CNN, and performing a block multiply accumulation operation on a block of data elements of the input feature map, the block of data elements corresponding to the coefficient value, wherein, for each data element of the block of data elements, a value of the data element is multiplied by the coefficient value and a result of the multiply is added to a corresponding data element in a corresponding output block of data elements comprised in an output feature map.
[0028] Each subsampling layer, which may also be referred to as a pooling layer, reduces the dimensionality of each feature map while retaining the most important information. The subsampling can be of different types, e.g., max pooling, average pooling, or striding.
[0029] FIG. 1 is an example of the process flow of a simple convolutional neural network (CNN) 100 that classifies an input image into one of four classes. More specifically, the CNN 100 includes two convolution layers outputting feature maps 104, 108. Each convolution layer is followed by a respective subsampling layer outputting feature maps 106, 110. The last pooling layer is followed by two fully connected layers outputting feature maps 110, 112 that classify the image 102 based on the features extracted by the convolution and pooling layers. In the example of FIG. 1, two sets of alternating convolution and pooling layers are shown.”)
Biadsy et al., Makeev et al. and Mathew et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in encoding/decoding signals. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Biadsy et al. in combination with Makeev et al. to incorporate the teachings of Mathew et al. of Convolution Neural Network (CNN) layers followed by pooling in time to reduce a number of the sequence of acoustic frames being processed by an initial Conformer block in the stack of Conformer blocks which provides the benefit of maintaining accuracy (¶ [0050] of Mathew et al.)
Claims 8 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biadsy et al. "Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation." arXiv preprint arXiv:1904.04169 (2019), https://arxiv.org/abs/1904.04169 further in view of Makeev et al. (US20210185320 A1) as applied to claim[ (s) numbers from 102? ] above, and further in view of Saito et al (US 20230130902 A1).
Regarding claims 8 and 18, Biadsy et al. in combination with Makeev et al. teaches the limitations as in claims 1 and 11, above.
However, Biadsy et al. in combination with Makeev et al. does not explicitly teach, but Saito et al. does teach:
8 and 18. The computer-implemented method/system of claims 1 and 11, wherein the stack of self-attention blocks of the encoder comprises a stack of Transformer blocks (see ¶ [0089]: “Transformer encoder blocks of the joint encoding unit 125 encode the embedded source and reference texts as a stack of Transformer blocks. Each Transformer encoder block has the same architecture as that of Reference 1. The Transformer encoder block includes two subcomponents, a multi-head self-attention network and a fully connected feed forward network. Each network applies residual connections. In this model, both the source text and the reference text are individually encoded in the encoder stack.”).
Biadsy et al., Makeev et al. and Saito et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in encoding/decoding speech/audio signals. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Biadsy et al. in combination with Makeev et al. to incorporate the teachings of Saito et al. of wherein the stack of self-attention blocks of the encoder comprises a stack of Transformer blocks which provides the benefit of improving the accuracy of sentence generation (¶ abstract of Saito et al.)
Claims 9-10 and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biadsy et al. "Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation." arXiv preprint arXiv:1904.04169 (2019), https://arxiv.org/abs/1904.04169 further in view of Makeev et al. (US20210185320 A1) as applied to claim[ (s) numbers from 102? ] above, and further in view of Dinh et al. ("Improving Speech Intelligibility Through Speaker Dependent and Independent Spectral Style Conversion." INTERSPEECH. 2020. http://www.interspeech2020.org/uploadfile/pdf/Mon-3-3-6.pdf).
Regarding claims 9 and 19, Biadsy et al. in combination with Makeev et al. teaches the limitations as in claims 1 and 11, above.
However, Biadsy et al. in combination with Makeev et al. does not explicitly teach, but Dinh et al. does teach:
9 and 19. The computer-implemented method/system of claims 1 and 11, wherein audio data corresponding to the utterance is extracted from input speech spoken by a speaker associated with atypical speech (see Fig. 5 (Spectrogram of habitual speech (HAB), DNN mapping (DNN), cGAN mapping (GAN), and clear speech (CLR), from speaker PDF7, reading “Four hours of steady work faced us”. Note the difference in formants between 2–4 kHz from the 50th–100th frame between the DNN and cGAN methods.) and
¶ 5. Experiment: Many-to-Many Mapping: “The disadvantage of the previous many-to-one mapping is that speaker characteristics cannot be preserved. Thus, we investigate the most realistic scenario of a many-to-many mapping that aims to learn solely the style differences, while preserving the linguistic message and speaker characteristics. This task is the hardest among our three experiments because not all speakers’ spectral changes across styles have been shown to benefit speech intelligibility [22]. We used the same data as the previous experiments. We aligned each HAB utterance to its parallel CLR utterance of the same speaker using DTW on 32nd-order log filter-bank features. Then we trained all one-to-one mappings from HAB VAE-12 to CLR VAE-12 simultaneously.”
¶ 5.1. Objective Evaluation: “The average LSD between the cGAN mapping and CLR spectrograms for the three test speakers, with the LSD between HAB and CLR spectrograms in parentheses, are: 16.36 (17.0) dB for CSM7, 16.66 (17.53) dB for PDF7, and 16.42 (18.06) dB for PDM6, confirming that the mapped speech is closer to CLR speech than the input HAB speech.”
¶ 5.2. Subjective Evaluation: “We designed a test consisting of 25 sentences × 3 speakers (CSM7, PDF7, PDM6) × 4 conditions (vocoded HAB, GAN, hybrid, vocoded CLR) = 300 unique trials. We conducted the experiment similarly to the previous ones, except the number of listeners was 24. Table 2 shows average keyword recall accuracy. The cGAN resulted in improvements for two speakers: CSM7 and PDF7. However, the results were not statistically significant using a two-tailed t-test (p > 0.05).”);
Biadsy et al., Makeev et al. and Dinh et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in encoding/decoding speech/audio signals. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Biadsy et al. in combination with Makeev et al. to incorporate the teachings of Dinh et al. of wherein audio data corresponding to the utterance is extracted from input speech spoken by a speaker associated with atypical speech which provides the benefit of preserving the linguistic message and speaker characteristics (¶ ¶ 5. Experiment: Many-to-Many Mapping of Dinh et al.)
Regarding claims 10 and 20, Biadsy et al. in combination with Makeev et al. and Dinh et al. teaches the limitations as in claims 9 and 19, above.
Biadsy et al. further teaches:
10 and 20. The computer-implemented method/system of claims 9 and 19,
wherein the operations further comprise
processing, using a spectrogram decoder, the second hidden representations to generate an output spectrogram (see ¶ 1 of 2. Spectrogram decoder: “The decoder targets are 1025-dim STFT magnitudes, computed with the same framing as the input features, and a 2048-point FFT. We use the decoder network described in [4], consisting of an autoregressive RNN to predict the output spectrogram from the encoded input sequence one frame at a time. The prediction from the previous decoder time step is first passed through a small pre-net containing 2 fully connected layers of 256 ReLU units, which was found to help to learn attention [4, 23]. The pre-net output and attention context vector are concatenated and passed through a stack of 2 unidirectional LSTM layers with 1024 units. The concatenation of the LSTM output and the attention context vector is then projected through a linear transform to produce a prediction of the target spectrogram frame.”)
Dinh et al. further teaches:
spectrogram corresponding to a synthesized speech representation of the utterance comprising a synthesized canonical fluent speech representation of the utterance in a voice of the speaker that spoke the input speech (see Fig. 5, ¶ 5. Experiment: Many-to-Many Mapping, ¶ 5.1. Objective Evaluation, and ¶ 5.2. Subjective Evaluation citations in the limitation above:
¶ 5. Experiment: Many-to-Many Mapping: “The disadvantage of the previous many-to-one mapping is that speaker characteristics cannot be preserved. Thus, we investigate the most realistic scenario of a many-to-many mapping that aims to learn solely the style differences, while preserving the linguistic message and speaker characteristics. This task is the hardest among our three experiments because not all speakers’ spectral changes across styles have been shown to benefit speech intelligibility [22]. We used the same data as the previous experiments. We aligned each HAB utterance to its parallel CLR utterance of the same speaker using DTW on 32nd-order log filter-bank features. Then we trained all one-to-one mappings from HAB VAE-12 to CLR VAE-12 simultaneously.”
¶ 5.1. Objective Evaluation: “The average LSD between the cGAN mapping and CLR spectrograms for the three test speakers, with the LSD between HAB and CLR spectrograms in parentheses, are: 16.36 (17.0) dB for CSM7, 16.66 (17.53) dB for PDF7, and 16.42 (18.06) dB for PDM6, confirming that the mapped speech is closer to CLR speech than the input HAB speech.”
¶ 5.2. Subjective Evaluation: “We designed a test consisting of 25 sentences × 3 speakers (CSM7, PDF7, PDM6) × 4 conditions (vocoded HAB, GAN, hybrid, vocoded CLR) = 300 unique trials. We conducted the experiment similarly to the previous ones, except the number of listeners was 24. Table 2 shows average keyword recall accuracy. The cGAN resulted in improvements for two speakers: CSM7 and PDF7. However, the results were not statistically significant using a two-tailed t-test (p > 0.05).”
Here, mapping many-to-many speakers is discussed to preserve linguistic and speaker characteristics.).
Biadsy et al., Makeev et al. and Dinh et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in encoding/decoding speech/audio signals. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Biadsy et al. in combination with Makeev et al. to incorporate the teachings of Dinh et al. of spectrogram corresponding to a synthesized speech representation of the utterance comprising a synthesized canonical fluent speech representation of the utterance in a voice of the speaker that spoke the input speech which provides the benefit of preserving the linguistic message and speaker characteristics (¶ ¶ 5. Experiment: Many-to-Many Mapping of Dinh et al.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Keisha Y Castillo-Torres whose telephone number is (571)272-3975. The examiner can normally be reached Monday - Friday, 9:00 am - 4:00 pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at (571)272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Keisha Y. Castillo-Torres
Examiner
Art Unit 2659
/Keisha Y. Castillo-Torres/Examiner, Art Unit 2659
