DETAILED ACTION
Continued Examination Under 37 CFR 1.114
1. A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 6/30/2026 has been entered.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
3. Claim 1 has been amended.
Response to Arguments
4. Applicant’s arguments filed have been fully considered but are moot based on the new grounds of rejection.
Claim Rejections - 35 USC § 102
5. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
6. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
7. Claims 1-6, 20, 24 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Polyak, Adam, et al. "High fidelity speech regeneration with application to speech enhancement." ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.
Regarding claim 1 Polyak et al teaches An audio processing apparatus comprising at least one processor and a memory storing instructions that are operable, when executed by the at least one processor (4.3: We have efficiently implemented a server-based module using PyTorch JIT that is able to fetch speech audio and regenerate it), to cause the audio processing apparatus to:
receive audio signals captured by one or more audio capture devices, wherein the audio signals comprise one or more first speech signals associated with a first speaking entity (Section 3 raw input speech; 3.1: raw noisy speech signal; fig 1);
divide the audio signals into one or more audio segments (3.1 sequence of samples; fig 1);
input a first audio segment of the one or more audio segments to a first audio transformation model to generate a first voice vector representation, wherein the first voice vector representation comprises one or more characteristics related to a first speaking voice associated with the first speaking entity (Abstract; introduction : wav-to-wav generative model; Section 3: encoder-decoder network; 3.1 Our encoding extracts several different representations in order to capture content, prosody, and identity features separately.; the identity representation is obtained in the form of d-vectors; fig 1);
input the one or more audio segments to the first audio transformation model to generate a first speech vector representation, wherein the first speech vector representation comprises one or more words spoken by the first speaking entity and one or more respective contextual attributes associated with the one or more words (Abstract; introduction : wav-to-wav generative model; 3.1 Our encoding extracts several different representations in order to capture content, prosody, and identity features separately; Specifically, given an input signal ˜x, the content representation is extracted using a pre-trained ASR network, Easr.; the prosody representation includes both the fundamental frequency and a loudness feature.; fig 1);
compile the first voice vector representation and the first speech vector representation into audio processing data (Section 3: synthesizes the final output;
3.2: The decoder then receives as input the concatenated and upsampled conditioning signal, E(˜x), sampled at 250Hz. G is conditioned on the concatenation of the noise vector, z, and the speaker identity, Eid(˜ x), while the input to the model is Easr(˜x) concatenated with F0(˜x) and Floud(˜x). Finally, the proposed model outputs a raw audio signal sampled at 24kHz.; fig 1 features are…concatenated); and
output the audio processing data.
(Section 3: synthesizes the final output
Abstract: speech enhancement; To enhance speech beyond the limitations of the original signal, we take a regeneration approach, in which we recreate the speech from its essence, including the semi-recognized speech, prosody features, and identity. We propose a wav-to-wav generative model for speech that can generate 24khz speech in a real time manner and which utilizes a compact speech representation, composed of ASR and identity features, to achieve a higher level of intelligibility.;
1 Introduction: We utilize these advances and introduce a speech regeneration pipeline, in which speech is encoded at the semantic level through ASR, and a speech synthesizer is used to produce an output that is not only cleaner than the input, but also has better perceptual metrics.
fig 1:
PNG
media_image1.png
429
710
media_image1.png
Greyscale
Regarding claim 2 Polyak teaches The audio processing apparatus of claim 1, wherein the first audio transformation model comprises a neural network (Abstract; introduction : wav-to-wav generative model; Section 3: encoder-decoder network);.
Regarding claim 3 Polyak teaches The audio processing apparatus of claim 1, wherein the instructions are operable, when executed by the at least one processor, to further cause the audio processing apparatus to:
prior to inputting the one or more audio segments to the first audio transformation model, extract the one or more words spoken by the first speaking entity from the one or more audio segments and convert the one or more words into a computer-readable format (Section 3 : First, the raw input speech is passed through a background removal method that masks out non-vocal audio.; fig 1).
Regarding claim 4 Polyak teaches The audio processing apparatus of claim 1, wherein the first audio transformation model is configured to generate the first speech vector representation based on one or more speech primitives associated with the one or more words spoken by the first speaking entity and the one or more respective contextual attributes associated with the one or more words (Abstract; introduction : wav-to-wav generative model; 3.1 Our encoding extracts several different representations in order to capture content, prosody, and identity features separately; Specifically, given an input signal ˜x, the content representation is extracted using a pre-trained ASR network, Easr.; the prosody representation includes both the fundamental frequency and a loudness feature.; fig 1).
Regarding claim 5 Polyak teaches The audio processing apparatus of claim 1, wherein the one or more respective contextual attributes comprise at least one of one or more acoustic features, one or more emotive qualities, or one or more speech delivery characteristics associated with the one or more words spoken by the first speaking entity (3.1 Our encoding extracts several different representations in order to capture content, prosody, and identity features separately; Specifically, given an input signal ˜x, the content representation is extracted using a pre-trained ASR network, Easr.; the prosody representation includes both the fundamental frequency and a loudness feature.; fig 1).
Regarding claim 6 Polyak teaches The audio processing apparatus of claim 5, wherein the one or more acoustic features comprise at least one of pitch, articulation, volume, or intensity (Abstract; introduction : wav-to-wav generative model; 3.1 Our encoding extracts several different representations in order to capture content, prosody, and identity features separately; Specifically, given an input signal ˜x, the content representation is extracted using a pre-trained ASR network, Easr.; the prosody representation includes both the fundamental frequency and a loudness feature.; fig 1).
Regarding claim 20 Polyak teaches The audio processing apparatus of claim 1, wherein the instructions are further operable to cause the audio processing apparatus to:
receive a second voice vector representation (section 3.1; fig 1 – related to speaking voice; identity);
receive a second speech vector representation (section 3.1 fig 1 – words and context; recognized speech and prosody);
input the second voice vector representation and the second speech vector representation into a second audio transformation model (section 3.1 fig 1 – using model for enhancement/regeneration);
generate, based on the second voice vector representation, the second speech vector representation, and model output generated by the second audio transformation model, one or more second speech signals (section 3.1, 3.2; fig 1 – regenerated speech output); and
output the one or more second speech signals in a second speaking voice associated with a second speaking entity related to the second voice vector representation. (section 3.1;3.2; fig 1)
Rejected for similar rationale and reasoning as claim 1 – where the voice information will be processed based on the specific speaker, and can be performed for multiple speakers).
Regarding claim 24 Polyak teaches The audio processing apparatus of claim 1, wherein the first audio transformation model is configured to (i) determine one or more frequency patterns associated with a speaking voice of the first speaking entity, (ii) extract one or more speech primitives comprising the one or more words, and (iii) determine the one or more respective contextual attributes associated with the one or more speech primitives (3.1: the prosody representation includes both the fundamental frequency and a loudness feature.; fig 1)
Claim Rejections - 35 USC § 103
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. Claims 7-8 are rejected under 35 U.S.C. 103 as being unpatentable over Polyak in view of Balasubramaniam et al (10,896,428).
Regarding claim 7 Polyak does not specifically teach where Balasubramaniam teaches The audio processing apparatus of claim 5, wherein the one or more speech delivery characteristics comprise at least one of pause duration, pace, or speech rate (Col 13 l 33-34: speech characteristics, such as speaking rate).
Balasubramaniam et al (10,896,428) teaches
Fig 3; Col 10 l. 15-30: ASR; Col 12 l. 45-62: speech analyzer may generate a feature vector for the frame using these statistics, such as a feature vector that includes a separate dimension for each of the computed statistics;
Col 13 l. 17-47: At block 316, the speech analyzer 116 or some other module or component of the dynamic contact management system 100 can generate speech feature data using the feature vectors generated above. In some embodiments, the speech analyzer 116 can analyze the feature vectors with respect to a model trained to classify the feature vectors into particular classifications, such as classifications associated with particular emotions. Illustratively, the model may be a k-means clustering trained model, a support vector machine (“SVM”) trained model, or some other classification model. In some embodiments, the speech analyzer 116 may use different models in different contexts. In this way, the speech analyzer 116 may accurately determine speech features (e.g., emotion classifications) across different contexts, when a general model may not accurately determine speech features in specific contexts. The different models may be designed for analyzing speech that has different speech characteristics, such as speaking rate, prosody, pitch, volume, language, dialect, accent, some combination thereof, etc. For example, the speech analyzer 116 may access data regarding the gender of a speaker (e.g., data associated with a profile of a customer or agent, modelling applied to predict the gender of a speaker, etc.). The speech analyzer may use a first model if the person whose voice is represented by the audio data is a male, and a second model if the person whose voice is represented by the audio data is a female. As another example, the speech analyzer 116 may access data regarding a geographic location of a speaker. The speech analyzer 116 may select a model tailored to speech characteristics of speakers (e.g., accent, prosody, etc.) in the geographic location.;
Col 14 l. 47-50).
It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Balasubramaniam to obtain additional speech components for improved speech enhancement and regeneration.
Polyak already teaches obtaining audio and speech components and processing using models to enhance the speech. In a similar field of endeavor, Balasubramaniam is directed to a system that analyzes aspects of speech from a customer contact and generates dynamic output based on the analysis (Col 1 l. 54-56). One could thus look to Balasubramaniam to further incorporate additional speech components for more comprehensive speech enhancement.
Regarding claim 8 Balasubramaniam teaches The audio processing apparatus of claim 5, wherein the one or more emotive qualities are characterized by one or more respective emotional dimensions, wherein the one or more respective emotional dimensions comprise at least one of valence, activation, or dominance (col 13 l. 48-50: the model may differentiate between a set of emotions, such as anger, boredom, disgust, anxiety, happiness, sadness, and neutral.).
Rejected for similar rationale and reasoning as claim 7
8. Claims 9-10, 13, 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Polyak in view of Hijazi et al (2020/0075033).
Regarding claim 9 Polyak does not specifically teach where Hijazi teaches The audio processing apparatus of claim 1, wherein the first speech vector representation and the first voice vector representation are configured for regeneration of the one or more first speech signals associated with the first speaking entity by a second audio processing apparatus
([0090] The process 500 includes storing, playing, or transmitting 560 an enhanced audio signal based on the enhanced audio segment. For example, determining the enhanced audio signal may include blending the sequence of respective enhanced audio segments, including combining (e.g., using weighted averaging) corresponding samples from multiple segments in overlapping portions of the enhanced audio segments. In some implementations, where the audio data is extracted from AV data, the enhanced audio signal is merged (e.g., synchronized and encoded) with a visual track from the AV data and stored or transmitted 560 as part of enhanced AV data;
0092: the enhanced audio signal may be transmitted 560 to an external device (e.g., the media server 410 or a personal computing device) for display or storage.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Hijazi for an improved system allowing for regeneration at a second apparatus. Polyak already addresses speech regeneration to overcome poor speech quality, and one could look to Hijazi to further allow for the transmission and receipt of speech signals (by a second apparatus), allowing for audio communication to take place while ensuring that a higher level of intelligibility is achieved.
Regarding claim 10 Hijazi teaches The audio processing apparatus of claim 1, wherein the instructions are further operable to cause the audio processing apparatus to:
transmit, in near real-time, the audio signals captured by the one or more audio capture devices to a second audio processing apparatus for outputting of the audio signals by the second audio processing apparatus ([0090] The process 500 includes storing, playing, or transmitting 560 an enhanced audio signal based on the enhanced audio segment.; 0092: the enhanced audio signal may be transmitted 560 to an external device (e.g., the media server 410 or a personal computing device) for display or storage).
Rejected for similar rationale and reasoning as claim 9
Regarding claim 13 Polyak does not specifically teach where Hijazi teaches The audio processing apparatus of claim 1, wherein the one or more audio segments correspond to a predetermined duration of time (0035: windows of samples of an appropriate duration; window duration).
It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Hijazi and specific segment duration to still obtain the audio segments (as required by Polyak), presenting a reasonable expectation of success in still processing the audio to obtain the necessary characteristics for speech enhancement and regeneration.
Regarding claim 15 Polyak does not specifically teach where Hijazi teaches The audio processing apparatus of claim 1, wherein the instructions are further operable to cause the audio processing apparatus to:
compress at least one of the first speech vector representation or the first voice vector representation (31: encoded, compression algorithm; 0045; 131).
It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Hijazi and compression for an improved system, allowing for lower storage or processing requirements, contributing to improved efficiency in the speech regeneration pipeline.
Regarding claim 16 Polyak does not specifically teach where Hijazi teaches The audio processing apparatus of claim 1, wherein the instructions are further operable to cause the audio processing apparatus to:
capture, via one or more video or image capture devices, one or more portions of image data, wherein the one or more portions of image data are associated with the first speaking entity (0033: extract an audio track from a video; 0053: the system includes a video capture device…that includes a processing apparatus that is configured to receive images).
It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Hijazi for an improved system, allowing for multiple modalities to determine speaking entity.
Regarding claim 17 Hijazi teaches The audio processing apparatus of claim 16, wherein the first voice vector representation associated with the first speaking entity is generated based on capturing the one or more portions of image data associated with the first speaking entity (0086: speaker is identified by voice and facial recognition).
Rejected for similar rationale and reasoning as claim 16
Regarding claim 18 Polyak does not specifically teach where Hijazi teaches The audio processing apparatus of claim 1, wherein the first audio transformation model is further configured to generate one or more portions of audio localization data associated with the first speaking entity, wherein the one or more portions of audio localization data comprises an estimated location of the first speaking entity relative to the audio processing apparatus
(0030: For example, application of an environment specific model for enhancement of audio from such environment may provide an advantage given the training and testing conditions are well matched. For example, training of an audio enhancement network may be achieved with less training examples by using data augmentation based on a collection of room acoustic models. Some implementations may provide speaker identification and/or customization for particular users.; 49-50;
0068: In some implementations, the selected audio enhancement network includes a neural network that has been trained using audio signals that have been augmented using room acoustic models.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Hijazi and localization for an improved system, allowing for the enhancement and regeneration to be customized for a specific location to properly adjust for the location and ensure improved speech intelligibility is achieved.
Regarding claim 19 Hijazi teaches The audio processing apparatus of claim 18, wherein the first voice vector representation associated with the first speaking entity is generated based on the audio localization data (0068: input the window of audio samples to a speaker identification network to obtain a speaker identification, in which the speaker identification network includes a neural network that has been trained using speech signals from one or more users to recognize speech signals as uttered by one of the one or more users; and select, based on the speaker identification, the audio enhancement network from the set of multiple audio enhancement networks, in which the selected audio enhancement network includes a speaker-specific embedding associated with the speaker identification. In some implementations, the selected audio enhancement network includes a neural network that has been trained using audio signals that have been augmented using room acoustic models).
Rejected for similar rationale and reasoning as claim 18
8. Claims 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over Polyak in view of Hijazi in further view of Balasubramaniam
Regarding claim 11 Polyak and Hijazi do not specifically teach where Balasubramaniam teaches The audio processing apparatus of claim 10, wherein the audio signals are configured for generating, at the second audio processing apparatus, a live text transcript representative of the one or more first speech signals associated with the first speaking entity (col 8 l. 2-5: may also include a contact transcript section that provides a transcript of the utterances made by the customer and/or the agent; col 10 l. 18-20).
It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate a transcript for recognition and additional representation of the speech signal.
Regarding claim 12 Hijazi teaches The audio processing apparatus of claim 11, wherein the {live text transcript is configured for} regeneration of the one or more first speech signals associated with the first speaking entity at the second audio processing apparatus, and wherein the regeneration of the one or more first speech signals is executed at the second audio processing apparatus simultaneously with the outputting of the audio signals by the second audio processing apparatus (fig 1; [0090] The process 500 includes storing, playing, or transmitting 560 an enhanced audio signal based on the enhanced audio segment.; 0092: the enhanced audio signal may be transmitted 560 to an external device (e.g., the media server 410 or a personal computing device) for display or storage;
Rejected for similar rationale and reasoning as claim 9/10),
And does not specifically teach where Balasubramaniam teaches the live text transcript (col 8 l. 2-5: may also include a contact transcript section that provides a transcript of the utterances made by the customer and/or the agent; col 10 l. 18-20).
It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate a transcript for recognition and additional representation of the speech/regenerated speech signal.
9. Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Polyak in view of Streit (2021/0141896).
Regarding claim 14 Polyak does not specifically teach where Streit teaches The audio processing apparatus of claim 1, wherein the instructions are further operable to cause the audio processing apparatus to:
encrypt both the first voice vector representation and the first speech vector representation (0067: generate encrypted embeddings or encrypted feature vectors (voice feature vectors)).
It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate encryption to further secure the specific data.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAUN A ROBERTS whose telephone number is (571)270-7541. The examiner can normally be reached Monday-Friday 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached on 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov.
For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHAUN ROBERTS/Primary Examiner, Art Unit 2655
