DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendments filed have been accepted and are considered in this office action. Claims 18 and 31 have been cancelled. Claims 16, 17, 19-30, and 32-35 are pending.
Response to Arguments
Applicant’s arguments with respect to the 35 U.S.C. 101 rejection of claims 16, 17, 19-30, and 32-35 have been considered and are persuasive regarding the amended limitations added. Subject matter from previous claims 18 and 31, which were indicated to have been allowable under 35 U.S.C. 101 in the previous office action, have been incorporated into independent claims 16 and 29 introducing the steps of “mixing… with a target-signal-to-interference ratio” to construct new datasets. These involve specific technical manipulations of digital data structures and are not characteristic of mental processes and are therefore not rejected under 35 U.S.C 101.
Applicant’s arguments with respect to the 35 U.S.C. 103 rejection of claims 16, 17, 19-30, and 32-35 have been considered but are moot in view of new grounds of rejection necessitated by the applicant’s amendments to the claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 16-17, 19, and 28 are rejected under 35. U.S.C. 103 as being unpatentable over Tzinis et al. (hereinafter Tzinis) (RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing) in view of Deng et al. (hereinafter Deng) (WO 2022151156 A1) (see attached copy for page number) in further view of Brigelb et al. (hereinafter Brigelb) (STATISTICAL ANALYSIS OF RANDOMNESS IN TRAINING OF SMALL-SCALE NEURALNETWORKSFORSPEECHENHANCEMENT) (see attached copy).
Regarding claim 16, Tzinis discloses:
method of interference signal reduction comprising:
receiving an input signal (Tzinis, Abstract: "a pre-trained teacher model on out-of-domain data infers estimated pseudo-target signals for in-domain mixtures.", Thus, teacher receives the in-domain mixture as its input)
providing the input signal to a machine learning model configured to output an estimate of a target signal from the input signal (Tzinis, Abstract: "a pre-trained teacher model on out-of-domain data infers estimated pseudo-target signals for in-domain mixtures.", The pretrained teacher is the machine-learning separation model, the in-domain mixture is its input, and the inferred pseudo-target is its estimate of the target signal);
and
adapting the machine learning model based on the target-only signal segments and the interference-only signal segments (Tzinis, Abstract: "Then, by permuting the estimated clean and noise signals and remixing
them together, we generate a new set of bootstrapped mixtures and corresponding pseudo-targets which are used to train the student network. ", Estimated clean/target and noise/interference material are both used in training/adapting the learned model),
wherein adapting the machine learning model further comprises generating a plurality of target-plus-interference signal segments by mixing (Tzinis, Abstract: "Then, by permuting the estimated clean and noise signals and remixing
them together, we generate a new set of bootstrapped mixtures and corresponding pseudo-targets which are used to train the student network.", Tzinis creates new target-plus-noise mixtures)
Tzinis does not explicitly disclose:
wherein the method further comprises:
classifying each input signal segment of the plurality of signal segments as one of a target- only signal segment, and an interference-only signal segment
plurality of signal segments;
providing the input signal to a machine learning model configured to output an estimate of a target signal from the input signal
for each target-plus-interference signal segment, a selected target-only signal segment and a selected interference-only signal segment having a target-signal-to-interference ratio.
However, Deng discloses:
wherein the method further comprises:
classifying each input signal segment of the plurality of signal segments as one of a target- only signal segment, and an interference-only signal segment (Deng, Fig 6, Elements S601-S603, Page 6: "
the SNR of the FF signal from the FF microphone is calculated.", Deng explains that ambient sound is treated as noise relative to the user's voice, and that when SNR reaches the high threshold the FF VAD flag becomes 1, elsewhere Deng states that a VAD flag of 1 indicates presence of the user's voice. Thus, the high-SNR/voice-present condition reads on the target only classification);’
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date to modify Tzinis’s self-training speech-enhancement system (Tzinis, Abstract, Page 1) using Deng’s SNR based speech/noise classification (Deng, Abstract, Pages 6). Doing so would have predictably enabled Tzinis to distinguish target-dominant and interference-dominant signal segments before adaptation, thus, improving selection of training signals for model refinement.
The combination of Tzinis and Deng does not explicitly disclose:
plurality of signal segments;
providing the input signal to a machine learning model configured to output an estimate of a target signal from the input signal
for each target-plus-interference signal segment, a selected target-only signal segment and a selected interference-only signal segment having a target-signal-to-interference ratio.
However, Briegleb discloses:
plurality of signal segments (Briegleb, Page 1: "the choice of the mixing order of the speech-only and noise-only signal segments for generating the noisy mixture. ", the underlying signal material comprises multiple signal segments);
providing the input signal to a machine learning model configured to output an estimate of a target signal from the input signal
for each target-plus-interference signal segment, a selected target-only signal segment and a selected interference-only signal segment (Briegleb, Page 1: "For generating training datasets for SE with ANNs the first instance of randomness is the choice of the mixing order of the speech-only and noise-only signal segments for generating the noisy mixture.", Individual speech-only and noise-only segments are selected/ordered to form each noisy mixture) having a target-signal-to-interference ratio (Briegleb, Page 1: "The next choice including randomness decides on the SNR of the training sample, which is usually chosen randomly from a range of possible SNRs according to the test scenario and determines the relative scaling of the mixture components.", The target/speech and interference/noise components have a selected SNR/relative ratio).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date to modify Tzinis’s self-training speech-enhancement system (Tzinis, Abstract, Page 1) using Deng’s SNR based speech/noise classification (Deng, Abstract, Pages 6) and Briegelb’s teaching of mixing selected speech and noise signals according to a selected SNR (Briegelb, Abstract, Page 1). Doing so would have predictably generated controlled noisy training mixtures from the classified target and interference signals, thus improving robustness of the trained speech enhancement model under different noise conditions.
Regarding claim 17, the combination of Tzinis, Deng, and Briegleb disclose the method of claim 16.
The combination further discloses:
wherein classifying each input signal segment further comprises:
estimating a target-signal-to-interference ratio of the input signal segment (Deng, Fig 6., S601, Page 6: "At S601, the SNR of the FF signal from the FF microphone is calculated.", Deng explains that the FF microphone captures the user's voice and ambient sound, with ambient sound treated as noise relative to the user's voice, therefore the calculated SNR corresponds to the target-to-interference ratio);
classifying the input signal segment as a target-only signal segment in response to the target-signal-to-interference ratio exceeding a first threshold value (Deng, Fig. 6/S602-S603: "If the SNR is greater than or equal to the high threshold TH.sub.H, then the flow goes to S603, in which the FF VAD flag is set to 1.", The high ratio condition produces the target/voice-present classification); and
classifying the input signal segment as an interference-only signal segment in response to the target-signal-to-interference ratio being less than a second threshold value lower than the first threshold value (Deng, Fig. 6/S604-S605: "If the SNR is less than or equal to the low threshold, then the method goes to S605, in which the FF VAD flag is set to 0.", The lower threshold produces the interference/voice-absent classification).
Regarding claim 19, the combination of Tzinis, Deng, and Briegleb disclose the method of claim 16.
The combination further discloses:
wherein the input signal further comprises at least one of an audio signal and a video signal (Tzinis, Abstract: "We present RemixIT, a simple yet effective self supervised method for training speech enhancement without the need of a single isolated in-domain speech nor a noise waveform.", RemixIT operates on speech/audio signals, satisfying the "at least one of" alternative).
Regarding claim 28, the combination of Tzinis, Deng, and Briegleb disclose the method of claim 16.
The combination further discloses:
receiving a further input signal comprising a plurality of further signal segments (Tzinis, Abstract: "RemixIT is based on a continuous self-training scheme in which a pre-trained teacher model on out-of-domain data infers estimated pseudo-target signals for in-domain mixtures.", The continuous scheme repeatedly processes additional in-domain mixtures, ex, further input data);
providing the further input signal to the machine learning model further configured to output an estimate of a further target signal from the further input signal (Tzinis, Abstract: "a pre-trained teacher model on out-of-domain data infers estimated pseudo-target signals for in-domain mixtures.", Each subsequently processed in-domain mixture is supplied to the learned teacher model and produces a corresponding further pseudo-target estimate);
wherein the method further comprises:
classifying each further input signal segment of the plurality of further signal segments as one of a target-only signal segment (Deng, Page 6: "If the SNR is greater than or equal to the high threshold TH.sub.H, then the flow goes to S603, in which the FF VAD flag is set to 1.", Deng states that VAD=1 indicates presence of the user's voice), an interference-only signal segment (Deng, Page 6: "If the SNR is less than or equal to the low threshold, then the method goes to S605, in which the FF VAD flag is set to 0.", Deng states that VAD=0 indicates absence of the user's voice) and, optionally, an undefined signal segment (Deng, Page 6: "If not, the method goes to S606, in which the FF VAD flag is set to a value between 0 and 1.", The intermediate SNR range supplies an intermediate/undefined class); and
adapting the machine learning model based on the target-only signal segments and the interference-only signal segments (Tzinis, Abstract: "Vice-versa, the teacher periodically refines its estimates using the updated parameters of the latest student models.", The model continues to adapt as successive data are processed).
Claims 20-22 are rejected under 35. U.S.C. 103 as being unpatentable over Tzinis et al. (hereinafter Tzinis) (RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing) in view of Deng et al. (hereinafter Deng) (WO 2022151156 A1) (see attached copy for page number) in further view of Brigelb et al. (hereinafter Brigelb) (STATISTICAL ANALYSIS OF RANDOMNESS IN TRAINING OF SMALL-SCALE NEURALNETWORKSFORSPEECHENHANCEMENT) (see attached copy) and Taneda (US 20060224382 A1).
Regarding claim 20, the combination of Tzinis, Deng, and Briegleb disclose the method of claim 16.
The combination further discloses:
classifying the input signal segment as a target-only signal segment in response to the noise power being less than a noise power threshold value (Deng, Page 6: "If the SNR is greater than or equal to the high threshold TH.sub.H, then the flow goes to S603, in which the FF VAD flag is set to 1.", Because Deng's SNR compares target voice against ambient noise, the favorable low-noise/high-SNR threshold condition supplies the target classification)
classifying the input signal segment as an interference-only signal segment in response to the noise power being greater than the noise power threshold value (Deng, Fig. 6/S604-S605, "If the SNR is less than or equal to the low threshold, then the method goes to S605, in which the FF VAD flag is set to 0.", A noise-dominant condition produces the non-target classification)
The combination does not explicitly disclose:
wherein the input signal further comprises an audio signal and a video signal and wherein classifying each input signal segment further comprises:
determining a noise power from an audio component of the input signal segment;
and voice activity being detected from a video component of the input signal segment; and
and no voice activity being detected from the video component of the input signal segment.
However, Taneda discloses:
wherein the input signal further comprises an audio signal and a video signal (Taneda, Abstract: "A noise reduction system including an audio-visual user interface combines visual features extracted from a digital video sequence with audio features extracted from an analog audio sequence.", supplies corresponding audio and video components) and wherein classifying each input signal segment further comprises:
determining a noise power from an audio component of the input signal segment (Taneda, P[0024]: "estimating the noise power density spectrum .PHl..sub.nn (f) of the statistically distributed background
noise n'(t) based on the result of the speaker detection step S3", Taneda determines an audio-background-noise power quantity);
and voice activity being detected from a video component of the input signal segment (Taneda, claim 8: "detecting the voice of said speaker from the discrete signal spectrum of the digital audio sequence using a learning procedure by analyzing the visual features extracted from the video sequence associated with the audio sequence", voice activity is detected using the associated video); and
and no voice activity being detected from the video component of the input signal segment (Tanada, P[0029]: "a visual speech activity estimate taken from a visual feature vector", P[0004]: "The final decision is made by comparing the output of the fuzzy system, which varies in a range between O and 1, with a fixed threshold experimentally chosen", P{0029]: "means are provided for switching said microphone 101 a off when the actual level of the speech activity indication signal s.sub.i(nT) falls below a predefined threshold value", Taneda teaches a thresholded speech-activity information that includes visual features).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date to modify Tzinis’s self-training speech-enhancement system (Tzinis, Abstract, Page 1) using Deng’s SNR based speech/noise classification (Deng, Abstract, Pages 6), Briegelb’s teaching of mixing selected speech and noise signals according to a selected SNR (Briegelb, Abstract, Page 1), and Taneda’s audio-visual speech detection (Taneda, Abstract, P[0029]). Doing so would have predictably supplemented audio-based target/interference classification with visual speech-activity information, thus, improving classification reliability when acoustic conditions are noisy or ambiguous.
Regarding claim 21, the combination of Tzinis, Deng, Briegleb, and Taneda disclose the method of claim 20.
The combination further discloses:
wherein adapting the machine learning model further comprises:
generating a plurality of target-plus-interference signal segments (Tzinis, Abstract: "Then, by permuting the estimated clean and noise signals and remixing them together, we generate a new set of bootstrapped mixtures and corresponding pseudo-targets which are used to train the student network.");
wherein the audio component of a target-plus-interference signal segment is generated by mixing the audio component of a selected target-only signal segment and the audio component of a selected interference-only signal segment (Briegleb, Page 1: "or generating training datasets for SE with ANNs the first instance of randomness is the choice of the mixing order of the speech-only and noise-only signal segments for generating the noisy mixture.") with a target-signal-to-interference ratio (Briegleb, Page 1: "The next choice including randomness decides on the SNR of the training sample, which is usually chosen randomly from a range of possible SNRs according to the test scenario and determines the relative scaling of the mixture components."); and
the video component of the target-plus-interference signal segment comprises the video component of the selected target-only signal segment (Taneda, Fig. 1, Abstract: "A noise reduction system including an audio-visual user interface combines visual features extracted from a digital video sequence with audio features extracted from an analog audio sequence.", The target-speaker video remains associated with the desired speech while Briegleb's augmentation modifies the audio by introducing noise).
Regarding claim 22, the combination of Tzinis, Deng, Briegleb, and Taneda disclose the method of claim 21.
The combination further discloses:
wherein adapting the machine learning model further comprises:
constructing a target-plus-interference data set from the plurality of target-plus- interference signal segments (Tzinis, Abstract: "we generate a new set of bootstrapped mixtures and corresponding pseudo-targets which are used to train the student network.", The bootstrapped mixtures constitute the target-plus-interference set));
constructing a target-only signal data set from a respective selected target-only signal segment of each target-plus-interference signal segment (Tzinis, Abstract: "we generate a new set of bootstrapped mixtures and corresponding pseudo-targets which are used to train the student network.", corresponding pseudo-targets form the target-side set associated with the bootstrapped mixtures); and
applying the target-plus-interference data set and the target-only signal data set to the machine learning model (Tzinis, Abstract: "we generate a new set of bootstrapped mixtures and corresponding pseudo-targets which are used to train the student network.", the mixtures and corresponding pseudo-targets "are used to train the student network.").
Claim 23 is rejected under 35. U.S.C. 103 as being unpatentable over Tzinis et al. (hereinafter Tzinis) (RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing) in view of Deng et al. (hereinafter Deng) (WO 2022151156 A1) (see attached copy for page number) in further view of Brigelb et al. (hereinafter Brigelb) (STATISTICAL ANALYSIS OF RANDOMNESS IN TRAINING OF SMALL-SCALE NEURALNETWORKSFORSPEECHENHANCEMENT) (see attached copy) and Borah et al. (hereinafter Borah) (US 7720012 B1).
Regarding claim 23, the combination of Tzinis, Deng, and Briegleb disclose the method of claim 16.
The combination does not explicitly disclose:
wherein the input signal comprises a digital audio signal, wherein each input signal segment comprises a digital audio packet comprising a plurality of audio samples within the digital audio signal, and wherein classifying each input signal segment further comprises:
classifying the input signal segment as a target-only signal segment in response to the digital audio packet being correctly received; and
classifying the input signal segment as an interference-only signal segment in response to the digital audio packet being incorrectly received However, Borah discloses:
wherein the input signal comprises a digital audio signal, wherein each input signal segment comprises a digital audio packet comprising a plurality of audio samples within the digital audio signal (Borah, P(18): "In order to simplify, the packet loss model (described below) is applied to fixed-sized packets each assumed to have a fixed number of speech samples.", each digital speech/audio packet contains a plurality of speech samples), and wherein classifying each input signal segment further comprises:
classifying the input signal segment as a target-only signal segment in response to the digital audio packet being correctly received (Borah, P(19): "When the state is `good`, the transmitted packet is received without any error; during a `bad` state the packet is considered lost.", the first half supplies a correctly received/good packet state); and
classifying the input signal segment as an interference-only signal segment in response to the digital audio packet being incorrectly received (Borah, P(19): "When the state is `good`, the transmitted packet is received without any error; during a `bad` state the packet is considered lost.", the bad state is a lost packet, supplying the impaired interference state).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date to modify Tzinis’s self-training speech-enhancement system (Tzinis, Abstract, Page 1) using Deng’s SNR based speech/noise classification (Deng, Abstract, Pages 6), Briegelb’s teaching of mixing selected speech and noise signals according to a selected SNR (Briegelb, Abstract, Page 1), and Borah’s fixed-size digital audio packets and received/lost packet states (Borah, Abstract, P(18)-P(19)). Doing so would have predictably permitted the system to classify packetized audio according to successful or unsuccessful reception, thus, extending the training and interference-reduction techniques to packet-loss conditions.
Claims 24-25 are rejected under 35. U.S.C. 103 as being unpatentable over Tzinis et al. (hereinafter Tzinis) (RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing) in view of Deng et al. (hereinafter Deng) (WO 2022151156 A1) (see attached copy for page number) in further view of Brigelb et al. (hereinafter Brigelb) (STATISTICAL ANALYSIS OF RANDOMNESS IN TRAINING OF SMALL-SCALE NEURALNETWORKSFORSPEECHENHANCEMENT) (see attached copy), Borah et al. (hereinafter Borah) (US 7720012 B1), and Liang (US 20220165280 A1).
Regarding claim 24, the combination of Tzinis, Deng, Briegleb, and Borah disclose the method of claim 23.
The combination further discloses:
of a packet loss pattern model (Borah, P(14): "The method optionally further comprises the steps of estimating or determining the packet loss rate for the test data; and applying the test data packet loss rate to the training data.", teaches characterizing the loss behavior and using that loss-rate parameter to control the packer-loss condition imposed on training data)
The combination does not explicitly disclose:
updating, based on classifying the input signal segment as an interference-only signal segment, at least one parameter with the interference-only signal segments
However, Liang discloses:
updating, based on classifying the input signal segment as an interference-only signal segment, at least one parameter (Liang, P[0072]: "S304a. Determine a packet loss parameter according to the received speech data packet in response to determining, according to the speech data packet, that a packet loss occurs.", observed lost packets determine a packet-loss parameter) with the interference-only signal segments (Liang, P[0072]-P[0075], Liang further states that the terminal determines whether packet loss occurs from packet serial numbers and determines the quantity of packet losses from those packets).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date to modify Tzinis’s self-training speech-enhancement system (Tzinis, Abstract, Page 1) using Deng’s SNR based speech/noise classification (Deng, Abstract, Pages 6), Briegelb’s teaching of mixing selected speech and noise signals according to a selected SNR (Briegelb, Abstract, Page 1), Borah’s fixed-size digital audio packets and received/lost packet states (Borah, Abstract, P(18)-P(19)), and Liang’s packet-loss parameter determination and packet loss based neural network training.. Doing so would have predictably used observed packet loss conditions to generate corresponding degraded and clean training data, thus, improving model performance for speech affected by packet loss.
Regarding claim 25, the combination of Tzinis, Deng, Briegleb, Borah, and Liang disclose the method of claim 24.
The combination further discloses:
wherein adapting the machine learning model further comprises:
constructing a target-only signal data set from a plurality of selected target-only signal segments (Liang, P[0099]: "the reference speech power spectrum being obtained by transforming an original speech data packet of the lost packets in the speech data sample.", The original, uncorrupted speech data corresponding to the subsequently lost packet provides the target/reference side of the training data);
constructing a target-plus-interference data set by muting a subset of the plurality of selected target-only signal segments (Liang, P[0094]: "If one packet loss needs to be simulated, one of the speech data packets with the packet serial numbers 1 to 10 is removed", teaching selective removal of otherwise-present target speech packets to simulate packet loss) according to the pattern loss packet model (Liang, P[0094]: "the packet loss parameter being used for representing a packet loss situation of a packet loss occurring in the speech data sample, and may be specifically a quantity of packet losses or a packet loss rate.", packet-loss corrupted examples are organized according to the represented packet-loss condition); and
applying the target-plus-interference data set and the target-only signal data set to the machine learning model (Liang, P[0107]: "The speech power spectrum sample in which one or more consecutive packet losses are simulated is used as an input of the neural network model", the corresponding original frame power spectrum is used as the training output, thus degraded packet-loss input and corresponding clean/reference target are applied to the neural network during training).
Claims 26-27 are rejected under 35. U.S.C. 103 as being unpatentable over Tzinis et al. (hereinafter Tzinis) (RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing) in view of Deng et al. (hereinafter Deng) (WO 2022151156 A1) (see attached copy for page number) in further view of Brigelb et al. (hereinafter Brigelb) (STATISTICAL ANALYSIS OF RANDOMNESS IN TRAINING OF SMALL-SCALE NEURALNETWORKSFORSPEECHENHANCEMENT) (see attached copy) and Preuss et al. (hereinafter Preuss) (US 20060184362 A1).
Regarding claim 26, the combination of Tzinis, Deng, and Briegleb disclose the method of claim 16.
The combination does not explicitly disclose:
further comprising storing target-only and interference-only signal segments in separate databases
However, Preuss discloses:
further comprising storing target-only (Preuss, P[0037]: " The database 118 stores a speech codebook 120, a noise codebook 122, and, optionally, a noisy speech codebook 124.", speech/target information is maintained in stored speech-codebook form) and interference-only signal segments in separate databases (Preuss, P[0037]: "According to alternative embodiments, the codebooks 120, 122, and 124 may be stored in separate databases.", Preuss further teaches that the noise representation is generated form speech-absent input segments).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date to modify Tzinis’s self-training speech-enhancement system (Tzinis, Abstract, Page 1) using Deng’s SNR based speech/noise classification (Deng, Abstract, Pages 6), Briegelb’s teaching of mixing selected speech and noise signals according to a selected SNR (Briegelb, Abstract, Page 1), and Preuss’s separate speech and noise codebooks and bounded codebook updating (Preuss, Abstract, P[0037]). Doing so would have predictably stored target and interference information separately while replacing older entries as new information is received, thus, maintaining representative signal information within a predetermined storage size.
Regarding claim 27, the combination of Tzinis, Deng, and Briegleb disclose the method of claim 26.
The combination further discloses:
wherein the databases are modified to stay within a predefined size after storing additional signal segments (Preuss, P[0013]: "According to another embodiment, the adaptive noise codebook includes at least 20 entries.", Preuss teaches replacement rather than unbounded addition: P[0077]: "According to a further embodiment, the noise codebook entry is updated (step 614) by replacing the least recently used noise codebook entry", the newly acquired noise information can replace an existing stored entry while the codebook remains bounded to its predetermined entry population).
Claim 29 is rejected under 35. U.S.C. 103 as being unpatentable over Tzinis et al. (hereinafter Tzinis) (RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing) in view of of Brigelb et al. (hereinafter Brigelb) (STATISTICAL ANALYSIS OF RANDOMNESS IN TRAINING OF SMALL-SCALE NEURALNETWORKSFORSPEECHENHANCEMENT) (see attached copy) and in further view of Borgstrom et al. (hereinafter Borgstrom) (US 20210074282 A1).
Regarding claim 29, Tzinis discloses:
a target signal estimator configured to output an estimate a target signal from the input signal, the target signal estimator comprising a machine learning model (Tzinis, Abstract: "a pre-trained teacher model on out-of-domain data infers estimated pseudo-target signals for in-domain mixtures.", The pretrained teacher is the learned estimator and the pseudo-target is the claimed target estimate) and having a target signal estimator input coupled to the signal input (Tzinis, Abstract: "a pre-trained teacher model on out-of-domain data infers estimated pseudo-target signals for in-domain mixtures.", Establishes that the in-domain mixtures are processed by the teacher model), a target signal estimator output coupled to the interference signal reduction system output (Tzinis, Abstract: "a pre-trained teacher model on out-of-domain data infers estimated pseudo-target signals for in-domain mixtures.", Establishes that the teacher outputs estimated pseudo-target signals), and an adaptor input (Tzinis, Abstract: "the teacher periodically refines its estimates using the updated parameters of the latest student models.", Updated model parameters feedback to modify/refine the estimator.);
an adaptor having a target signal input coupled to the target signal output, an interference signtzial input coupled to the interference signal output (Tzinis, Abstract: "Then, by permuting the estimated clean and noise signals and remixing them together, we generate a new set of bootstrapped mixtures and corresponding pseudo-targets which are used to train the student network. ", Both estimated clean/target and estimated noise/interference streams are supplied to the adaptation/remixing operation), and a model adaptor output coupled to the adaptor input (Tzinis, Abstract: "Vice-versa, the teacher periodically refines its estimates using the updated parameters of the latest student models. ", The model adaptation process produces updated parameters that are fed back to the teacher/model through the estimator's adaptation input);
adapt the machine learning model based on the target-only signal segments and the interference-only signal segments (Tzinis, Abstract: "Then, by permuting the estimated clean and noise signals and remixing them together, we generate a new set of bootstrapped mixtures and corresponding pseudo-targets which are used to train the student network.", Tzinis uses the clean/target and noise/interference estimates to train/adapt the model); and
generate a plurality of target-plus-interference signal segments by mixing (Tzinis, Abstract: "Then, by permuting the estimated clean and noise signals and remixing them together, we generate a new set of bootstrapped mixtures and corresponding pseudo-targets which are used to train the student network.", Tzinis generates a plurality of new target-plus-interference mixtures)
Tziris does not explicitly disclose:
An interference signal reduction system comprising:
a signal input configured to receive an input signal comprising a plurality of signal segments;
an interference signal reduction system output;
a signal classifier having a signal classifier input coupled to the signal input, a target signal output;
wherein the signal classifier is configured to classify each input signal segment of the plurality of signal segments as one of a target-only signal segment
having a target-signal-to-interference ratio
However, Briegleb discloses:
having a target-signal-to-interference ratio (Briegleb, Page 1: "The next choice including randomness decides on the SNR of the training sample, which is usually chosen randomly from a range of possible SNRs according to the test scenario and determines the relative scaling of the mixture components. ", The target/speech and interference/noise components are combined according to a selected SNR).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date to modify Tzinis’s self-training speech-enhancement system (Tzinis, Abstract, Page 1) using Briegelb’s teaching of mixing selected speech and noise signals according to a selected SNR (Briegelb, Abstract, Page 1). Doing so would have predictably generated varied target-plus-interference training mixtures from selected target and interference components, thus, improving robustness of the trained model across different noise conditions.
The combination of Tziris and Briegleb does not explicitly disclose:
An interference signal reduction system comprising:
a signal input configured to receive an input signal comprising a plurality of signal segments;
an interference signal reduction system output;
a signal classifier having a signal classifier input coupled to the signal input, a target signal output;
wherein the signal classifier is configured to classify each input signal segment of the plurality of signal segments as one of a target-only signal segment
However, Borgstrom discloses:
An interference signal reduction system comprising:
a signal input configured to receive an input signal comprising a plurality of signal segments (Borgstrom, P[0044]: "The system 100 can also include an input processor 90 configured to receive a raw audio signal or input waveform 101 (e.g., a time-domain signal) from, for example, a microphone, and generate the processed input spectrogram 110.", P[0045]: "In operation, the input processor 90 can generate, from a time-domain signal 101, a series of frames of pseudo-stationary segments and then apply the Fourier transform of each segment to generate the local spectra for each frame.", The system produces a noise-reduced/enhanced output);
an interference signal reduction system output (Borgstrom, P[0044]: "The system 100 is configured to receive, as an input, an initial spectrum 110 or spectrogram and output an enhanced spectrum 199, which represents the initial spectrum 110 with noise attenuation.", The system produces a noise-reduced/enhanced output);
a signal classifier having a signal classifier input coupled to the signal input (Borgstrom, P[0044]: "FIG. 1B illustrates a speech enhancement system 100 in accordance with the present disclosure, the system 100 using a DNN 120 speech detection classifier.", P[0045]: "Together, the frames can comprise the initial spectrum 110 to be provided to the DNN 120.", The segmented input signal is provided to the DNN classifier), a target signal output (Borgstrom, P[0045]: "The DNN 120 can be trained to make binary classifications for each spectrogram bin, where a bin is an individual frequency band of a single frame.", "DNN 120 can be trained to output 0 if the bin does not contain active speech with 100% probability, and 1 if the bin contains active speech with 100% probability", output corresponding to active/target speech is produced), and interference signal output (Borgstrom, P[0045]: "the DNN 120 can be trained to output 0 if the bin does not contain active speech with 100% probability", "the noise estimator 130 can detect non-stationary noises that the DNN 120 has identified as non-speech", the complimentary non-speech classifier output identifies the interference/noise side);
wherein the signal classifier is configured to classify each input signal segment of the plurality of signal segments as one of a target-only signal segment (Borgstrom, P[0049]: "Specifically, the network can be trained to label each time-frequency bin as belonging to one of two classes: H.sub.0 (k, m), where speech is absent in Y, or H.sub.1 (k, m), where speech is present in Y.", P[0050]: "The output of the DNN 120 is the posterior probability of active speech, P (H.sub.1|Y (k; m)), for each rime-frequency bin.", DNN classifies the individual frame/bin portions as speech-present/target), and an interference-only signal segment (Borgstrom, P[0049]: "Specifically, the network can be trained to label each time-frequency bin as belonging to one of two classes: H.sub.0 (k, m), where speech is absent in Y, or H.sub.1 (k, m), where speech is present in Y.", P[0045]: "noise estimator 130 can detect non-stationary noises that the DNN 120 has identified as non-speech", H0/non-speech classifications correspond to the interference/noise only portions);
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date to modify Tzinis’s self-training speech-enhancement system (Tzinis, Abstract, Page 1) using Briegelb’s teaching of mixing selected speech and noise signals according to a selected SNR (Briegelb, Abstract, Page 1) and Bergstrom’s DNN based speech enhancement system (Bergstrom, Abstract, P[0044]-P[0050]). Doing so would have predictably implemented the self training and controlled mixture techniques within an integrated interference reduction system capable of classifying and enhancing received signal segments, thus, improving automated target signal estimation and noise reduction.
Claim 30 is rejected under 35. U.S.C. 103 as being unpatentable over Tzinis et al. (hereinafter Tzinis) (RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing) in view of of Brigelb et al. (hereinafter Brigelb) (STATISTICAL ANALYSIS OF RANDOMNESS IN TRAINING OF SMALL-SCALE NEURALNETWORKSFORSPEECHENHANCEMENT) (see attached copy) and in further view of Borgstrom et al. (hereinafter Borgstrom) (US 20210074282 A1) and Deng et al. (hereinafter Deng) (WO 2022151156 A1) (see attached copy).
Regarding claim 30 claim 30 recites the interference signal reduction system corresponding to the method of claim 17 and is rejected for the same reasons as above.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date to modify Tzinis’s self-training speech-enhancement system (Tzinis, Abstract, Page 1) using Briegelb’s teaching of mixing selected speech and noise signals according to a selected SNR (Briegelb, Abstract, Page 1) and Bergstrom’s DNN based speech enhancement system (Bergstrom, Abstract, P[0044]-P[0050]) and Deng’s SNR threshold based speech presence classification. Doing so would have predictably enabled the signal classifier to distinguish target only and interference only segment according to relative target and noise levels, thus, improving selection of signal segments used for model adaptation.
Claims 32-34 are rejected under 35. U.S.C. 103 as being unpatentable over Tzinis et al. (hereinafter Tzinis) (RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing) in view of of Brigelb et al. (hereinafter Brigelb) (STATISTICAL ANALYSIS OF RANDOMNESS IN TRAINING OF SMALL-SCALE NEURALNETWORKSFORSPEECHENHANCEMENT) (see attached copy) and in further view of Borgstrom et al. (hereinafter Borgstrom) (US 20210074282 A1) and Deng et al. (hereinafter Deng) (WO 2022151156 A1) (see attached copy) and Taneda (US 20060224382 A1).
Regarding claim 32 claim 32 recites the interference signal reduction system corresponding to the method of claim 20 and is rejected for the same reasons as above.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date to modify Tzinis’s self-training speech-enhancement system (Tzinis, Abstract, Page 1) using Briegelb’s teaching of mixing selected speech and noise signals according to a selected SNR (Briegelb, Abstract, Page 1) and Bergstrom’s DNN based speech enhancement system (Bergstrom, Abstract, P[0044]-P[0050]) and Deng’s SNR threshold based speech presence classification (Deng, Abstract, Page 6). Doing so would have predictably enabled the signal classifier to distinguish target only and interference only segment according to relative target and noise levels, thus, improving selection of signal segments used for model adaptation.
Regarding claim 33 claim 33 recites the interference signal reduction system corresponding to the method of claim 21 and is rejected for the same reasons as above.
Regarding claim 34 claim 34 recites the interference signal reduction system corresponding to the method of claim 22 and is rejected for the same reasons as above.
Claim 35 is rejected under 35. U.S.C. 103 as being unpatentable over Tzinis et al. (hereinafter Tzinis) (RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing) in view of of Brigelb et al. (hereinafter Brigelb) (STATISTICAL ANALYSIS OF RANDOMNESS IN TRAINING OF SMALL-SCALE NEURALNETWORKSFORSPEECHENHANCEMENT) (see attached copy) and in further view of Borgstrom et al. (hereinafter Borgstrom) (US 20210074282 A1) and Deng et al. (hereinafter Deng) (WO 2022151156 A1) (see attached copy) and Borah et al. (hereinafter Borah) (US 7720012 B1) (see attached copy) and Liang et al. (hereinafter Liang) (US 20220165280 A1).
Regarding claim 35, the combination of Tzinis, Brigelb, Borgstrom, and Deng disclose the interference reduction system of claim 35.
The combination does not explicitly disclose:
However, Borah discloses:
wherein the input signal comprises a digital audio signal, the input signal segment comprises a digital audio packet comprising a plurality of audio samples within the digital audio signal (Borah, P(18): "the packet loss model (described below) is applied to fixed-sized packets each assumed to have a fixed number of speech samples.", each packet contains a plurality of speech/audio samples), and wherein the signal classifier is further configured to:
classify the input signal segment as a target-only signal segment in response to the digital audio packet being correctly received (Borah, P(19): "When the state is `good`, the transmitted packet is received without any error;);
classify the input signal segment as an interference-only signal segment in response to the digital audio packet being incorrectly received (Borah, P(19): "during a `bad` state the packet is considered lost.");
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date to modify Tzinis’s self-training speech-enhancement system (Tzinis, Abstract, Page 1) using Briegelb’s teaching of mixing selected speech and noise signals according to a selected SNR (Briegelb, Abstract, Page 1) and Bergstrom’s DNN based speech enhancement system (Bergstrom, Abstract, P[0044]-P[0050]), Deng’s SNR threshold based speech presence classification (Deng, Abstract, Page 6), and Borah’s classification of correctly received and lost digital audio packets (Borah, P(19)). Doing so would have predictably enabled packetized audio segments to be classified according to packet-reception status, thus, extending the interference-reduction system to packet-loss impaired audio. The combination of Tzinis, Brigelb, Borgstrom, Deng, and Borah does not explicitly disclose:
update, based on classification of the input signal segment as an interference-only signal segment, at least one parameter of a packet loss pattern model with the interference-only signal segments; and wherein
the adaptor is further configured to:
construct a target-only signal data set from a plurality of selected target-only signal segments;
construct a target-plus-interference data set by muting a subset of the plurality of selected target-only signal segments according to the pattern loss packet model; and
apply the target-plus-interference data set and the target-only signal data set to the machine learning model.
However, Liang discloses:
update, based on classification of the input signal segment as an interference-only signal segment, at least one parameter (Liang, P[0125]: "[0125] The determining module 704 is further configured to determine a packet loss parameter according to the received speech data packet;", "The packet loss parameter may be a quantity of packet losses or a packet loss rate.", Parameter representing the packer-loss condition is derived from the received packet stream) of a packet loss pattern model with the interference-only signal segments (Liang, P[0074]: "the terminal determines, according to packet serial numbers of the speech data packets, whether the packet loss occurs in the received speech data packets, and determines a quantity of packet losses according to the packet serial numbers in response to determining, according to the speech data packets, that a packet loss occurs.", P[0104]: "different neural network models are trained by using speech data samples with different quantities of packet losses", The observed lost-packet pattern is characterized by a packet-loss parameter and used to select/train a corresponding model); and wherein
the adaptor is further configured to:
construct a target-only signal data set from a plurality of selected target-only signal segments (Liang, P[0099]: "the reference speech power spectrum being obtained by transforming an original speech data packet of the lost packets in the speech data sample.", The original clean packet corresponding to each simulated-loss location provides the target/reference training data);
construct a target-plus-interference data set by muting a subset of the plurality of selected target-only signal segments (Liang, P[0094]: "If one packet loss needs to be simulated, one of the speech data packets with the packet serial numbers 1 to 10 is removed, or two nonconsecutive packets are removed. ", This is nearly a direct teaching of muting/removing selected portions of otherwise clean target speech) according to the pattern loss packet model (Liang, P[0094]: "the packet loss parameter being used for representing a packet loss situation of a packet loss occurring in the speech data sample, and may be specifically a quantity of packet losses or a packet loss rate.", P[0083]: "different quantities of packet losses correspond to different neural network models.", The corrupted training data are generated according to a represented packet-loss condition); and
apply the target-plus-interference data set and the target-only signal data set to the machine learning model (Liang, P[0103]: "the terminal back propagates the loss value to each layer of the neural network model, to obtain a gradient of a parameter of each layer; and adjusts the parameter of each layer in the neural network model according to the gradient", P[0107]: "The speech power spectrum sample in which one or more consecutive packet losses are simulated is used as an input of the neural network model, and an original frame power spectrum at a position of a packet loss is used as an output of the neural network model, to perform model training.").
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date to modify Tzinis’s self-training speech-enhancement system (Tzinis, Abstract, Page 1) using Briegelb’s teaching of mixing selected speech and noise signals according to a selected SNR (Briegelb, Abstract, Page 1) and Bergstrom’s DNN based speech enhancement system (Bergstrom, Abstract, P[0044]-P[0050]), Deng’s SNR threshold based speech presence classification (Deng, Abstract, Page 6), Borah’s classification of correctly received and lost digital audio packets (Borah, P(19)), and Liang’s packet loss parameter determination and packet-loss based neural network training (Liang, Abstract, P[0074], P[0094]-P[0099]). Doing so would have predictably generated packet loss corrupted inputs and corresponding clean target data according to observed packet loss conditions, thus, improving model performance for reconstructing or enhancing packet loss impaired speech.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHASHIDHAR S MANOHARAN whose telephone number is (571)272-6772. The examiner can normally be reached M-F 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHASHIDHAR SHANKAR MANOHARAN/Examiner, Art Unit 2655
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655
Prosecution Insights