DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in the application.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 06/28/2026 was filed in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
1. Claims 1-9, 11, 14-22, and 24 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1 “A method for generating a speech enhancement model” is recited, which is directed to one of the four statutory categories of invention (method; Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts:
training the first model and the second model based on the audio training data, wherein the training comprises…: training a first and second model amounts to mathematical calculations
…processing to obtain a first output result: a person analyzes a first audio data to obtain a first result
…processing to obtain a second output result: a person analyzes a second audio data to obtain a second result
updating parameters of the first model and parameters of the second model based on the second output result and the output audio data to obtain the trained first model and the trained second model; generating a speech enhancement model based on the trained first model and trained second model: updating parameters of a first and second model based on model outputs to obtain a speech enhancement model amounts to mathematical calculations
Claim 1 does not contain any additional elements which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). The additional limitations in claim 1 either amount to mere instructions to implement the judicial exception using a generic computer (“obtaining a first model and a second model wherein the first model is a deep neural network model, and resources occupied by the second model during operation is less than that occupied by the first model ”; “inputting the first audio data into the first model for processing…”; “inputting the second audio data and the first output result into the second model for processing…”) or amount to insignificant extra-solution activity in the form of mere data gathering (“obtaining audio training data, wherein the audio training data comprises noisy input audio data and noiseless output audio data corresponding to the noisy input audio data”,” obtaining first audio data having a frame length of M from the input audio data”, “obtaining second audio data having a frame length of N from the input audio data, wherein the second audio data is subsequent to the first audio data, and N<M”), which do not integrate the judicial exception into a practical application as they do not provide any meaningful limits on practicing the abstract ideas. Therefore, claim 1 is directed to an abstract idea.
Claim 1 does not contain any additional limitations which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations amount to mere instructions to implement the judicial exception using a generic computer or amount to insignificant extra-solution activity in the form of mere data gathering. Further, the acts of obtaining audio training data, obtaining first audio data, and obtaining second audio data amount to receiving data at a high level of generality, which is well-understood, routine, and conventional in the art (see MPEP 2106.05(d)(II)). These additional limitations do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claim 1 is not patent eligible.
Regarding claims 2-9 and 11, “The method” is recited, which is directed to one of the four statutory categories of invention (method; Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite further mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts:
Claim 2:
Claim 2 recites “wherein the second model is a deep neural network model, and the parameters of the second model are fewer than the parameters of the first model”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 3:
Claim 3 recites “wherein a ratio of a quantity of the parameters of the second model to a quantity of the parameters of the first model is less than 1/2”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 4:
Claim 4 recites “wherein the first model is a convolutional recurrent neural network model, and the second model is an RNNoise model”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 5:
combining the third audio data with the second audio data to form fourth audio data having a frame length of P+N: mathematical calculations
…processing to obtain an intermediate output result having a frame length of P+N: a person writes down an intermediate output result corresponding to the frames
obtaining a result having a length of N and corresponding to the second audio data from the intermediate output result to serve as the second output result: a person can write down information about the second audio data based on an intermediate output
Claim 5 recites “obtaining third audio data…”, which amounts to further insignificant extra-solution activity in the form of mere data gathering. Claim 5 further recites “inputting the fourth audio data and the first output result into the second model for processing…”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 6:
Claim 6 contains the additional limitation “wherein the second model comprises a digital signal processing model and a simplified neural network model”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 7:
Claim 7 contains the additional limitation “wherein the second model comprises a power spectrum estimation model and a fully connected layer”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 8:
Claim 8 contains the additional limitation “inputting the second audio data into the digital signal processing model for processing…” and “inputting the intermediate output result and the first output result to the simplified neural network model for processing…”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 9:
wherein M=KxN, and K is an integer greater than 1: mathematical calculations
processing to obtain K second output results: a person writes down K results corresponding to K pieces of second audio data
and the parameters of the first model and the parameters of the second model are updated based on the K second output results and the output audio data to obtain the trained first model and the trained second model: updating parameters based on model outputs amounts to mathematical calculations
Claim 9 further recites “wherein, during training the first model and the second model based on the audio data, K consecutive pieces of second audio data each having a frame length of N subsequent to the first audio data are obtained from the input audio data”, which amounts to further insignificant extra-solution activity in the form of mere data gathering. Claim 9 further recites “K pieces of second audio data are sequentially inputted, together with the first output result, into the second model for processing to obtain K second output results”, which amounts to mere instructions to implement the judicial exception using a generic computer.
Claim 11:
wherein a step of training the first model and the second model based on the audio training data is executed iteratively for a plurality of times, wherein in an iterative execution process, updated values of the parameters of the trained first model and updates values of the parameters of the trained second model that are generated in a previous training are respectively assigned to the first model to be trained and second model to be trained for using in a subsequent training: training to update parameters over a plurality of iterations amounts to mathematical calculations
Claims 2-9 and 11 do not contain any additional limitations which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). As discussed above, the additional limitations amount to insignificant extra-solution activity or mere instructions to implement the judicial exception using a generic computer, which do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the abstract ideas. Therefore, claims 2-9 and 11 are directed to abstract ideas.
Claims 2-9 and 11 do not contain any additional limitations which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations amount to mere instructions to implement the judicial exception using a generic computer or amount to insignificant extra-solution activity in the form of mere data gathering. Further, the acts of obtaining audio data amount to receiving data at a high level of generality, which is well-understood, routine, and conventional in the art (see MPEP 2106.05(d)(II)). These additional limitations do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claims 2-9 and 11 are not patent eligible.
Regarding claim 14 and 24 “A speech enhancement method” and “A speech enhancement device” are recited, which are each directed to one of the four statutory categories of invention (method and machine; Step 1: YES). However, the claims limitations of claim 14, and by extension claim 24 as claim 24 recites the user of the method of claim 14, under their broadest reasonable interpretation, recite mental processes or mathematical concepts which fall into the category of abstract idea (Step 2A Prong 1: YES).
The following limitations, under their broadest reasonable interpretation, recite mental processes or mathematical concepts:
…processing to obtain a first output result: a person analyzes a first audio data to obtain a first result
…processing to obtain a second output result; and outputting the second output result as an enhanced audio data: a person analyzes a second audio data to obtain a second result
Claims 14 and 24 do not contain any additional elements which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). The additional limitations in claims 14 and 24 either amount to mere instructions to implement the judicial exception using a generic computer (“obtaining a speech enhancement model, wherein the speech enhancement model comprises a first model and a second model, the first model is a deep neural network model, and resources occupied by the second model during operation is less than that occupied by the first model ”; “processing the audio data by using the speech enhancement model to reduce or remove a noise signal in the audio data”, “inputting the first audio data into the first model for processing…”; “inputting the second audio data and the first output result into the second model for processing…”; a processor; and a storage apparatus configured to store a computer program executable by the processor, wherein the computer program, when executed by the processor, causes the processor to perform the speech enhancement method of claim 14”) or amount to insignificant extra-solution activity in the form of mere data gathering (“obtaining an audio data”,” obtaining first audio data having a frame length of M from the input audio data”, “obtaining second audio data having a frame length of N from the input audio data, wherein the second audio data is subsequent to the first audio data, and N<M”), which do not integrate the judicial exception into a practical application as they do not provide any meaningful limits on practicing the abstract ideas. Therefore, claims 14 and 24 are directed to an abstract idea.
Claims 14 and 24 does not contain any additional limitations which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations amount to mere instructions to implement the judicial exception using a generic computer or amount to insignificant extra-solution activity in the form of mere data gathering. Further, the acts of obtaining audio training data, obtaining first audio data, and obtaining second audio data amount to receiving data at a high level of generality, which is well-understood, routine, and conventional in the art (see MPEP 2106.05(d)(II)). These additional limitations do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claim 1 is not patent eligible.
Regarding claims 15-22, “The speech enhancement method” is recited, which is directed to one of the four statutory categories of invention (method; Step 1: YES). However, the claims limitations, under their broadest reasonable interpretation, recite limitations similar to those recited in claims 2-9 and 11, and thus also recite mental processes or mathematical concepts (Step 2A Prong 1: YES).
Claims 15-22 do not contain any additional limitations which integrate the judicial exception into a practical application (Step 2A Prong 2: NO). As discussed above, the additional limitations amount to insignificant extra-solution activity or mere instructions to implement the judicial exception using a generic computer, which do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the abstract ideas. Therefore, claims 15-22 are directed to abstract ideas.
Claims 15-22 do not contain any additional limitations which amount to significantly more than the judicial exception (Step 2B: NO). As discussed above, the only additional limitations amount to mere instructions to implement the judicial exception using a generic computer or amount to insignificant extra-solution activity in the form of mere data gathering. Further, the acts of obtaining audio data amount to receiving data at a high level of generality, which is well-understood, routine, and conventional in the art (see MPEP 2106.05(d)(II)). These additional limitations do not amount to significantly more than the judicial exception as they do not provide an inventive concept. Therefore, claims 15-22 are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
2. Claims 1-2, 5, 9, 11, 14-15, 18, 22, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Hsiung (US 2020/0312343 A1) in view of Strimel et al. (US 12,205,574 B1, hereinafter Strimel).
Regarding claim 1, Hsiung discloses A method for generating a speech enhancement model, comprising: obtaining audio training data, wherein the audio training data comprises noisy input audio data and noiseless output audio data corresponding to the noisy input data (para. 0034 “Two databases are required for the above-described second-stage noise suppression device 160, i.e. clean speech database and ambient noise database. The clean speech data is mixed with the ambient noise data based on various signal-to-noise ratios to provide input samples in the training process, while the clean speech data serves as output noise suppression samples of the training process.”); obtaining a first model and a second model, wherein the first model is a deep neural network model…(Fig. 2, 162 and 164; model 162 is a recurrent neural network, which is a deep neural network (para. 0034)); training the first model and the second model based on the audio training data, wherein the training comprises (para. 0034): obtaining first audio data having a frame length of M from the input audio data (Fig. 2, MS; para. 0028 “In this embodiment, the pre-processor 120 receives the digital audio input DA, and then divides/frames a section of the digital audio input DA into audio frames.”; para. 0030 “One digital audio signal corresponds to one audio frame in a one-to-one manner. Each digital audio signal includes a magnitude signal MS and a phase signal PS. For example, the data in the nth audio frame is processed to generate the nth digital audio signal which includes a magnitude signal MS and a phase signal PS. Similarly, the data in the (n+1)th audio frame is processed to generate the (n+1)th digital audio signal which also includes a magnitude signal MS and a phase signal PS..”); inputting the first audio data into the first model for processing to obtain a first output result (Fig. 2, MS input to first model 162 to obtain output VD); obtaining second audio data having a frame length of N from the input audio data…( Fig. 2, MS; para. 0028 “In this embodiment, the pre-processor 120 receives the digital audio input DA, and then divides/frames a section of the digital audio input DA into audio frames.”; para. 0030 “One digital audio signal corresponds to one audio frame in a one-to-one manner. Each digital audio signal includes a magnitude signal MS and a phase signal PS. For example, the data in the nth audio frame is processed to generate the nth digital audio signal which includes a magnitude signal MS and a phase signal PS. Similarly, the data in the (n+1)th audio frame is processed to generate the (n+1)th digital audio signal which also includes a magnitude signal MS and a phase signal PS..”); inputting the second audio data and the first output result into the second model for processing to obtain a second output result (Fig. 2, second audio ‘MS’ and first output ‘VD’ input to model 164 to obtain second output result ‘NS’); and updating parameters of the first model and parameters of the second model based on the second output result and the output audio data to obtain the trained first model and the trained second model (para. 0034 “In the training process, the voice activity detection belongs to a binary classification problem, and the loss function is the cross entropy between the ground truth and the estimated voice activity detection signal generated by the recurrent neural network 162 of the second-stage noise suppression device 160 in response to each input sample. On the other hand, the ambient noise suppression belongs to a regression problem, and the loss function is the mean square error (MSE) between the clean speech signal and the noise suppression signal NS generated by the recurrent neural network 164 of the second-stage noise suppression device 160 in response to each input sample. It is to be noted that the network layer can adopt any proper variant of recurrent neural network layer, and the loss functions of the classification and regression can adopt any function suitable for training deep neural network (DNN).”); and generating a speech enhancement model based on the trained first model and the trained second model (trained first and second models used for enhancing speech: para. 0027).
Hsiung does not specifically disclose that resources occupied by the second model during operation is less than that occupied by the first model… and wherein the second audio is subsequent to the first audio data, and N<M.
Strimel teaches resources occupied by the second model during operation is less than that occupied by the first model (Col. 5 Lines 54-58 “For example, the first component may include a first ML model that may have more processing layers and/or processing nodes than a second ML model of the second component. In another example, the first ML model may have more parameters than the second ML model.”)… and wherein the second audio is subsequent to the first audio data, and N<M (a first and second machine learning model are taught, with a first and second frame processed via a first model and a third frame processed subsequent to the first and second frame processed via a third model: (e.g. M=2, N=1, thus N<M) Col. 3 Lines 39-49 “The arbitrator module of the present disclosure may make a frame-by-frame decision on which machine learning model to use or may allocate audio data to a particular machine learning model using different sizes of sections of audio data (e.g., certain time windows worth of data, subframe quantities of data, etc.). For example, the arbitrator module may determine to use a first machine learning model to process a first audio frame and a second audio frame of the input audio data, then determine to use a second machine learning model to process a third audio frame of the input audio data. In switching between machine learning models, the system, in some embodiments, may share state data from the model that processed a first/prior audio frame in a sequence of audio frames of the input audio data with the model that is to process a second/later audio frame in the sequence.”).
Hsiung and Strimel are considered to be analogous to the claimed invention as
they both are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hsiung to incorporate the teachings of Strimel in order to specifically have resources occupied by the second model during operation be less than that occupied by the first model, and to have the second audio data be subsequent to the first audio data and N<M. Doing so would be beneficial, as this would allow for frame-by-frame decision making on which machine learning model to use for different section s of audio data (Col. 3 Lines 39-44), allowing for faster processing of input data while maintaining high accuracy (Col. 4, Lines 36-40).
Regarding claim 2, Hsiung in view of Strimel discloses wherein the second model is a deep neural network model (Hsiung, Fig. 2, 162 and 164; model 164 is a recurrent neural network, which is a deep neural network (para. 0034)), and the parameters of the second model are fewer than the parameters of the first model (Strimel, Col. 5 Lines 54-58 “For example, the first component may include a first ML model that may have more processing layers and/or processing nodes than a second ML model of the second component. In another example, the first ML model may have more parameters than the second ML model.”).
Hsiung and Strimel are considered to be analogous to the claimed invention as
they both are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hsiung to incorporate the teachings of Strimel in order to specifically have the parameters of the second model be fewer than the parameters of the first model. Doing so would be beneficial, given the same rationale as for claim 1.
Regarding claim 5, Hsiung in view of Strimel discloses wherein training the first model and the second model based on the audio training data further comprises (Hsiung, see claim mapping for claim 1): obtaining third audio data with a frame length of P previous to the second audio data from the first input audio data (Hsiung, teaches obtaining frames previous to second audio data from first audio data (Fig. 7, e.g. at time t, audio frame fn is accompanied with preceding frames f1 to fn-1)); and combining the third audio with the second audio to form fourth audio data having a frame length of P + N (Hsiung, Fig. 7, second audio (frame fn) combined with third audio (frames f1 to fn-1) to obtain fourth audio data (collection of frames input ‘MS’ to RNN 162); see para. 0043) wherein inputting the second audio data and the first output result into the second model for processing to obtain the second output result comprises: inputting the fourth audio data and the first output result into the second model for processing to obtain an intermediate output result having a frame length of P + N (Hsiung, Fig. 2, collection of frames ‘MS’ input, along with first output result ‘VD’, input to second model 164 to obtain an intermediate result (output of 1640a) corresponding to the collection of frames) and obtaining a result having a length of N and corresponding to the second audio data from the intermediate output result to serve as the second result (intermediate output (output of LSTM layer) used to generate second result ‘NS’, which corresponds to the second audio data N (the nth frame): para. 0033 “Then, the recurrent neural network 164 generates an nth noise suppression signal NS according to the magnitude signal MS of the nth digital audio signal, the nth artifact signal AS and the nth voice activity detection signal VD. The nth noise suppression signal NS is obtained by reducing the non-stationary noise in the nth artifact signal AS.”).
Regarding claim 9, Hsiung in view of Strimel discloses wherein M=K x N, and K is an integer greater than 1 (Strimel, teaches inputting M frames to a first model, and N frames to a second model, where K is greater than one (in this case, M=2 and N=1): Col. 3 Lines 39-49 “The arbitrator module of the present disclosure may make a frame-by-frame decision on which machine learning model to use or may allocate audio data to a particular machine learning model using different sizes of sections of audio data (e.g., certain time windows worth of data, subframe quantities of data, etc.). For example, the arbitrator module may determine to use a first machine learning model to process a first audio frame and a second audio frame of the input audio data, then determine to use a second machine learning model to process a third audio frame of the input audio data. In switching between machine learning models, the system, in some embodiments, may share state data from the model that processed a first/prior audio frame in a sequence of audio frames of the input audio data with the model that is to process a second/later audio frame in the sequence.”); and wherein training the first model and the second model based on the audio training data (Hsiung teaches training of first and second models, see claim mapping for claim 1), K consecutive pieces of second audio data each having a frame length of N subsequent to the first audio data are obtained from the audio data (Strimel, a plurality of sequential audio frames each of length N=1 determined: Col. 5 Lines 13-16 “The system(s) 120 or the device 110 determines (132) a plurality of audio frames from the audio data. In some embodiments, the system(s) 120 or the device 110 may divide the audio data into a sequence of audio frames.”), and the K pieces of second audio data are sequentially inputted, together with the first output result, into the second model for processing to obtain K second output results (Strimel, a first output result (Fig. 4, output of model 220) along with frame 406 (and potentially K subsequent frames each input sequentially: Col. 10 Lines 62-65 “It should be understood that more than two audio frames may be determined from the input audio data 402 even though only two audio frames are shown in FIG. 4.”) input to second model (Fig. 4, model 230) to obtain K second outputs results (Fig. 4, output data 452)), and the parameters of the first model and the parameters of the second model are updated based on the K second output results and the output audio data to obtain the trained model and the trained second model (Hsiung discloses training of a first and second model based on outputs of the first and second model: para. 0034 “In the training process, the voice activity detection belongs to a binary classification problem, and the loss function is the cross entropy between the ground truth and the estimated voice activity detection signal generated by the recurrent neural network 162 of the second-stage noise suppression device 160 in response to each input sample. On the other hand, the ambient noise suppression belongs to a regression problem, and the loss function is the mean square error (MSE) between the clean speech signal and the noise suppression signal NS generated by the recurrent neural network 164 of the second-stage noise suppression device 160 in response to each input sample. It is to be noted that the network layer can adopt any proper variant of recurrent neural network layer, and the loss functions of the classification and regression can adopt any function suitable for training deep neural network (DNN).”).
Hsiung and Strimel are considered to be analogous to the claimed invention as
they both are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hsiung to incorporate the teachings of Strimel in order to specifically have M=KxN, K be an integer greater than 1; and wherine during processing the audio data by using the speech enhancement model, K consecutive pieces of second audio data each having a frame length of N subsequent to the first audio data are obtained from the audio data, and the K pieces of second audio data are sequentially inputted, together with the first output result, into the second model for processing to obtain K second output results. Doing so would be beneficial, given the same rationale as for claim 1.
Regarding claim 11, Hsiung in view of Strimel discloses wherein a step of training the first model and the second model based on the audio training data is executed iteratively for a plurality of times (Hsiung discloses training first and second models (see above mapping for claim 1); Strimel teaches training iterations for a first and second branch/model Col. 12 Lines 28-35 “Each of the branches 520 and 530 may retain its own set of parameters. As such, the branches can have different compute costs and/or can be specialized to model different sub-distributions. During training, the system 500 may train both the arbitrator component 510 and the branches 520, 530 to give the system 500 the capability to act in a dynamic capacity on a frame-by-frame basis.”; Col. 8 Lines 47-49 “During a first training iteration, illustrated by 270 in the Fig. 2, the system 200 may process an audio frame 204…”’ Col. 9 Lines 16-17 “During a second training iteration, illustrated by 272 in the FIG. 2, the system 200 may process an audio frame 206.”), wherein in an iterative execution process, updated values of the parameters of the trained first model and updated values of the parameters of the trained second model that are generated in a previous training are respectively assigned to the first model to be trained and second model to be trained for using in a subsequent training (Col. 13 Lines 7-17 “As shown in the Fig. 5A, the output of the arbitrator component 510 may be forwarded directly to the compute cost aggregation layer 550 for use in the loss function for training. In some embodiments, the compute cost aggregation layer 550 may penalize the use of the dense RNN/the branch 520 more than the use of the sparse RNN/the branch 530. In this example system 500, each of the RNNs/branches 520, 530 are also trained, for ASR processing, along with the arbitrator component 510, and each of the branch 520, 530 has an opportunity to learn different things, with the sparse RNN/branch 530 being well optimized for the “typical case” and the dense RNN/branch 520 having the capacity to handle atypical, harder cases.”; Col. 8 Lines 47-49 “During a first training iteration, illustrated by 270 in the Fig. 2, the system 200 may process an audio frame 204…”’ Col. 9 Lines 16-17 “During a second training iteration, illustrated by 272 in the FIG. 2, the system 200 may process an audio frame 206.”).
Hsiung and Strimel are considered to be analogous to the claimed invention as
they both are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hsiung to incorporate the teachings of Strimel in order to specifically train the first and second model via training steps based on the audio training data and executed iteratively for a plurality of times, and to update values of the parameters of the trained first model and second model, and to use previous values of parameters in previous training for use in subsequent training. Doing so would be beneficial, as training over multiple iterations via a loss function would lead to better model accuracy.
Regarding 14, Hsiung discloses A speech enhancement method, comprising: obtaining an audio data (para. 0036 “In the embodiment, at first, the sound collecting device 100 or the like acquires an audio input (e.g. analog audio input IS or digital audio input DA) (step S300)”); obtaining a speech enhancement model, wherein the speech enhancement model comprises a first model and a second model, the first model is a deep neural network model…(trained first and second models used for enhancing speech: para. 0027; Fig. 2, first model 162 and second model 164; first model 162 is a recurrent neural network, which is a deep neural network (para. 0034)); processing the audio data by using the speech enhancement model to reduce or remove a noise signal in the audio data (para. 0041 “The second-stage noise suppression device 160 performs the non-stationary noise suppression process (using neural network processing approach) on the received signals (step S330)…”; para. 0033 “In the embodiment, the second-stage noise suppression device 160 (using neural network approach) includes a recurrent neural network (RNN) 162 for voice activity detection and a recurrent neural network 164 for noise suppression.”), wherein the processing comprises: obtaining first audio data having a frame length of M from the audio data (Fig. 2, MS; para. 0028 “In this embodiment, the pre-processor 120 receives the digital audio input DA, and then divides/frames a section of the digital audio input DA into audio frames.”; para. 0030 “One digital audio signal corresponds to one audio frame in a one-to-one manner. Each digital audio signal includes a magnitude signal MS and a phase signal PS. For example, the data in the nth audio frame is processed to generate the nth digital audio signal which includes a magnitude signal MS and a phase signal PS. Similarly, the data in the (n+1)th audio frame is processed to generate the (n+1)th digital audio signal which also includes a magnitude signal MS and a phase signal PS..”); inputting the first audio data into the first model for processing to obtain a first output result (Fig. 2, MS input to first model 162 to obtain output VD); obtaining a second audio data having a frame length of N from the input audio…( Fig. 2, MS; para. 0028 “In this embodiment, the pre-processor 120 receives the digital audio input DA, and then divides/frames a section of the digital audio input DA into audio frames.”; para. 0030 “One digital audio signal corresponds to one audio frame in a one-to-one manner. Each digital audio signal includes a magnitude signal MS and a phase signal PS. For example, the data in the nth audio frame is processed to generate the nth digital audio signal which includes a magnitude signal MS and a phase signal PS. Similarly, the data in the (n+1)th audio frame is processed to generate the (n+1)th digital audio signal which also includes a magnitude signal MS and a phase signal PS..”); and inputting the second audio data and the first output result into the second model for processing to obtain a second output result (Fig. 2, second audio ‘MS’ and first output ‘VD’ input to model 164 to obtain second output result ‘NS’); outputting the second output result as an enhanced audio data (Fig. 2, output ‘NS’; para. 0033 “The nth noise suppression signal NS is obtained by reducing the non-stationary noise in the nth artifact signal AS.”).
Hsiung does not specifically disclose that resources occupied by the second model during operation is less than that occupied by the first model… and wherein the second audio is subsequent to the first audio data, and N<M.
Strimel teaches resources occupied by the second model during operation is less than that occupied by the first model (Col. 5 Lines 54-58 “For example, the first component may include a first ML model that may have more processing layers and/or processing nodes than a second ML model of the second component. In another example, the first ML model may have more parameters than the second ML model.”)… and wherein the second audio is subsequent to the first audio data, and N<M (a first and second machine learning model are taught, with a first and second frame processed via a first model and a third frame processed subsequent to the first and second frame processed via a third model: (e.g. M=2, N=1, thus N<M) Col. 3 Lines 39-49 “The arbitrator module of the present disclosure may make a frame-by-frame decision on which machine learning model to use or may allocate audio data to a particular machine learning model using different sizes of sections of audio data (e.g., certain time windows worth of data, subframe quantities of data, etc.). For example, the arbitrator module may determine to use a first machine learning model to process a first audio frame and a second audio frame of the input audio data, then determine to use a second machine learning model to process a third audio frame of the input audio data. In switching between machine learning models, the system, in some embodiments, may share state data from the model that processed a first/prior audio frame in a sequence of audio frames of the input audio data with the model that is to process a second/later audio frame in the sequence.”).
Hsiung and Strimel are considered to be analogous to the claimed invention as
they both are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hsiung to incorporate the teachings of Strimel in order to specifically have resources occupied by the second model during operation be less than that occupied by the first model, and to have the second audio data be subsequent to the first audio data and N<M. Doing so would be beneficial, as this would allow for frame-by-frame decision making on which machine learning model to use for different section s of audio data (Col. 3 Lines 39-44), allowing for faster processing of input data while maintaining high accuracy (Col. 4, Lines 36-40).
Regarding claim 15, claim 15 is rejected for analogous reasons to claim 2.
Regarding claim 18, claim 18 is rejected for analogous reasons to claim 5.
Regarding claim 22, claim 22 is rejected for analogous reasons to claim 9.
Regarding claim 24, claim 24 is a device claim which recites the use of the speech enhancement method of claim 14, and thus is rejected under similar rationale as claim 14.
Additionally, Hsiung in view of Strimel discloses A speech enhancement device (Hsiung, device 160), comprising: a processor (Strimel, Fig. 10, 1004); and a storage apparatus configured to store a computer program executable by the processor (Fig. 10, 1006), wherein the computer program, when executed by the processor, causes the processor (Col. 27 Lines 63-67) to perform the speech enhancement method of claim 14 (see claim mapping for claim 14).
3. Claims 3 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Hsiung in view of Strimel, and further in view of Aslan et al. (US 2017/0132528 A1, hereinafter Aslan).
Regarding claim 3¸ Hsiung in view of Strimel discloses the concept of the second model having less parameters than the first model (see above claim mapping for claim 1), but does not specifically disclose wherein a ratio of a quantity of the parameters of the second model to a quantity of the parameters of the first model is less than ½.
Aslan teaches wherein a ratio of a quantity of the parameters of the second model to a quantity of the parameters of the first model is less than ½ (para. 0044 “For example, the first (teacher) model 100 of FIG. 1 can comprise a large, complex ensemble of machine learning models that is often too large and/or slow to be used at run-time in particular scenarios. Meanwhile, the second (student) model 102 can comprise a much smaller machine learning model (e.g., a neural net with 1000 times fewer parameters than the first model 100) that has the size and/or speed that is advantageous at run-time in particular scenarios.”).
Hsiung, Strimel, and Aslan are considered to be analogous to the claimed invention as Hsiung and Strimel are in the same field of speech processing, and Aslan is in the same field of machine learning model architectures. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hsiung in view of Strimel to incorporate the teachings of Aslan in order to specifically have the ratio of the quantity of parameters in the second model to the quantity of parameters in the first model be less than 1/2. Doing so would be beneficial, as such configuration would lead to the second model taking much less memory and being able to operate faster and on less processing power at runtime (para. 0044).
Regarding claim 16, claim 16 is rejected for analogous reasons to claim 3.
4. Claims 4 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Hsiung in view of Strimel, and further in view of Zhao et al. (NPL Convolutional-Recurrent Neural Networks for Speech Enhancement, hereinafter Zhao) and Valin (NPL A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement).
Regarding claim 4, Hsiung in view of Strimel does not specifically disclose wherein the first model is a convolutional recurrent neural network model…
Zhao teaches wherein the first model is a convolutional recurrent neural network model… (see Fig. 1 and caption, “Model architecture. EHNet consists of three components: noisy spectrograms is first convolved with kernels to form feature maps, which are then concatenated to form a 2D feature map. The 2D feature map is further transformed by a bidirectional RNN along the time dimension…”).
Hsiung, Strimel, and Zhao are considered to be analogous to the claimed invention as they are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hsiung in view of Strimel to incorporate the teachings of Zhao in order to specifically have the first model be a convolutional recurrent neural network. Doing so would be beneficial, as the convolution kernels can efficiently detect local patterns in spectrograms and the recurrent connections can automatically model the dynamic correlations between adjacent frames, allowing for effect speech enhancement with less computational power required (pg. 4, section 4 “Conclusion”).
Hsiung in view of Strimel and Zhao does not specifically disclose the second model is an RNNoise model.
Valin teaches that the second model is an RNNoise model (Fig. 3, RNNoise architecture; Abstract “…In this paper, we demonstrate a hybrid DSP/deep learning approach to noise suppression… ”).
Hsiung, Strimel, Zhao, and Valin are considered to be analogous to the claimed invention as they are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hsiung in view of Strimel and Zhao to incorporate the teachings of Valin in order to specifically have the second model be a RNNoise model. Doing so would be beneficial, as such architecture achieves significantly higher quality while keeping low complexity enough for real-time operation (Abstract).
Regarding claim 17, claim 17 is rejected for analogous reasons to claim 4.
5. Claims 6-8 and 19-21 are rejected under 35 U.S.C. 103 as being unpatentable over Hsiung in view of Strimel, and further in view of Valin.
Regarding claim 6, Hsiung in view of Strimel discloses wherein the second model comprises a …simplified neural network model (para. 0034 “In the second-stage noise suppression device 160, the recurrent neural network 162 and the recurrent neural network 164 include long short-term memory (LSTM) layers 1620a-1620m and long short-term memory (LSTM) layers 1640a-1640m, respectively, and generate outputs through a fully connected layer of an activation function (e.g. sigmoid function, hyperbolic tangent function or linear function) circuit 1625 and a fully connected layer of a output circuit 1645, respectively.”). However, Hsiung in view of Strimel does not specifically disclose wherein the second model comprises a digital signal processing model.
Valin teaches wherein the second model comprises a digital signal processing model (Fig. 2, Feature Extraction block; pg. 2, section C “It only makes sense for the input of the network to include the log spectrum of the noisy signal based on the same bands used for the output. To improve the conditioning of the training data, we apply a DCT on the log spectrum, which results in 22 Bark-frequency cepstral coefficients (BFCC). In addition to these, we also include the temporal derivative and the second temporal derivative of the first 6 BFCCs. Since we already need to compute the pitch in (5), we compute the DCT of the pitch correlation across frequency bands and include the first 6 coefficients in our set of features. At last, we include the pitch period as well as a spectral non-stationarity metric that can help in speech detection. In total we use 42 input features…”).
Hsiung, Strimel, and Valin are considered to be analogous to the claimed invention as they are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hsiung in view of Strimel to incorporate the teachings of Valin in order to specifically have the second model comprise a digital signal processing model. Doing so would be beneficial, as this extract features from the audio used for subsequently processing via machine learning models for speech enhancement.
Regarding claim 7, Hsiung in view of Strimel and Valin discloses wherein the second model comprises a power spectrum estimation model (Valin, Fig. 2, Feature Extraction block; pg. 2, section C “It only makes sense for the input of the network to include the log spectrum of the noisy signal based on the same bands used for the output. To improve the conditioning of the training data, we apply a DCT on the log spectrum, which results in 22 Bark-frequency cepstral coefficients (BFCC). In addition to these, we also include the temporal derivative and the second temporal derivative of the first 6 BFCCs. Since we already need to compute the pitch in (5), we compute the DCT of the pitch correlation across frequency bands and include the first 6 coefficients in our set of features. At last, we include the pitch period as well as a spectral non-stationarity metric that can help in speech detection. In total we use 42 input features…”) and a fully connected layer (Hsiung, para. 0034 “In the second-stage noise suppression device 160, the recurrent neural network 162 and the recurrent neural network 164 include long short-term memory (LSTM) layers 1620a-1620m and long short-term memory (LSTM) layers 1640a-1640m, respectively, and generate outputs through a fully connected layer of an activation function (e.g. sigmoid function, hyperbolic tangent function or linear function) circuit 1625 and a fully connected layer of a output circuit 1645, respectively.”).
Hsiung, Strimel, and Valin are considered to be analogous to the claimed invention as they are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hsiung in view of Strimel to incorporate the teachings of Valin in order to specifically have the second model comprise a power spectrum estimation model. Doing so would be beneficial, given the same rationale as for the claim 6.
Regarding claim 8, Hsiung in view of Strimel and Valin discloses wherein inputting the second audio data and the first output result into the second model for processing to obtain the second output result comprises: inputting the second audio data into the digital signal processing model for processing to obtain an intermediate output result (Valin, Fig. 2, Feature Extraction block; pg. 2, section C “It only makes sense for the input of the network to include the log spectrum of the noisy signal based on the same bands used for the output. To improve the conditioning of the training data, we apply a DCT on the log spectrum, which results in 22 Bark-frequency cepstral coefficients (BFCC). In addition to these, we also include the temporal derivative and the second temporal derivative of the first 6 BFCCs. Since we already need to compute the pitch in (5), we compute the DCT of the pitch correlation across frequency bands and include the first 6 coefficients in our set of features. At last, we include the pitch period as well as a spectral non-stationarity metric that can help in speech detection. In total we use 42 input features…”); and inputting the intermediate output result and the first output result to the simplified neural network model for processing to obtain the second output result (Hsiung teaches inputting of intermediate output result (‘AS’) and a first output result (output of first model 162 (‘VD’)) into a neural network model (164) to obtain a second output result (‘NS’)).
Hsiung, Strimel, and Valin are considered to be analogous to the claimed invention as they are in the same field of speech processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hsiung in view of Strimel to incorporate the teachings of Valin in order to specifically have the second audio data be input to a digital signal processing model for processing to obtain an intermediate output result. Doing so would be beneficial, given the same rationale as for the claim 6.
Regarding claim 19, claim 19 is rejected for analogous reasons to claim 6.
Regarding claim 20, claim 20 is rejected for analogous reasons to claim 7.
Regarding claim 21, claim 21 is rejected for analogous reasons to claim 8.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Lee et al. (US 11,790,926): multi-stage speech enhancement via a plurality of neural network models (Fig. 1)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CODY DOUGLAS HUTCHESON whose telephone number is (703)756-1601. The examiner can normally be reached M-F 8:00AM-5:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at (571)-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CODY DOUGLAS HUTCHESON/Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659
