Detailed Action
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . See 35 U.S.C. § 100 (note).
Art Rejections
Anticipation
The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 11 and 15 are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Andong Li et al., Learning to Inference with Early Exit in the Progressive Speech Enhancement, 29th European Signal Processing Conf. 466 (2021) (“Li”).
Claim 1 is drawn to “an audio device.” The following table illustrates the correspondence between the claimed device and the Li reference.
Claim 1
The Li Reference
“1. An audio device comprising:
The Li reference similarly describes a method and device to enhance speech by extracting target speech from noise in noisy mixtures. Li at § I, ¶ 1.
“an audio enhancement module comprising a first neural network with first model layers including a first input layer, a plurality of first intermediate layers, and a first output layer; and
Li’s device includes a deep neural network (DNN) corresponding to the claimed audio enhancement module and first neural network. Id. One of ordinary skill in the art would have understood that a DNN, by definition, includes at least three layers: an input layer (e.g., STFT feature extraction), a hidden layer (e.g., inference layers) and an output layer (e.g., output of speech). See id. at § II(A), § II(C), FIG.1. Further, Li’s device includes an early exit mechanism (EEM), or first exit module, that allows for early exiting from one of the hidden layers, indicating that Li’s DNN includes a plurality of intermediate layers. Id. at § I, ¶ 3, § II(B), ¶ 1, FIG.1.
“a first exit module;
As discussed above, Li’s device includes an EEM corresponding to the claimed first exit module. Id.
“wherein the audio enhancement module is configured to process an audio input signal for provision of an audio output signal using the first neural network, and
Li’s DNN similarly processes speech audio to, for example, separate speech from noise in a noisy mix. Id. at § I, ¶ 1.
“wherein at least one of the first intermediate layers has an exit possibility for providing an intermediate layer output, and
Li’s DNN intermediate layers likewise include an exit possibility. Id. at § I, ¶ 3, § II(B), ¶ 1, FIG.1.
“wherein the first exit module is configured to determine whether the intermediate layer output satisfies a first criterion,
“wherein the first criterion is indicative of a performance, a quality, and/or an efficiency of the intermediate layer output, and
“wherein in accordance with the intermediate layer output satisfying the first criterion, the audio device is configured to determine the audio output signal based on the intermediate layer output.”
The EEM chooses a layer for early exit, and as the basis for a speech enhancement output, when the layer’s inference satisfies a threshold corresponding to performance and/or efficiency. Id. at § II(B), FIG.1. In particular, the EEM calculates a value
D
i
s
t
corresponding to the spectral distance between the outputs of two adjacent layers. Id. When the distance is less than a threshold
τ
, the current layer is chosen for output of enhanced speech. Id.
Table 1
For the foregoing reasons, the Li reference anticipates all limitations of the claim.
Claim 11 is drawn to “a method.” The following table illustrates the correspondence between the claimed method and the Miccini reference.
Claim 11
The Miccini Reference
“11. A method, performed by an audio device, for enabling efficient neural network processing, wherein the audio device comprises an audio enhancement module comprising a first neural network with first model layers including a first input layer, a plurality of first intermediate model layers, and a first output layer; and a first exit module, wherein the method comprises:
The Li reference similarly describes a method and device to enhance speech by extracting target speech from noise in noisy mixtures. Li at § I, ¶ 1. The elements of the audio device are addressed in the anticipation rejection of claim 1, incorporated herein.
“processing an audio input signal for provision of an audio output signal using the first neural network,
Li’s DNN similarly processes speech audio to, for example, separate speech from noise in a noisy mix. Id. at § I, ¶ 1.
“wherein at least one of the first intermediate layers has an exit possibility for providing an intermediate layer output,
Li’s DNN intermediate layers likewise include an exit possibility. Id. at § I, ¶ 3, § II(B), ¶ 1, FIG.1.
“determining, using the first exit module, whether the intermediate layer output satisfies a first criterion, wherein the first criterion is indicative of a performance, a quality, and/or an efficiency of an intermediate layer output, and
“in accordance with the intermediate layer output satisfying the first criterion, determining the audio output signal based on the intermediate layer output.”
The EEM chooses a layer for early exit, and as the basis for a speech enhancement output, when the layer’s inference satisfies a threshold corresponding to performance and/or efficiency. Id. at § II(B), FIG.1. In particular, the EEM calculates a value
D
i
s
t
corresponding to the spectral distance between the outputs of two adjacent layers. Id. When the distance is less than a threshold
τ
, the current layer is chosen for output of enhanced speech. Id.
Table 2
For the foregoing reasons, the Li reference anticipates all limitations of the claim.
Claim 15 depends on claim 1, and further requires the following:
“A computer-implemented method for training the first neural network of claim 1, wherein the method comprises:
“obtaining an audio dataset comprising one or more audio signals; and
“training, based on the audio dataset, the first neural network to perform an audio processing task at [sic “at”] least one of the first intermediate layers of the first neural network for provision of a first intermediate layer having an exit possibility for provision of an intermediate layer output.”
Similarly, the Li reference uses a trained DNN that has multiple intermediate layers, each of which may be used as an early exit. Li at § I. Li trains the DNN with multiple audio signals. See id. at § II(A), (D), § III(A) (discussing training of the DNN with a dataset of speech utterances). For the foregoing reasons, the Li reference anticipates all limitations of the claim.
Obviousness
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2–10 and 12–14 are rejected under 35 U.S.C. § 103 as being unpatentable over Li and US Patent Application Publication 2022/0398451 (published 15 December 2022) (“Surya”).
Claim 2 depends on claim 1, and further requires the following:
“wherein the audio device comprises a second exit module configured to obtain one or more features of the audio input signal including a first feature, and
“wherein the second exit module is configured to predict, based on the first feature, which first predicted layer of the first model layers to exit from when processing the audio input signal,
“wherein the first predicted layer is configured to provide a first predicted layer output, and
“wherein the audio device is configured to determine the audio output signal based on the first predicted layer output.”
The Li reference describes a device and method for enhancing speech with a trained deep neural network (DNN) having an early exit mechanism (EEM). The EEM corresponds to the claimed first exit module because it decides whether it is appropriate to exit the DNN processing from an intermediate layer. Li’s EEM makes the determination by calculating a spectral difference between the output of two adjacent layers and exiting when the difference is less than a threshold. However, Li does not further describe a second exit module to predict from audio input signal features a first layer from which an exit may be made.
The Surya reference is related to the Li reference because they both are drawn to DNN with early exiting. Li at § I; Surya at Abs. Surya teaches a combination EEM that combines a static agent 426 and a dynamic agent to determine when to exit. Surya at ¶¶ 53–57, FIG.4A. Static agent 426 receives input features 402 and predicts a minimum number of layers to use prior to engaging in an exit analysis. Id. The dynamic agent (not depicted), which is either used in combination with static agent 426 or by itself, functions like the claimed first exit module by determining whether to exit after each layer.
Read in light of the Li device, the Surya reference reasonably teaches improving the efficiency of the Li device by including a static agent, or second exit module, that analyzes Li’s input audio features (e.g., STFT of a noisy speech mixture) to predict a minimum number of layers to process prior to performing Li’s exit verification analysis. And as suggested by Surya, the static agent, or second exit module, would operate in conjunction with a dynamic agent, or first exit module. For the foregoing reasons, the combination of the Li and the Surya references makes obvious all limitations of the claim.
Claim 3 depends on claim 2, and further requires the following:
“wherein the first predicted layer is an intermediate layer of the one or more first intermediate layers.”
The obviousness rejection of claim 2, incorporated herein, shows the obviousness of modifying Li’s device and method to include Surya’s static agent 426, or second exit module, in order to predict a minimum number of layers (i.e., a first predicted layer) to process prior to determining an early exit point. Cf. Surya at ¶¶ 53–57, FIG.4A. In this regard, because the prior art combination predicts an early exit layer, the predicted layer is an intermediate layer among a plurality of intermediate layers. See Li at FIG.2 (depicting a number of intermediate layers that may be used as an early exit). For the foregoing reasons, the combination of the Li and the Surya references makes obvious all limitations of the claim.
Claim 4 depends on claim 2, and further requires the following:
“wherein the second exit module is configured to obtain one or more audio device parameters including a first audio device parameter, and
“wherein the second exit module is configured to predict, based on the first audio device parameter, which second predicted layer of the first model layers to exit from when processing the audio input signal,
“wherein the second predicted layer is configured to provide a second predicted layer output, and
“wherein the audio device is configured to determine the audio output signal based on the second predicted layer output.”
The obviousness rejection of claim 2, incorporated herein, shows the obviousness of modifying Li’s device and method to include Surya’s static agent 426, or second exit module, in order to predict a minimum number of layers (i.e., a first predicted layer) to process prior to determining an early exit point. Cf. Surya at ¶¶ 53–57, FIG.4A. The Surya reference further teaches and suggests predicting a minimum number of layers based on audio device parameters. Id. at ¶ 55 (describing the use of more layers when more idle resources are available). For the foregoing reasons, the combination of the Li and the Surya references makes obvious all limitations of the claim.
Claim 5 depends on claim 4, and further requires the following:
“wherein the first audio device parameter is a power parameter, a battery parameter, and/or a processing capability parameter, and
“wherein the second exit module is configured to predict, based on the power parameter, the battery parameter, and/or the processing capability parameter, which second predicted layer of the first neural network to exit from when processing the audio input signal.”
The Surya reference further teaches and suggests predicting a minimum number of layers based on audio device parameters. Surya at ¶ 55 (describing the use of more layers when more idle resources, or processing capability parameter, are available). For the foregoing reasons, the combination of the Li and the Surya references makes obvious all limitations of the claim.
Claim 6 depends on claim 2, and further requires the following:
“wherein the second exit module is configured to predict, based on the audio input signal, which third predicted layer of the first model layers at which a performance, a quality, and/or an efficiency of processing of the audio input signal converges, and
“wherein the audio device is configured to determine the audio output signal based on the third predicted layer.”
The obviousness rejection of claim 2, incorporated herein, shows the obviousness of modifying Li’s device and method to include Surya’s static agent 426, or second exit module, in order to predict a minimum number of layers (i.e., a first predicted layer) to process prior to determining an early exit point. Cf. Surya at ¶¶ 53–57, FIG.4A. In particular, Surya describes training the static agent to predict based on an input query (i.e., Li’s STFT audio features) when an output exhibits a degree of confidence above a threshold (i.e., when the output converges). Id. at ¶¶ 5, 8, 9. Applying this teaching to Li’s device, one of ordinary skill would have reasonably trained the static agent to predict an output layer whose output converges on clean speech (i.e., quality) since the goal of Li’s device and method is to enhance speech by separating it from noise. See Li at § I, ¶ 1. For the foregoing reasons, the combination of the Li and the Surya references makes obvious all limitations of the claim.
Claim 7 depends on claim 6, and further requires the following:
“wherein to determine the audio output signal based on the prediction comprises to determine, based on the third predicted layer, which layer of the first model layers to exit from when processing the audio input signal.”
Claim 8 depends on claim 6, and further requires the following:
“wherein the third predicted layer is configured to provide a third predicted layer output, and
“wherein the audio device is configured to determine the audio output signal based on the third predicted layer output.”
Claims 7 and 8 are analyzed together. As explained in the obviousness rejection of claim 2, incorporated herein, it would have been obvious to modify Li’s device and method to similarly combine a static agent and dynamic agent. The static agent would predict a minimum number of layers to process before considering an early exit while the dynamic layer (e.g., Li’s EEM) would make a final determination of whether it is appropriate to exit. Cf. Surya at ¶¶ 53–57, FIG.4A. For the foregoing reasons, the combination of the Li and the Surya references makes obvious all limitations of the claim. And if the static layer’s predicted layer satisfies Li’s conditions, then Li’s device will use the predicted layer as the output layer. See id. For the foregoing reasons, the combination of the Li and the Surya references makes obvious all limitations of the claims.
Claim 9 depends on claim 6, and further requires the following:
“wherein the audio device is configured to determine the audio output signal based on an output of the layer before the third predicted layer.”
Regardless of which layer is selected as an early exit, each of Li’s intermediate layers is configured to be used as an early exit. See Li at § II(B), FIG.2. For the foregoing reasons, the combination of the Li and the Surya references makes obvious all limitations of the claim.
Claim 10 depends on claim 2, and further requires the following:
“wherein the first exit module comprises a second neural network, and
“wherein to determine whether the intermediate layer output satisfies a first criterion comprises to determine whether the intermediate layer output satisfies the first criterion using the second neural network.”
The Surya reference teaches and suggests providing a dynamic agent to determine whether a layer is appropriate as an early exit. Surya at ¶ 57. This reasonably suggests modifying Li’s approach by using a dynamic agent trained to consider numerous factors (e.g., spectral difference between adjacent layers, confidence and resources) to gauge whether a predicted layer is appropriate as an exit. For the foregoing reasons, the combination of the Li and the Surya references makes obvious all limitations of the claim.
Claim 12 depends on claim 11, and further requires the following:
“the method comprising:
“obtaining, using a second exit module, one or more features of the audio input signal including a first feature,
“predicting, based on the first feature, which first predicted layer of the first model layers to exit from when processing the audio input signal, wherein the first predicted layer is configured to provide a first predicted layer output, and
“determining the audio output signal based on the first predicted layer output.”
The Li reference describes a device and method for enhancing speech with a trained deep neural network (DNN) having an early exit mechanism (EEM). The EEM corresponds to the claimed first exit module because it decides whether it is appropriate to exit the DNN processing from an intermediate layer. Li’s EEM makes the determination by calculating a spectral difference between the output of two adjacent layers and exiting when the difference is less than a threshold. However, Li does not further describe a second exit module to predict from audio input signal features a first layer from which an exit may be made.
The Surya reference is related to the Li reference because they both are drawn to DNN with early exiting. Li at § I; Surya at Abs. Surya teaches a combination EEM that combines a static agent 426 and a dynamic agent to determine when to exit. Surya at ¶¶ 53–57, FIG.4A. Static agent 426 receives input features 402 and predicts a minimum number of layers to use prior to engaging in an exit analysis. Id. The dynamic agent (not depicted), which is either used in combination with static agent 426 or by itself, functions like the claimed first exit module by determining whether to exit after each layer.
Read in light of the Li device, the Surya reference reasonably teaches improving the efficiency of the Li device by including a static agent, or second exit module, that analyzes Li’s input audio features (e.g., STFT of a noisy speech mixture) to predict a minimum number of layers to process prior to performing Li’s exit verification analysis. And as suggested by Surya, the static agent, or second exit module, would operate in conjunction with a dynamic agent, or first exit module. For the foregoing reasons, the combination of the Li and the Surya references makes obvious all limitations of the claim.
Claim 13 depends on claim 12, and further requires the following:
“obtaining, using the second exit module, one or more audio device parameters including a first audio device parameter,
“predicting, using the second exit module and based on the first audio device parameter, which second predicted layer of the first model layers to exit from when processing the audio input signal, wherein the second predicted layer is configured to provide a second predicted layer output, and
“determining the audio output signal based on the second predicted layer output.”
The obviousness rejection of claim 12, incorporated herein, shows the obviousness of modifying Li’s device and method to include Surya’s static agent 426, or second exit module, in order to predict a minimum number of layers (i.e., a first predicted layer) to process prior to determining an early exit point. Cf. Surya at ¶¶ 53–57, FIG.4A. The Surya reference further teaches and suggests predicting a minimum number of layers based on audio device parameters. Id. at ¶ 55 (describing the use of more layers when more idle resources are available). For the foregoing reasons, the combination of the Li and the Surya references makes obvious all limitations of the claim.
Claim 14 depends on claim 13, and further requires the following:
“wherein predicting which second predicted layer of the first model layers to exit from comprises predicting, based on the power parameter, the battery parameter, and/or the processing capability parameter, which second predicted layer of the first neural network to exit from when processing the audio input signal.”
The Surya reference further teaches and suggests predicting a minimum number of layers based on audio device parameters. Surya at ¶ 55 (describing the use of more layers when more idle resources, or processing capability parameter, are available). For the foregoing reasons, the combination of the Li and the Surya references makes obvious all limitations of the claim.
Summary
Claims 1–15 are rejected under 35 U.S.C. §§ 102 and 103 as being unpatentable over the cited prior art. In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 C.F.R. § 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
Additional Citations
The following table includes additional documents that were identified during a search of the subject matter disclosed and claimed in this Application. While this Office action does not rely on these documents, they are relevant and should be reviewed in responding to this Office action.
Citation
Relevance
US 2023/0127001
Early exiting based on performance
Table 3
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WALTER F BRINEY III whose telephone number is (571)272-7513. The examiner can normally be reached M-F 8 am-4:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Carolyn Edwards can be reached at 571-270-7136. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Walter F Briney III/
Walter F Briney IIIPrimary ExaminerArt Unit 2692
6/11/2026
Prosecution Insights