DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant amended claim 31 on the claims filed on 01/15/2026 by canceling the portion of claim 31 that caused claim objection. Considering the given amendment, the previous claim objection on claim 31 does not apply anymore.
Applicant has not responded to the claim interpretations - 35 USC § 112(f) with respect to the claim interpretation on claim 42, though the Applicant seems to think that they have already made changes, for instance, “apparatus …for” is removed from claim 42. Thus, it looks like the Examiner’s argument has been addressed and therefore the 35 USC § 112(f) claim interpretation has been withdrawn.
Applicant’s arguments, see Arguments pages 16-17, filed on 01/15/2026, with respect to the 35 USC § 101 abstract rejections have been fully considered but they are not persuasive. The 35 USC § 101 rejections of claims 1-17 and 29-47 has been kept. By amending the independent claims 1, 46 and 47 taking the claim languages from claim 42 in which the amended claim is directing the subject-matter to an apparatus that is configured to modify the audio signal depending on the estimate of the loudness of the signal components of interest of the audio signal to acquire an audio output signal, as well as adding a limitation that states “and wherein the apparatus is implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer”, the Applicant asserts that the claims can overcome the 101 rejection (Arguments, page 17).
The Examiner respectfully disagrees. Mere implementation using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer does not overcome the 101 rejection, because implementation with such a generalized computer can be in itself an abstract idea (Spec., Page 26, line 15-19). Moreover, a human can mentally modify the audio input signal depending on the estimate of the loudness of the speech components of the audio input signal to acquire the audio output signal. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, the 101 claim rejections of independent claims 1, 46 and 47 will be still be valid.
Applicant’s arguments, see Arguments pages 17-23, filed on 01/15/2026, with
respect to the 35 USC § 102 and 35 USC § 103 rejections of claims 1-47 have been fully considered and but they are not persuasive. The Applicant has not argued against the secondary reference of Paulus which was used to teach the limitations newly added.
The Applicant argues that the amendment on claim 1 addresses the 35 USC § 101 abstract idea rejections and therefore the rejection has to be withdrawn. The Applicant asserts that responding to the 101 rejection, the Applicant has amended claim 1 to recite the further features that the apparatus is configured to modify the audio signal depending on the estimate of the loudness of the signal components of interest of the audio signal to acquire an audio output signal. The Applicant has also clearly stated that the amendment has been obtained from claim 42. The Applicant further states that similar amendments has been made on independent claims 46 and 47 (Arguments, page 16-17).
The Examiner respectfully disagrees. An important point that needs to be noted is that the amendment idea on claim 1 has been obtained from claim 42 while claim 42 itself has been rejected with the 35 USC § 101 abstract idea rejection. The similar amendments on independent claims 46 and 47 do not overcome the 35 USC § 101 rejection. As mentioned above, mere implementation using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer does not really overcome the 101 rejection, because an implementation with such a generalized computer can in itself be an abstract idea (Spec., Page 26, line 15-19). Considering the other limitation added to independent claims 1, 46 and 47, a human can mentally modify the audio input signal depending on the estimate of the loudness of the speech components of the audio input signal to acquire the audio output signal. The claimed invention is, thus, directed to an abstract idea and a mental process without significantly more and thus, the 101 claim rejections of independent claims 1, 46 and 47 will be still be under 101 rejection.
The Applicant argues that the cited prior art Meléndez does not disclose the feature of claim 1 that an influence of the other signal components on the estimate of the loudness of the signal components of interest is reduced or not present (Arguments, page 18).
The Examiner respectfully disagrees. Meléndez teaches about the influence of other signal components on the estimate of the loudness of the signal components of interest because, for instance, it discloses the first approach to music and speech detection using deep learning architectures in which the signal components of interest of the audio signal are speech components of the audio signal. Meléndez, in Figure 2 also displays a simplified representation of a TCN model where yt ∈ RNtcn is the output of the last residual block for that time-frame wherein the neural network can be configured to determine one output value from the plurality of input values such that the one output value indicates the estimate of the loudness of the speech components of the audio signal (Meléndez, page 274, 1st col, 2nd para and page 275, 1st col, 4th para).
In the limitation of claim 2 of “the audio signal simultaneously comprises the signal components of interest and other signal components …”, “an influence of the other signal components on the estimate of the loudness of the signal components of interest is reduced or not present …”, was drafted cover mental activity. More specifically, for claim 2, a human recognizes that audio signal simultaneously comprises the signal components of interest and other signal components. The claimed invention is, therefore, directed to an abstract idea, a mental process without significantly more and thus, claims 2 is rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-17 and 29-47 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Independent claims 1, 46, and 47 relate to the statutory category of method/process and machine/apparatus. The independent claim 1 recites “an apparatus for providing an estimate of a loudness of signal components of interest of an audio signal, wherein the apparatus comprises: an input interface configured to receive a plurality of samples of the audio signal, and a neural network configured to receive as input values the plurality of samples of the audio signal or a plurality of derived values being derived from the plurality of samples of the audio signal, and configured to determine at least one output value from the plurality of input values, such that the at least one output value indicates the estimate of the loudness of the signal components of interest of the audio… wherein the method comprises modifying the audio signal depending on the estimate of the loudness of the signal components of interest of the audio signal to acquire an audio output signal, and wherein the method is implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.” The limitation of claim 1 of “receiving a plurality of samples…”, “estimating the loudness…”, “determine at least one output…” as drafted cover mental activity and data gathering. More specifically, for claim 1 a human can collect/receive samples of data and make an estimation of the loudness. Mere implementation using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer does not really overcome the 101 rejection, because an implementation with such a generalized computer can in itself be an abstract idea (Spec., Page 26, line 15-19). Considering the other limitation added to independent claims 1, 46 and 47, a human can mentally modify the audio input signal depending on the estimate of the loudness of the speech components of the audio input signal to acquire the audio output signal. The claimed invention is, thus, directed to an abstract idea and a mental process without significantly more and thus, the 101 claim rejections of independent claims 1, 46 and 47. The claimed invention is, therefore, directed to an abstract idea, a mental process and data gathering without significantly more and thus, claims 1, 46 and 47 are rejected under 35 U.S.C. 101.
Claim 42 recites “an apparatus according to claim 1 a signal processor configured to modify the audio input signal depending on the estimate of the loudness of the signal components of interest of the audio input signal to acquire the audio output signal, .” The limitation of claim 42 “…”, “a signal processor configured to modify the audio input signal depending on the estimate of the loudness of the signal components of interest of the audio input signal to acquire the audio output signal …” as drafted cover mental activity while the “signal processor” is just an additional element. More specifically, for claim 42, a human can mentally apply modification of an audio input signal to acquire an audio output signal for providing an estimate of a loudness of signal components of interest of the audio input signal. For instance, based on the available audio data collected using a simple audio data collection device such as audio data recorder, in an excel data sheet in a device or any generic computer, etc., one can easily apply some mathematical formula to modify the audio data to estimate the loudness. The claimed invention is, therefore, directed to an abstract idea, a mental process and data gathering without significantly more and thus, claim 42 is rejected under 35 U.S.C. 101.
The independent claim 46 recites “receiving a plurality of samples of the audio signal, andwherein the method comprises modifying the audio signal depending on the estimate of the loudness of the signal components of interest of the audio signal to acquire an audio output signal, and wherein the method is implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.” The limitation of claim 46 of “receiving a plurality of samples…”, “estimating the loudness…”, “determine at least one output…” as drafted cover mental activity and data gathering. More specifically, for claim 46 a human can collect/receive samples of data and make an estimation of the loudness. Just modifying the audio signal depending on the estimate of the loudness of the signal components of interest of the audio signal to acquire an audio output signal, and mere implementation using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer does not really overcome the 101 rejection, because an implementation with such a generalized computer can in itself be an abstract idea (Spec., Page 26, line 15-19). Considering the other limitation added to independent claims 1, 46 and 47, a human can mentally modify the audio input signal depending on the estimate of the loudness of the speech components of the audio input signal to acquire the audio output signal. The claimed invention is, thus, directed to an abstract idea and a mental process without significantly more. The claimed invention is, therefore, directed to an abstract idea, a mental process and data gathering without significantly more and thus, claims 46 are rejected under 35 U.S.C. 101.
The independent claim 47 recites “receiving a plurality of samples of the audio signal, and wherein the method comprises modifying the audio signal depending on the estimate of the loudness of the signal components of interest of the audio signal to acquire an audio output signal.” The limitation of claim 47 of “receiving a plurality of samples…”, “estimating the loudness…”, “determine at least one output…” as drafted cover mental activity and data gathering. Just modifying the audio signal depending on the estimate of the loudness of the signal components of interest of the audio signal to acquire an audio output signal, and a human can mentally modify the audio input signal depending on the estimate of the loudness of the speech components of the audio input signal to acquire the audio output signal. The claimed invention is, thus, directed to an abstract idea and a mental process without significantly more. More specifically, for claim 6 a human can collect/receive samples of data and make an estimation of the loudness. The claimed invention is, therefore, directed to an abstract idea, a mental process and data gathering without significantly more and thus, claims 47 are rejected under 35 U.S.C. 101.
Claim 2 recites “the audio signal simultaneously comprises the signal components of interest and other signal components of the audio signal, wherein an influence of the other signal components on the estimate of the loudness of the signal components of interest is reduced or not present.…” The limitation of claim 2 of “the audio signal simultaneously comprises the signal components of interest and other signal components …”, “an influence of the other signal components on the estimate of the loudness of the signal components of interest is reduced or not present …”, as drafted cover mental activity. More specifically, for claim 2, a human recognizes that audio signal simultaneously comprises the signal components of interest and other signal components. The claimed invention is, therefore, directed to an abstract idea, a mental process without significantly more and thus, claims 2 is rejected under 35 U.S.C. 101.
Claim 3 recites “wherein the signal components of interest of the audio signal are speech components of the audio signal, and wherein the neural network is configured to determine the at least one output value from the plurality of input values, such that the at least one output value indicates the estimate of the loudness of the speech components of the audio signal…” The limitation of claim 3 of “the signal components of interest of the audio signal are speech …” as drafted cover mental activity. More specifically, for claim 3, a human can mentally recognize if the signal components of interest of the audio signal has speech in it. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 3 is rejected under 35 U.S.C. 101.
Claim 4 recites “wherein the audio signal simultaneously comprises the speech components and background components of the audio signal, wherein an influence of the background components on the estimate of the loudness of the speech components is reduced or not present.” The limitation of claim 4 of “audio signal simultaneously comprises the speech components and background components…”, “an influence of the background components on the estimate of the loudness of the speech components is reduced or not present …” as drafted cover mental activity. More specifically, for claim 4, a human mentally recognize audio signal simultaneously comprising the speech components and background components. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 4 is rejected under 35 U.S.C. 101.
Claim 5 recites “wherein the signal components of interest of the audio signal are sound components of at least one first sound source out of a plurality of sound sources in an environment, wherein the audio signal simultaneously comprises the sound components of the at least one first sound source and other sound components of one or more other sound sources out of the plurality of sound sources in the environment, wherein the neural network is configured to determine the at least one output value from the plurality of input values, such that the at least one output value indicates the estimate of the loudness of the sound components of the at least one first sound source, wherein an influence of the other sound components of the one or more other sound sources on the estimate of the loudness of the sound components of the at least one first sound source is reduced or not present…” The limitation of claim 5 of “the signal components of interest of the audio signal are sound components of at least one first sound source out of a plurality of sound sources in an environment …”, “wherein the audio signal simultaneously comprises the sound components of the at least one first sound source and other sound components of one or more other sound sources out of the plurality of sound sources in the environment …”, “ an influence of the other sound components of the one or more other sound sources on the estimate of the loudness of the sound components of the at least one first sound source is reduced or not presents …” as drafted cover mental activity. More specifically, for claim 5, a human can mentally recognize the simultaneous coexistence of the sound components of at least one first sound source and other sound components of one or more other sound sources out of the plurality of sound sources in the environment, and a human can mentally determine/estimate how much the influence of the sound components of the different sound sources on the estimate of the loudness of the sound components of the other sound sources. The claimed invention is, therefore, directed to an abstract idea, a mental process and data gathering without significantly more and thus, claim 5 is rejected under 35 U.S.C. 101.
Claim 6 recites “wherein the neural network is configured to determine the at least one output value from the plurality of input values, such that the at least one output value indicates the estimate of the loudness of the speech components of the first person, wherein an influence of the other speech components of the one or more other persons on the estimate of the loudness of the speech components of the first person is reduced or not present, wherein the sound components of the at least one first sound source are speech components of a first person out of a plurality of persons speaking in the environment, wherein the other sound components of the one or more other sound sources are other speech components of one or more other persons out of the plurality of persons speaking in the environment, wherein the audio signal simultaneously comprises the speech components of the first person and the other speech components of the one or more other persons speaking in the environment.” The limitation of claim 6 of “wherein the neural network is configured to determine the at least one output value …”, “at least one output value indicates the estimate of the loudness of the speech components of the first person …”, “wherein the sound components of the at least one first sound source are speech components of a first person out of a plurality of persons speaking in the environment …” , “wherein the audio signal simultaneously comprises the speech components of the first person and the other speech components of the one or more other persons speaking in the environment …” as drafted cover mental activity. More specifically, for claim 6, a human can, based on data previously collected using pen and pencil, make estimate of the loudness of the speech of the first person and this can be implemented with a general purpose (generic) neural network. The claimed invention is, therefore, directed to an abstract idea, a mental process and data gathering without significantly more and thus, claim 6 is rejected under 35 U.S.C. 101.
Claim 7 recites “wherein the sound components of the at least first sound source are sound components of at least one non-human sound source out of a plurality of non-human sound sources in an environment, wherein the other sound components of the one or more other sound sources are other sound components of one or more other non-human sound source out of the plurality of non-human sound sources, wherein the audio signal simultaneously comprises the sound components of the at least one first non-human sound source and the other sound components of the one or more other non-human sound sources in the environment, wherein an influence of the other sound components of the one or more other non-human sound sources on the estimate of the loudness of the sound components of the at least one first non-human sound source is reduced or not present, wherein the neural network is configured to determine the at least one output value from the plurality of input values, such that the at least one output value indicates the estimate of the loudness of the sound components of the at least one first non-human sound source.” The limitation of claim 7 “wherein the sound components of the at least first sound source are sound components of at least one non-human sound source out of a plurality of non-human sound sources in an environment …”, “wherein the audio signal simultaneously comprises the sound components of the at least one first non-human sound source and the other sound components of the one or more other non-human sound sources in the environment …”, “wherein an influence of the other sound components of the one or more other non-human sound sources on the estimate of the loudness of the sound components of the at least one first non-human sound source is reduced or not present …”, “wherein the neural network is configured to determine the at least one output value from the plurality of input values, such that the at least one output value indicates the estimate of the loudness of the sound components of the at least one first non-human sound source …” as drafted cover mental activity and data gathering. More specifically, for claim 7, a human can mentally determine that the sound components of the at least first sound source are sound components of at least one non-human sound source out of a plurality of non-human sound sources in an environment, a human can also mentally determine that the audio signal simultaneously comprises the sound components of the at least one first non-human sound source and the other sound components of the one or more other non-human sound sources in the environment, and also a human can, based on data previously collected using pen and pencil, determine if an influence of the other sound components of the one or more other non-human sound sources on the estimate of the loudness of the sound components of the at least one first non-human sound source is reduced or not present, and this can be implemented with a general purpose (generic) neural network The claimed invention is, therefore, directed to an abstract idea, a mental process and data gathering without significantly more and thus, claim 7 is rejected under 35 U.S.C. 101.
Claim 8 recites “wherein the sound components of the at least one first sound source is a singing of one or more singers in the environment, wherein the other sound components of the one or more other sound sources are sound components of accompanying musical instruments, which accompany the singing of the one or more singers in the environment, wherein the audio signal simultaneously comprises the signing of the one or more singers and the sound components of the accompanying musical instruments, wherein the neural network is configured to determine the at least one output value from the plurality of input values, such that the at least one output value indicates the estimate of the loudness of the singing, wherein an influence of the sound components of accompanying musical instruments on the estimate of the loudness of the singing is reduced or not present” The limitation of claim 8 “the sound components of the at least one first sound source is a singing of one or more singers in the environment, …”, “wherein the other sound components of the one or more other sound sources are sound components of accompanying musical instruments, …”, “which accompany the singing of the one or more singers in the environment,…” as drafted covers a mental activity. More specifically, for claim 8, a human can mentally determine if sound components of the at least one first sound source is a singing of one or more singers in the environment, wherein the other sound components of the one or more other sound sources are sound components of accompanying musical instruments, which accompany the singing of the one or more singers in the environment, wherein the audio signal simultaneously comprises the signing of the one or more singers and the sound components of the accompanying musical instruments and if there is some influence of the sound components of accompanying musical instruments on the estimate of the loudness of the singing is reduced or not present. The claimed invention is, therefore, directed to an abstract idea, a mental process without significantly more and thus, claim 8 is rejected under 35 U.S.C. 101.
Claim 9 recites “wherein the neural network is configured to determine at least one further output value indicating an estimate of a loudness of the entire audio signal.” The limitation of claim 9 “wherein the neural network is configured to determine at least one further output value … estimate of a loudness of the entire audio signal…” as drafted covers a mental activity. More specifically, for claim 9, a human can mentally make an estimation of the loudness, with or without a general neural network. For instance, a generalized neural network and training is provided in the Spec (Spec. page 22, lines 1-14). The claimed invention is, therefore, directed to an abstract idea, a mental process without significantly more and thus, claim 9 is rejected under 35 U.S.C. 101.
Claim 10 recites “wherein the neural network is configured to determine one or more further output values indicating an estimate of a loudness of the audio signal when speech is present.” The limitation of claim 10 “determine one or more further output values indicating an estimate of a loudness of the audio signal when speech is present” as drafted cover mental activity. More specifically, for claim 10, a human can mentally estimate a loudness of the audio signal when speech is present. The generalized neural network is just an additional element, as specified above, and it doesn’t integrate the abstract idea into a practical application. The claimed invention is, therefore, directed to an abstract idea, a mental process and data gathering without significantly therefore, directed to an abstract idea, a mental process without significantly more and thus, claim 10 is rejected under 35 U.S.C. 101.
Claim 11 recites “wherein the neural network is configured to determine another one or more output values indicating an estimate of a loudness of background components of the audio signal.” The limitation of claim 11 of “determine another one or more output values indicating an estimate of a loudness of background components of the audio signal …” as drafted cover mental activity. The generalized neural network is just an additional element, as indicated above, and it doesn’t integrate the abstract idea into a practical application. More specifically, for claim 11, a human can mentally determine another one or more output values and an estimate of a loudness of background components of the audio signal. The claimed invention is, therefore, directed to an abstract idea, a mental process without significantly more and thus, claim 11 is rejected under 35 U.S.C. 101.
Claim 12 recites “ wherein the apparatus is configured to determine and output at least one other output value indicating an estimate of a partial loudness of the speech components of the audio signal, wherein the partial loudness of the speech components of the audio signal depends on the loudness of the speech components of the audio signal and on the loudness of background components of the audio signal.” The limitation of claim 12 “the apparatus is configured to determine and output at least one other output value indicating an estimate of a partial loudness of the speech components of the audio signal …”, “the partial loudness of the speech components of the audio signal depends on the loudness of the speech components of the audio signal and on the loudness of background components of the audio signal …” as drafted cover mental activity. More specifically, for claim 12, a human can mentally estimate a partial loudness of the speech components of the audio signal. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 12 is rejected under 35 U.S.C. 101.
Claim 13 recites “wherein the apparatus comprises a postprocessor, configured to modify the estimate of the loudness of the signal components of interest of the audio signal depending on confidence information, and/or configured to output the confidence information, wherein the confidence information indicates a reliability on whether or not the estimate of the loudness of the signal components of interest of the audio signal conducted by the neural network is reliable, or wherein the confidence information indicates one or more values indicating a degree of reliability of the estimate of the loudness of the signal components of interest of the audio signal conducted by the neural network.” The limitation of claim 13 “the apparatus comprises a postprocessor, configured to modify the estimate of the loudness of the signal components of interest of the audio signal depending on confidence information …”, “and/or configured to output the confidence information, wherein the confidence information indicates a reliability on whether or not the estimate of the loudness of the signal components of interest of the audio signal conducted by the neural network is reliable …” as drafted cover mental activity. The postprocessor is an additional element that doesn’t make the idea inventive, but rather this is just a general purpose element as it is provided in the Specification (Spec. page 11, line 25 – page 12, line 12). More specifically, for claim 13, a human can mentally determine the confidence level and then mentally modify the estimate of the loudness of the signal components of interest of the audio signal depending on confidence level. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 13 is rejected under 35 U.S.C. 101.
Claim 14 recites “wherein, if the at least one output value provided by the neural network indicates that the estimate of the loudness of the signal components of interest of the audio signal would be higher than a total loudness of the audio signal, wherein the postprocessor is configured to determine as the confidence information whether or not the at least one output value provided by the neural network indicates that the estimate of the loudness of the signal components of interest of the audio signal would be higher than a total loudness of the audio signal, and the postprocessor is configured to modify the estimate of the loudness of the signal components of interest such that the loudness of the signal components of interest of the audio signal is equal to the total loudness of the audio signal, or the postprocessor is configured to output the confidence information comprising an indication that the estimate of the loudness of the signal components of interest of the audio signal is not reliable.” The limitation of claim 14 “at least one output value provided by the neural network indicates that the estimate of the loudness of the signal components of interest of the audio signal would be higher than a total loudness of the audio signal …”, “the postprocessor is configured to determine as the confidence information whether or not the at least one output value provided by the neural network indicates that the estimate of the loudness of the signal components of interest of the audio signal would be higher than a total loudness of the audio signal …”, “…the postprocessor is configured to output the confidence information comprising an indication that the estimate of the loudness of the signal components of interest of the audio signal is not reliable …” as drafted covers a mental activity. More specifically, for claim 14, a human can mentally estimate and also modify their estimate of the loudness of the signal components of interest of the audio signal and also a human can mentally estimate or determine the confidence information, or with just a general neural network which is basically an additional element. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 14 is rejected under 35 U.S.C. 101.
Claim 15 recites “wherein the postprocessor is configured to determine and to output the confidence information comprising a confidence value that indicates the degree of reliability of the estimate of the loudness of the signal components of interest of the audio signal conducted by the neural network, such that the confidence value depends on the estimate of the loudness of the signal components of interest of the audio signal and further depends on a loudness or an estimate of a loudness of the other signal components of the audio signal.” The limitation of claim 15 “the postprocessor is configured to determine and to output the confidence information comprising a confidence value …”, “the confidence value depends on the estimate of the loudness of the signal components of interest of the audio signal …” as drafted covers a mental activity. The postprocessor is just an additional element, as indicated in the Spec. (Spec. page 11, line 25 – page 12, line 13), and this element doesn’t integrate the abstract idea into a practical application. More specifically, for claim 15, a human can mentally determine the output confidence information comprising a confidence value. For instance, one can estimate the confidence level to be confident (>60%) or very confident (75-80%) or even estimate at more confidence about the estimate of the loudness. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 15 is rejected under 35 U.S.C. 101.
Claim 16 recites “wherein the confidence value depends on a difference between the estimate of the loudness of the signal components of interest of the audio signal and the loudness or the estimate of the loudness of the other signal components of the audio signal, or wherein the confidence value depends on a ratio of the estimate of the loudness of the signal components of interest of the audio signal and the loudness or the estimate of the loudness of the other signal components of the audio signal.” The limitation of claim 16 “the confidence value depends on a difference between the estimate of the loudness of the signal components of interest of the audio signal and the loudness or the estimate of the loudness of the other signal components of the audio signal …”, “or … the confidence value depends on a ratio of the estimate of the loudness of the signal components of interest of the audio signal and the loudness or the estimate of the loudness of the other signal components of the audio signal …” as drafted cover mental activity. More specifically, for claim 16, a human can mentally estimate that the confidence value depends on a difference between the estimate of the loudness of the signal components of interest of the audio signal and the loudness or the estimate of the loudness of the other signal components of the audio signal or the confidence value depends on a ratio of the estimate of the loudness of the signal components of interest of the audio signal and the loudness or the estimate of the loudness of the other signal components of the audio signal. The claimed invention is, therefore, directed to an abstract idea, a mental process and data gathering without significantly more and thus, claim 16 is rejected under 35 U.S.C. 101.
Claim 17 recites “wherein the neural network has been trained using a plurality of data training items, wherein each of the plurality of data training items comprises one of a plurality of audio training signal portions and one or more reference loudness values.” The limitation of claim 17 “wherein the neural network has been trained using a plurality of data training items …”, “each of the plurality of data training items comprises one of a plurality of audio training signal portions and one or more reference loudness values …” as drafted cover mental activity. More specifically, for claim 17, a general neural network is cited that doesn’t show an inventive concept. For instance, a generalized neural network and training is provided in the Spec (Spec. page 22, lines 1-14). Specifically, the limitation that each of the plurality of data training items comprises one of a plurality of audio training signal portions and one or more reference loudness values is an abstract idea of mental process. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 17 is rejected under 35 U.S.C. 101.
Claim 29 recites “wherein the neural network comprises an input layer, two or more hidden layers, and an output layer, wherein the input layer comprises a plurality of input nodes, wherein each of the plurality of input nodes is configured to receive one of the plurality of input values; wherein each of the two or more hidden layers comprises one or more neural nodes, and wherein the output layer comprises at least one output node, wherein the at least one output node is configured to output the at least one output value indicating the estimate of the loudness of the signal components of interest of the audio signal.” The limitation of claim 29 “wherein the neural network comprises an input layer, two or more hidden layers, and an output layer …”, “the input layer comprises a plurality of input nodes, wherein each of the plurality of input nodes is configured to receive one of the plurality of input values …”, “each of the two or more hidden layers comprises one or more neural nodes, and wherein the output layer comprises at least one output node, wherein the at least one output node is configured to output the at least one output value indicating the estimate of the loudness of the signal components of interest of the audio signal …” as drafted cover mental activity. More specifically, for claim 29, any general neural network can comprise of an input layer, two or more hidden layers, and an output layer, wherein the input layer comprises a plurality of input nodes, wherein each of the plurality of input nodes is configured to receive one of the plurality of input values; wherein each of the two or more hidden layers comprises one or more neural nodes, and wherein the output layer comprises at least one output node, wherein the at least one output node is configured to output the at least one output value indicating the estimate of the loudness of the signal components of interest of the audio signal. For instance, a generalized neural network is provided in the Spec (Spec. page 22, lines 1-14). Besides, the output node can be considered just an additional element. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 29 is rejected under 35 U.S.C. 101.
Claim 30 recites “ wherein at least one layer of the two or more hidden layers is a convolutional layer.” The limitation of claim 30 “at least one layer of the two or more hidden layers is a convolutional layer …” as drafted cover just an additional limitation and additional element. More specifically, for claim 30, at least one layer of the two or more hidden layers can be a convolutional layer. A generalized neural network as provided in the Spec can be used to implement this limitation (Spec. page 22, lines 1-14).The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 30 is rejected under 35 U.S.C. 101.
Claim 31 recites “wherein the neural network is configured to employ a convolutional filter for the convolutional layer, which comprises a shape (x , y), with x = y or with x
≠
y, wherein max (x, y) ≤ 10.” The limitation of claim 31 “the neural network is configured to employ a convolutional filter for …”, “the convolutional layer, which comprises a shape (x , y), with x = y or with x
≠
y, wherein max (x, y) ≤ 10 …” as drafted cover mental activity and a mathematical procedure. More specifically, for claim 31, a human can mathematically determine that the convolutional layer, which comprises a shape (x , y), with x = y or with x
≠
y, wherein max (x, y) ≤ 10. The claimed invention is, therefore, directed to an abstract idea, a mental process and mathematical procedure without significantly more and thus, claim 31 is rejected under 35 U.S.C. 101.
Claim 32 recites “wherein at least one layer of the two or more hidden layers is a fully connected layer.” The limitation of claim 32 “at least one layer of the two or more hidden layers is a fully connected layer” as drafted cover just an additional limitation and additional element. More specifically, for claim 32, at least one layer of the two or more hidden layers can be a fully connected layer for the general neural network. As mentioned before, a generalized neural network as provided in the Spec can be used to implement this limitation. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 32 is rejected under 35 U.S.C. 101.
Claim 33 recites “wherein the hidden layers comprise at least one convolutional layer, at least one pooling layer, and at least one fully connected layer, wherein the filter depends on a psychoacoustic model, or wherein the spectral weighting depends on the psychoacoustic model.” The limitation of claim 33 “wherein the hidden layers comprise at least one convolutional layer, at least one pooling layer …”, “at least one fully connected layer, wherein the filter depends on a psychoacoustic model …”, “wherein the spectral weighting depends on the psychoacoustic model …” as drafted cover just an additional limitation and additional element. More specifically, for claim 33, a general network can have hidden layers comprising at least one convolutional layer, at least one pooling layer, and at least one fully connected layer, wherein the filter depends on a psychoacoustic model, or wherein the spectral weighting depends on the psychoacoustic model. As mentioned before, a generalized neural network as provided in the Spec can be used to implement this limitation. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 33 is rejected under 35 U.S.C. 101.
Claim 34 recites “wherein the apparatus is configured to employ linear activation in the output layer of the neural network, linear activation in the output layer of the neural network.” The limitation of claim 34 “wherein the apparatus is configured to employ linear activation in the output layer of the neural network, linear activation in the output layer of the neural network …” as drafted cover just an additional limitation and additional element. More specifically, for claim 34, the implementation of the apparatus can be with a general neural network that is configured to employ linear activation in the output layer of the neural network. As mentioned before, a generalized neural network as provided in the Spec can be used to implement this limitation. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 34 is rejected under 35 U.S.C. 101.
Claim 35 recites “wherein the input interface is configured to receive a plurality of spectral samples of the audio signal as the plurality of input values, and the neural network is configured to determine the estimate of the loudness of the signal components of interest of the audio signal depending on the plurality of power spectral samples of the audio signal.” The limitation of claim 35 “the input interface is configured to receive a plurality of spectral samples of the audio signal as the plurality of input values …”, “the neural network is configured to determine the estimate of the loudness of the signal components of interest of the audio signal depending on the plurality of power spectral samples of the audio signal …” as drafted cover mental activity and just an additional limitation or additional element. More specifically, for claim 35, a general neural network with input interface is configured to receive a plurality of spectral samples of the audio signal as the plurality of input values, and the neural network is configured to determine the estimate of the loudness of the signal components of interest of the audio signal depending on the plurality of power spectral samples of the audio signal. As mentioned before, a generalized neural network as provided in the Spec (Spec. page 22, lines 1-14) can be used to implement this limitation.-The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 35 is rejected under 35 U.S.C. 101.
Claim 36 recites “wherein the plurality of spectral samples are power spectral samples of at least 32 frequency bands” The limitation of claim 36 “the plurality of spectral samples are power spectral samples of at least 32 frequency bands …” as drafted cover mental activity. More specifically, for claim 36, a human can mathematically determine that the plurality of spectral samples are power spectral samples of at least 32 frequency bands. The claimed invention is, therefore, directed to an abstract idea and a mathematical process without significantly more and thus, claim 36 is rejected under 35 U.S.C. 101.
Claim 37 recites “wherein the plurality of spectral samples of the audio signal represent the audio signal in a time-frequency domain.” The limitation of claim 37 “the plurality of spectral samples of the audio signal represent the audio signal in a time-frequency domain” drafted cover mental activity. More specifically, for claim 37, a human can mathematically determine that the plurality of spectral samples of the audio signal represent the audio signal in a time-frequency domain. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 37 is rejected under 35 U.S.C. 101.
Claim 38 recites “wherein the apparatus further comprises a transform module configured for transforming the audio signal from a time domain to the time-frequency domain to acquire the plurality of spectral samples of the audio signal.” The limitation of claim 38 “the apparatus further comprises a transform module configured for transforming the audio signal from a time domain to the time-frequency domain to acquire the plurality of spectral samples of the audio signal” as drafted cover mental activity. More specifically, for claim 38, a human can mentally transform the audio signal from a time domain to the time-frequency domain to acquire the plurality of spectral samples of the audio signal. The claimed invention is, therefore, directed to an abstract idea and a mathematical process without significantly more and thus, claim 38 is rejected under 35 U.S.C. 101.
Claim 39 recites “wherein the transform module is configured to transform segments of the audio signal of at least 100 ms length from the time domain to the time-frequency domain to acquire the plurality of spectral samples of the audio signal.” The limitation of claim 39 “transform segments of the audio signal of at least 100 ms length from the time domain to the time-frequency domain to acquire the plurality of spectral samples of the audio signal …” as drafted cover mental activity. More specifically, for claim 39, a human can mathematically transform segments of the audio signal of at least 100 ms length from the time domain to the time-frequency domain to acquire the plurality of spectral samples of the audio signal. The claimed invention is, therefore, directed to an abstract idea and a mathematical process without significantly more and thus, claim 39 is rejected under 35 U.S.C. 101.
Claim 40 recites “wherein a first group of two or more of the plurality of spectral samples relate to a first group of frequency bands, which each exhibit a bandwidth that deviates by no more than 10 % from a predefined first bandwidth, wherein a second group of two or more of the plurality of spectral samples relate to a second group of frequency bands, which each exhibit a higher center frequency than each frequency band of the first group of frequency bands, and which each exhibit a bandwidth being higher than the bandwidth of each frequency band of the first group.” The limitation of claim 40 “a first group of two or more of the plurality of spectral samples relate to a first group of frequency bands …”, “each exhibit a bandwidth that deviates by no more than 10 % from a predefined first bandwidth …”, “a second group of two or more of the plurality of spectral samples relate to a second group of frequency bands, which each exhibit a higher center frequency than each frequency band of the first group of frequency bands, and which each exhibit a bandwidth being higher than the bandwidth of each frequency band of the first group …” as drafted cover mental activity. More specifically, for claim 40, a human can mathematically determine that a first group of two or more of the plurality of spectral samples relate to a first group of frequency bands, which each exhibit a bandwidth that deviates by no more than 10 % from a predefined first bandwidth, wherein a second group of two or more of the plurality of spectral samples relate to a second group of frequency bands, which each exhibit a higher center frequency than each frequency band of the first group of frequency bands, and which each exhibit a bandwidth being higher than the bandwidth of each frequency band of the first group. The claimed invention is, therefore, directed to an abstract idea and a mathematical process without significantly more and thus, claim 40 is rejected under 35 U.S.C. 101.
Claim 41 recites “wherein a third group of two or more of the plurality of spectral samples relate to a third group of frequency bands, which each exhibit a higher center frequency than each frequency band of the second group of frequency bands, which each exhibit a bandwidth being higher than the bandwidth of each frequency band of the second group, and wherein the bandwidth of each frequency band of the third group deviates less from an equivalent rectangular bandwidth than the bandwidth of each frequency band of the second group.” The limitation of claim 41 “a third group of two or more of the plurality of spectral samples relate to a third group of frequency bands …”each exhibit a higher center frequency than each frequency band of the second group of frequency bands …”, “each exhibit a bandwidth being higher than the bandwidth of each frequency band of the second group, and wherein the bandwidth of each frequency band of the third group deviates less from an equivalent rectangular bandwidth than the bandwidth of each frequency band of the second group …” as drafted cover mental activity. More specifically, for claim 41, a human can mentally and mathematically determine that a third group of two or more of the plurality of spectral samples relate to a third group of frequency bands, which each exhibit a higher center frequency than each frequency band of the second group of frequency bands, which each exhibit a bandwidth being higher than the bandwidth of each frequency band of the second group, and wherein the bandwidth of each frequency band of the third group deviates less from an equivalent rectangular bandwidth than the bandwidth of each frequency band of the second group. The claimed invention is, therefore, directed to an abstract idea and a mathematical process without significantly more and thus, claim 41 is rejected under 35 U.S.C. 101.
Claim 43 recites “wherein the signal components of interest of the audio signal are speech components of the audio signal, wherein the signal processor configured to modify the audio input signal depending on the estimate of the loudness of the speech components of the audio input signal to acquire the audio output signal.” The limitation of claim 43 “the signal components of interest of the audio signal are speech components of the audio signal …”, “the signal processor configured to modify the audio input signal depending on the estimate of the loudness of the speech components of the audio input signal to acquire the audio output signal …” as drafted cover mental activity. More specifically, for claim 43, a human can mentally modify the audio input signal depending on the estimate of the loudness of the speech components of the audio input signal to acquire the audio output signal. The claimed invention is, therefore, directed to an abstract idea and a mental process without significantly more and thus, claim 43 is rejected under 35 U.S.C. 101.
Claim 44 recites “wherein the signal processor is configured to modify the audio input signal depending on the estimate of the loudness of the speech components of the audio input signal and depending on an estimation of the loudness of the background components of the audio input signal to acquire the audio output signal.” The limitation of claim 44 “modify the audio input signal depending on the estimate of the loudness of the speech components of the audio input signal and depending on an estimation of the loudness of the background components of the audio input signal to acquire the audio output signal…” as drafted cover mental activity. More specifically, for claim 44, a human can mentally modify the audio input signal depending on the estimate of the loudness of the speech components of the audio input signal and depending on an estimation of the loudness of the background components of the audio input signal to acquire the audio output signal. For instance, based on the available data for an estimate of the loudness of the speech components and based on the data for an estimation of the loudness of the background components, one can easily apply a mathematical formula and modify the audio input signal. The claimed invention is, therefore, directed to an abstract idea, a mental process and data gathering. The claimed invention is, therefore, directed to an abstract idea, data gathering and a mental process without significantly more and thus, claim 44 is rejected under 35 U.S.C. 101.
Claim 45 recites “wherein the apparatus for providing an estimate of a loudness of speech components of the audio input signal is an apparatus configured to determine and output at least one other output value indicating an estimate of a partial loudness of the speech components of the audio signal, wherein the partial loudness of the speech components of the audio signal depends on the loudness of the speech components of the audio signal and on the loudness of background components of the audio signal, wherein the signal processor is configured to modify a level of the audio input signal depending on the partial loudness of the speech components of the audio signal.” The limitation of claim 45 “providing an estimate of a loudness of speech components of the audio input signal is an apparatus configured to determine and output at least one other output value indicating an estimate of a partial loudness of the speech components of the audio signal …”, “partial loudness of the speech components of the audio signal depends on the loudness of the speech components of the audio signal and on the loudness of background components of the audio signal …”, “modify a level of the audio input signal depending on the partial loudness of the speech components of the audio signal …” as drafted cover mental activity. More specifically, for claim 45, for instance, based on some collected speech and other combined audio data, a human can mentally estimate the loudness of the speech components of an audio input and also make an estimate for a partial loudness of the speech components of an audio signal depends on the loudness of the speech components of the audio signal as well as on the loudness of the background components of the audio signal. Thus one can mentally modify the level of the audio input signal depending on the partial loudness of the speech components of the audio signal. The claimed invention is, therefore, directed to an abstract idea, data gathering, and a mental process without significantly more and thus, claim 45 is rejected under 35 U.S.C. 101.
Thus, claims 1-17 and 29-47 as drafted cover a mental process and abstract idea of data gathering/retrieval and analysis/processing steps, and they are mental processes directed to an abstract idea of implementing mathematical formulae for data processing and data analysis using a conventional/generic (general-purpose) computer as well and thus, all the claims are directed to an abstract idea.
This judicial exception is not integrated into a practical application. In particular, several claims recite additional element of “signal processor,” “postprocessor,” and “computer program” as per the independent and dependent claims. Several claims also recite additional element of “neural network” , which is a generalized neural network as provided in the Spec (Spec. page 22, lines 1-14) as per the independent and dependent claims. The Spec also provides a general purpose “signal processor” or computer (Spec. page 5, line 20). Independent claim 47 recites additional element “storage medium” as well. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Thus, the claims are directed to an abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of using a computer is noted as a general computer as noted. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept (Spec., Page 26, line 15-19). Further, the additional limitation in the claims noted above are directed towards insignificant solution activity. The claims are not patent eligible.
Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional general purpose computer implementation. Claims 1-17 and 29-47, are therefore not drawn to patent eligible subject matter as they are directed to an abstract idea without significantly more. Thus, the claimed invention is directed to an abstract idea and a mental process without significantly more and thus, claims 1-17 and 29-47 are rejected under 35 U.S.C. 101.
Dependent claims 1-17 and 29-47 are also directed toward an abstract idea and do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea. Therefore, claims 1-17 and 29-47 do not contain patent eligible subject matter that has been identified by the courts.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 5, 9-11, 29-31, 35-39, 42-43 and 46 are rejected under 35 U.S.C. 103 as being unpatentable over Meléndez et al., “Relative music loudness estimation using temporal convolutional networks and a cnn feature extraction front-end,” In Proceedings of the 23rd International Conference on Digital Audio Effects (DAFx-20) (Vol. 5, pp. 273-280), 2020, in view of in view of Paulus et al. Pat App No. US 20230306973 A1 (Paulus).
Regarding Claim 1, Meléndez discloses an apparatus for providing an estimate of a loudness of signal components of interest of an audio signal, wherein the apparatus (Meléndez, page 274, 2nd col, 1st para, In this section, we detail the models that we propose for the task of relative music loudness estimation: the TCN and the CNN-TCN, which is the combination of a CNN front-end with a TCN) comprises:
an input interface configured to receive a plurality of samples of the audio signal (Meléndez, page 274, 2nd col, 2nd para, 3.1. Feature generation We use audio at 8000 samples per second with 16 bits per sample and normalized to have a maximum amplitude value of 1. From the audio data, we compute the power spectrogram with a Hanning window of length 512 samples (64 ms) and a hop size of 128 samples (16 ms). We then apply a Mel filter bank with Nmels = 128 filters to obtain the Mel-spectrogram; Meléndez, page 275, 1st col, Fig. 2, “xt - Input”; Meléndez, page 275, 1st col, 4th para, Fig. 2 is a simplified representation of a TCN model where xt ∈ RNmels is the time-frame t of the input features [i.e., Fig. 2, “xt - Input” as “input interface”]), and
a neural network configured to receive as input values the plurality of samples of the audio signal or a plurality of derived values being derived from the plurality of samples of the audio signal (Meléndez, page 275, 1st col, Figure 2 caption, Figure 2: An example of a TCN’s receptive field used to classify a single time-frame. The architecture includes 2 residual blocks with dilations d = [1, 2] and non-causal filters of length L = 3. This network’s receptive field is equal to 7 time-frames; [i.e., TCN stands for “Temporal Convolutional Network” as “neural network”]), and configured to determine at least one output value from the plurality of input values, such that the at least one output value indicates the estimate of the loudness of the signal components of interest of the audio signal (Meléndez, page 275, 1st col, 4th para, Fig. 2 is a simplified representation of a TCN model where xt ∈ RNmels is the time-frame t of the input features, yt ∈ RNtcn is the output of the last residual block for that time-frame and yt ∈ R3 is the vector that carries the probability of the 3 relative music loudness estimation classes for that time-frame; [“xt …the input features” as “input values the plurality of samples of the audio signal”]).
Meléndez does not specifically disclose wherein the apparatus is configured to modify the audio signal depending on the estimate of the loudness of the signal components of interest of the audio signal to acquire an audio output signal, and wherein the apparatus is implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.
However, Paulus, in the same field of endeavour, discloses wherein the apparatus is configured to modify the audio signal depending on the estimate of the loudness of the signal components of interest of the audio signal to acquire an audio output signal, and wherein the apparatus is implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer (Paulus, para 0140, The output loudness estimator 740 may be configured to determine the loudness compensation value. The object-based audio decoding unit 750 may be configured to determine a modified audio signal from an audio signal, being input to the decoder, by applying the rendering information R. Applying the loudness compensation value on the modified audio signal to compensate a total loudness change caused by the rendering is not shown in FIG. 7, para [0020], computer).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Paulus in the method of Meléndez because this would enable the volume levels of the audio tracks of various programs to be normalized based on various aspects, such as the peak signal level or the loudness level, in TV and radio broadcast (Paulus, para 0004).
Regarding Claim 3, Meléndez in view of Paulus disclose the apparatus according to claim 1,
wherein the signal components of interest of the audio signal are speech components of the audio signal (Meléndez, page 274, 1st col, 2nd para, In 2012, Schlüter et al. [11] proposed the first approach to music and speech detection using deep learning architectures. They designed two identical networks, one for music detection and another one for speech detection), and
wherein the neural network is configured to determine the at least one output value from the plurality of input values, such that the at least one output value indicates the estimate of the loudness of the speech components of the audio signal (Meléndez, page 275, 1st col, 4th para, Fig. 2 is a simplified representation of a TCN model where …yt ∈ RNtcn is the output of the last residual block for that time-frame).
Regarding Claim 5, Meléndez in view of Paulus disclose the apparatus according to claim 1.
wherein the signal components of interest of the audio signal are sound components of at least one first sound source out of a plurality of sound sources in an environment (Meléndez, page 278, 2nd col, 1st para, Speech mixed with background non-music noises with an identifiable pitch such as engine sounds (No Music) classified as Background music ),
wherein the audio signal simultaneously comprises the sound components of the at least one first sound source and other sound components of one or more other sound sources out of the plurality of sound sources in the environment (Meléndez, page 278, 2nd col, 1st para, Speech mixed with background non-music noises with an identifiable pitch such as engine sounds (No Music) classified as Background music),
wherein the neural network is configured to determine the at least one output value from the plurality of input values, such that the at least one output value indicates the estimate of the loudness of the sound components of the at least one first sound source (Meléndez, page 275, 1st col, 4th para, Fig. 2 is a simplified representation of a TCN model where …yt ∈ RNtcn is the output of the last residual block for that time-frame and yt ∈ R3 is the vector that carries the probability of the 3 relative music loudness estimation classes for that time-frame),
wherein an influence of the other sound components of the one or more other sound sources on the estimate of the loudness of the sound components of the at least one first sound source is reduced or not present (Meléndez, page 273, 1st col, 2nd para, music is used many times in the background, for instance, as a means to create a certain atmosphere. In this scenario, music detection algorithms fall short as we need to estimate the loudness of music in relation to other simultaneous non-music sounds, i.e., its relative loudness).
Regarding Claim 9, Meléndez in view of Paulus disclose the apparatus according to claim 1,
wherein the neural network is configured to determine at least one further output value indicating an estimate of a loudness of the entire audio signal (Meléndez, page 275, 1st col, 4th para, Fig. 2 is a simplified representation of a TCN model where …yt ∈ RNtcn is the output of the last residual block for that time-frame and yt ∈ R3 is the vector that carries the probability of the 3 relative music loudness estimation classes for that time-frame).
Regarding Claim 10, Meléndez in view of Paulus disclose the apparatus according to claim 1,
wherein the neural network is configured to determine one or more further output values indicating an estimate of a loudness of the audio signal when speech is present (Meléndez, page 275, 1st col, 4th para, Fig. 2 is a simplified representation of a TCN model where …yt ∈ RNtcn is the output of the last residual block for that time-frame and yt ∈ R3 is the vector that carries the probability of the 3 relative music loudness estimation classes for that time-frame).
Regarding Claim 11, Meléndez in view of Paulus disclose the apparatus according to claim 1,
wherein the neural network is configured to determine another one or more output values indicating an estimate of a loudness of background components of the audio signal (Meléndez, page 276, 1st col, 4th para, M1 and M2 consist in a CNN with 3 convolutional blocks and 2 dense layers. Each of the convolutional blocks is composed by a 2D-convolutional layer … The difference in accuracy between M2 and M1 in MIREX 2019 was approximately 2 percentage points (pp) for the task of music detection and 3 pp for the task of relative music loudness estimation; Meléndez, page 275, 2nd col, 5th para - page 276, 1st col, 2nd para, In this work, we use OpenBMAT [5]3 , an open dataset annotated for the tasks of music detection and relative music loudness estimation that contains over 27 hours of TV broadcast audio from France, Germany, Spain and the United Kingdom distributed over 1647 one-minute long excerpts. . It is the first dataset to include annotations about the loudness of music in relation to other simultaneous non-music sounds… This dataset comes with 10 predefined splits containing approximately 15% Foreground Music, 35% Background Music and 50% No Music; [The dataset OpenBMAT for loudness estimation includes estimation of a loudness of background components which can be anything, including music. OpenBMAT includes 15% Foreground Music, 35% Background Music and 50% No Music]).
Regarding Claim 29, Meléndez in view of Paulus disclose the apparatus according to claim 1,
wherein the neural network comprises an input layer, two or more hidden layers, and an output layer (Meléndez, page 275, 1st col, Figure 2: An example of a TCN’s receptive field used to classify a single time-frame. The architecture includes 2 residual blocks with dilations d = [1, 2] and non-causal filters of length L = 3. This network’s receptive field is equal to 7 time-frames; Meléndez, page 274, 1st col, 2nd para, CNN for music detection already including six consecutive separable 2D-convolutional layers; [i.e., Figure 2, d=2 and d=2 are hidden layers coming between the input and output layers.]),
wherein the input layer comprises a plurality of input nodes, wherein each of the plurality of input nodes is configured to receive one of the plurality of input values; (Meléndez, page 275, 1st col, 5th para, In the left part of Fig. 1, we show the CNN-TCN model, which is a combination of a CNN front-end and the TCN described above. The CNN consists in a stack of 7 blocks that comprehend: (1) a 2D-convolutional layer with Ncnn 3x3 filters and a ReLU activation function, (2) a spatial dropout layer with a dropout rate dr and (3) a max-pooling layer; Meléndez, page 275, 1st col, Figure 2:x is the input layer),
wherein each of the two or more hidden layers comprises one or more neural nodes (Meléndez, page 275, 2nd col, Figure 1: (left) CNN-TCN architecture. (right-top) Convolutional block. (right-bottom) Residual block; Meléndez, page 274, 1st col, 2nd para, CNN for music detection already including six consecutive separable 2D-convolutional layers; [i.e., Figure 2, d=2 and d=2 are hidden layers coming between the input and output layers; Meléndez, page 274, 2nd col, 3rd - 4th para: The TCN model is formed by a stack of 6 residual blocks…Our residual blocks contain two 1D-convolutional layers as proposed by Bai et al. [19]; Meléndez, page 276, 1st col, 3rd – 4th para, Melendez-Catalan et al. [21] submitted a regression algorithm based on a CNN to this competition; Meléndez, page 276, 1st col, 5th para, We choose two algorithms to compare our models with. They are two of the algorithms that Melendez-Catalan et al. presented to the tasks of music detection and relative music loudness estimation of MIREX 2019: M1 [21] and M2 [22]. M1 was already submitted to MIREX 2018, where it obtained first place out of 5 participants in the music detection task, and was the first algorithm to participate in the relative music loudness estimation task. In 2019, M2 and M1 obtained second and third place, respectively, in both tasks. The winner was a CNN-TCN prototype that Melendez-Catalan et al. produced during the elaboration of this paper. M1 and M2 consist in a CNN with 3 convolutional blocks and 2 dense layers. Each of the convolutional blocks is composed by a 2D-convolutional layer; [i.e., the Melendez-Catalan M1 and M2 algorithms for the tasks of music detection and relative music loudness estimation in MIREX 2019 which gave rise to the CNN-TCN prototype as in Figure 2, and both residual blocks in Figure 2 are between the input and the output layers, meaning at least two hidden layers of the CNN/neural network]), and
wherein the output layer comprises at least one output node, wherein the at least one output node is configured to output the at least one output value indicating the estimate of the loudness of the signal components of interest of the audio signal (Meléndez, page 275, 1st col, 2nd para, The output layer for both TCN and CNN-TCN architectures has 3 neurons, each of them corresponding to one of the three classes of the relative music loudness estimation task; Meléndez, page 274, 1st col, 5th para, The relative music loudness estimation task appeared for the first time in MIREX 2018. Melendez-Catalan et al. [21] submitted a regression algorithm based on a CNN to this competition.).
Regarding Claim 30, Meléndez in view of Paulus disclose the apparatus according to claim 29,
wherein at least one layer of the two or more hidden layers is a convolutional layer (Meléndez, page 274, 2nd col, 3rd – 4th para, The TCN model is formed by a stack of 6 residual blocks. A residual block, as defined by He et al. [23], applies a certain function (F) to an input (x) that depends on the weights ({Wn}) and biases
({bn}) of the N layers contained in the residual block. The output
of this function is then added back to the input and passed to an
activation function to obtain the output of the residual block (y).
This way, the layers inside the residual block learn modifications
to the input instead of a complete transformation. This has proven
to ease their optimization [23]. Eq. 1 presents the formal definition
of a residual block.
PNG
media_image1.png
24
304
media_image1.png
Greyscale
In the right-bottom part of Fig. 1, we show the structure of
the residual blocks that we use in this paper. Our residual blocks
contain two 1D-convolutional layers as proposed by Bai et al. [19]).
Regarding Claim 31, Meléndez in view of Paulus disclose the apparatus according to claim 30,
wherein the neural network is configured to employ a convolutional filter for the convolutional layer, which comprises a shape (x , y), with x = y or with x
≠
y, wherein max (x, y) ≤ 10 (Meléndez, page 276, 1st col, 4th para, M1 and M2 consist in a CNN with 3 convolutional blocks and 2 dense layers. Each of the convolutional blocks is composed by a 2D-convolutional layer with a ReLU activation, and a max-pooling layer. The two algorithms differ in several hyper-parameters such
as the number of 2D-convolutional filters and their size; Meléndez, page 275, 1st col, 5th para, In the left part of Fig. 1, we show the CNN-TCN model, which is a combination of a CNN front-end and the TCN described above. The CNN consists in a stack of 7 blocks that comprehend: (1) a 2D-convolutional layer with Ncnn 3x3 filters; [i.e., the Examiner assumed that the x=y filter is a 3 by 3 filter]).
Regarding Claim 35, Meléndez in view of Paulus disclose the apparatus according to claim 1,
wherein the input interface is configured to receive a plurality of spectral samples of the audio signal as the plurality of input values (Meléndez, page 275, 1st col, 2nd para, The first 1D-convolutional layer of the TCN reads the logmagnitude Mel-spectrogram as Nmels scalar temporal sequences by interpreting the frequency axis as channels), and
the neural network is configured to determine the estimate of the loudness of the signal components of interest of the audio signal depending on the plurality of power spectral samples of the audio signal (Meléndez, page 276, 1st col, 3rd – 4th para, We choose two algorithms to compare our models with. They are two of the algorithms that Meléndez-Catalán et al. presented to the tasks of music detection and relative music loudness estimation of MIREX 2019: M1 [21] and M2 [22]… M1 and M2 consist in a CNN with 3 convolutional blocks and 2 dense layers. Each of the convolutional blocks is composed by a 2D-convolutional layer with a ReLU activation, and a max-pooling layer. The two algorithms differ in several hyper-parameters such as the number of 2D-convolutional filters and their size. M1 and M2 have a total of 97,779 and 453,763 parameters, respectively. The input to both networks is the log-magnitude Mel-spectrogram, with 128 frequency bins, of approximately 2 seconds of audio, which translates to 128 time-frames).
Regarding Claim 36, Meléndez in view of Paulus disclose the apparatus according to claim 35,
wherein the plurality of spectral samples are power spectral samples of at least 32 frequency bands (Meléndez, page 276, 1st col, 4th para, The input to both networks is the log-magnitude Mel-spectrogram, with 128 frequency bins, of approximately 2 seconds of audio, which translates to 128 time-frames).
Regarding Claim 37, Meléndez in view of Paulus disclose the apparatus according to claim 35,
wherein the plurality of spectral samples of the audio signal represent the audio signal in a time-frequency domain (Meléndez, page 276, 1st col, 4th para, The input to both networks is the log-magnitude Mel-spectrogram, with 128 frequency bins, of approximately 2 seconds of audio, which translates to 128 time-frames).
Regarding Claim 38, Meléndez in view of Paulus disclose the apparatus according to claim 37,
wherein the apparatus further comprises a transform module configured for transforming the audio signal from a time domain to the time-frequency domain to acquire the plurality of spectral samples of the audio signal (Meléndez, page 278, 2nd col, 2nd para, The top and mid parts of Fig. 4 show how the CNN frontend works as a feature extractor transforming and reducing the dimensionality of the input log-magnitude Mel-spectrogram; Meléndez, page 278, 2nd col, Figure 4 Cation, Figure 4: (top) Example of the log-magnitude Mel-spectrogram, which we use as input features for both the TCN and CNN-TCN architectures. (mid) Output of the CNN in the CNN-TCN architecture for these features. (bottom-top) CNNTCNbest classification for these features without smoothing. (bottom-bottom) Ground truth of these features.).
Regarding Claim 39, Meléndez in view of Paulus discloses the apparatus according to claim 38,
wherein the transform module is configured to transform segments of the audio signal of at least 100 ms length from the time domain to the time-frequency domain to acquire the plurality of spectral samples of the audio signal (Meléndez, page 274, 2nd col, 2nd para, We use audio at 8000 samples per second with 16 bits per sample and normalized to have a maximum amplitude value of 1. From the audio data, we compute the power spectrogram with a Hanning window of length 512 samples (64 ms) and a hop size of 128 samples (16 ms)).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate 100ms samples instead of the 64ms or 16ms lengths in the method of Meléndez in view of Paulus because this would introduce an additional design choice and also this would result in the appropriate duration being used to compute the frequency portion of the signal.
Regarding Claim 42, Meléndez in view of Paulus disclose
an apparatus according to claim 1 nd col, 1st para, In this section, we detail the models that we propose for the task of relative music loudness estimation: the TCN and the CNN-TCN, which is the combination of a CNN front-end with a TCN)
Meléndez in view of Paulus do not specifically disclose wherein the apparatus comprises a signal processor configured to modify the audio input signal depending on the estimate of the loudness of the signal components of interest of the audio input signal to acquire the audio output signal.
Furthermore, Paulus discloses:
wherein the apparatus comprises a signal processor configured to modify the audio input signal depending on the estimate of the loudness of the signal components of interest of the audio input signal to acquire the audio output signal (Paulus, para 0140, The output loudness estimator 740 may be configured to determine the loudness compensation value. The object-based audio decoding unit 750 may be configured to determine a modified audio signal from an audio signal, being input to the decoder, by applying the rendering information R. Applying the loudness compensation value on the modified audio signal to compensate a total loudness change caused by the rendering is not shown in FIG. 7).
Regarding Claim 43, Meléndez in view of Paulus disclose the
Furthermore, Meléndez further discloses:
wherein the signal components of interest of the audio signal are speech components of the audio signal (Meléndez, page 274, 1st col, 2nd para, In 2012, Schlüter et al. [11] proposed the first approach to music and speech detection using deep learning architectures. They designed two identical networks, one for music detection and another one for speech detection),
Furthermore, Paulus discloses:
wherein the signal processor configured to modify the audio input signal depending on the estimate of the loudness of the speech components of the audio input signal to acquire the audio output signal (Paulus, para 0140, The output loudness estimator 740 may be configured to determine the loudness compensation value. The object-based audio decoding unit 750 may be configured to determine a modified audio signal from an audio signal, being input to the decoder, by applying the rendering information R. Applying the loudness compensation value on the modified audio signal to compensate a total loudness change caused by the rendering is not shown in FIG. 7.).
Regarding Claim 46, Meléndez discloses a method for providing an estimate of a loudness of signal components of interest of an audio signal (Meléndez, page 274, 2nd col, 1st para, In this section, we detail the models that we propose for the task of relative music loudness estimation: the TCN and the CNN-TCN, which is the combination of a CNN front-end with a TCN), wherein the method comprises:
receiving a plurality of samples of the audio signal (Meléndez, page 274, 2nd col, 2nd para, 3.1. Feature generation We use audio at 8000 samples per second with 16 bits per sample and normalized to have a maximum amplitude value of 1. From the audio data, we compute the power spectrogram with a Hanning window of length 512 samples (64 ms) and a hop size of 128 samples (16 ms). We then apply a Mel filter bank with Nmels = 128 filters to obtain the Mel-spectrogram; Meléndez, page 275, 1st col, Fig. 2, “xt - Input”; Meléndez, page 275, 1st col, 4th para, Fig. 2 is a simplified representation of a TCN model where xt ∈ RNmels is the time-frame t of the input features), and
estimating the loudness of the signal components of interest of the audio signal (Meléndez, page 274, 2nd col, 1st para, In this section, we detail the models that we propose for the task of relative music loudness estimation: the TCN and the CNN-TCN, which is the combination of a CNN front-end with a TCN),
wherein a neural network receives as input values the plurality of samples of the audio signal or a plurality of derived values being derived from the plurality of samples of the audio signal (Meléndez, page 275, 1st col, Figure 2 caption, Figure 2: An example of a TCN’s receptive field used to classify a single time-frame. The architecture includes 2 residual blocks with dilations d = [1, 2] and non-causal filters of length L = 3. This network’s receptive field is equal to 7 time-frames; [i.e., TCN stands for “Temporal Convolutional Network” as “neural network”]), and
wherein the neural network determines at least one output value from the plurality of input values, such that the at least one output value indicates the estimate of the loudness of the signal components of interest of the audio signal (Meléndez, page 275, 1st col, 4th para, Fig. 2 is a simplified representation of a TCN model where xt ∈ RNmels is the time-frame t of the input features, yt ∈ RNtcn is the output of the last residual block for that time-frame and yt ∈ R3 is the vector that carries the probability of the 3 relative music loudness estimation classes for that time-frame; [“xt …the input features” as “input values the plurality of samples of the audio signal”]).
Meléndez does not specifically disclose wherein the method comprises modifying the audio signal depending on the estimate of the loudness of the signal components of interest of the audio signal to acquire an audio output signal, and wherein the method is implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.
However, Paulus, in the same field of endeavour, discloses wherein the method comprises modifying the audio signal depending on the estimate of the loudness of the signal components of interest of the audio signal to acquire an audio output signal , and wherein the method is implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer (Paulus, para 0140, The output loudness estimator 740 may be configured to determine the loudness compensation value. The object-based audio decoding unit 750 may be configured to determine a modified audio signal from an audio signal, being input to the decoder, by applying the rendering information R. Applying the loudness compensation value on the modified audio signal to compensate a total loudness change caused by the rendering is not shown in FIG. 7 para [0020], computer).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Paulus in the method of Meléndez because this would enable the volume levels of the audio tracks of various programs to be normalized based on various aspects, such as the peak signal level or the loudness level, in TV and radio broadcast (Paulus, para 0004).
Claims 2, 4, and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Meléndez in view of Paulus, and furtherin view of Vinton et al. Pat No. US 7454331 B2 (Vinton).
Regarding Claim 2, Meléndez in view of Paulus disclose the apparatus according to claim 1.
Meléndez in view of Paulus do not specifically disclose wherein the audio signal simultaneously comprises the signal components of interest and other signal components of the audio signal, wherein an influence of the other signal components on the estimate of the loudness of the signal components of interest is reduced or not present.
However, Vinton, in the same field of endeavour, discloses:
wherein the audio signal simultaneously comprises the signal components of interest and other signal components of the audio signal (Vinton, col 5, ln 59 – col 6, ln 4, Table III includes information for the relative loudness of different sounds in three different scenes of one or more motion pictures. In Scene 1, people are speaking on the deck of a ship. Background sounds include the lapping of waves and a distant fog horn at levels significantly below the speech level. The scene also includes a blast from the ship's horn, which is substantially louder than the speech. In Scene 2, people are whispering and a clock is ticking in the background. The voices in this scene are not as loud as normal speech and the loudness of the clock ticks is even lower. In Scene 3, people are shouting near a machine that is making an even louder sound. The shouting is louder than normal speech),
wherein an influence of the other signal components on the estimate of the loudness of the signal components of interest is reduced or not present (Vinton, col 2, ln 43-48, measure or estimate the loudness of speech in the selected interval... The selected interval should contain representative speech but not contain other types of audio material that would distort the loudness measurement).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Vinton in the method of Meléndez in view of Paulus because this would reduce the effects of very undesirable background sound effects such as wind, distant traffic, or gently flowing water which need not have the same loudness as speech (Vinton, col 2, ln 55-57).
Regarding Claim 4, Meléndez in view of Paulus disclose the apparatus according to claim 3,
wherein the audio signal simultaneously comprises the speech components and background components of the audio signal (Meléndez, page 278, 2nd col, 1st para, Speech mixed with background non-music noises with an identifiable pitch such as engine sounds (No Music) classified as Background music).
Meléndez in view of Paulus do not specifically teach wherein an influence of the background components on the estimate of the loudness of the speech components is reduced or not present.
However, Vinton, in the same field of endeavour, discloses wherein an influence of the background components on the estimate of the loudness of the speech components is reduced or not present (Vinton, col 2, ln 43-48, measure or estimate the loudness of speech in the selected interval... The selected interval should contain representative speech but not contain other types of audio material that would distort the loudness measurement. ).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Vinton in the method of Meléndez in view of Paulus because this would reduce the effects of very undesirable background sound effects such as wind, distant traffic, or gently flowing water which need not have the same loudness as speech (Vinton, col 2, ln 55-57).
Regarding Claim 6, Meléndez in view of Paulus disclose the apparatus according to claim 5:
wherein the neural network is configured to determine the at least one output value from the plurality of input values, such that the at least one output value indicates (Meléndez, page 275, 1st col, 4th para, Fig. 2 is a simplified representation of a TCN model where …yt ∈ RNtcn is the output of the last residual block for that time-frame),
Meléndez in view of Paulus do not specifically disclose the estimate of the loudness of the speech components of the first person, wherein an influence of the other speech components of the one or more other persons on the estimate of the loudness of the speech components of the first person is reduced or not present, wherein the sound components of the at least one first sound source are speech components of a first person out of a plurality of persons speaking in the environment, wherein the other sound components of the one or more other sound sources are other speech components of one or more other persons out of the plurality of persons speaking in the environment, and wherein the audio signal simultaneously comprises the speech components of the first person and the other speech components of the one or more other persons speaking in the environment.
However, Vinton, in the same field of endeavour, discloses:
the estimate of the loudness of the speech components of the first person (Vinton, col 7, ln 23-27, Each portion of the audio signal that is represented by a segment of audio information has a respective loudness. The loudness estimator 14 examines the speech segments and obtains an estimate of this loudness for the speech segments),
wherein an influence of the other speech components of the one or more other persons on the estimate of the loudness of the speech components of the first person is reduced or not present (Vinton, col 5, ln 24-27, Table I includes information for the relative loudness of speech in three programs like those that may be broadcast to television receivers. In Newscast 1, two people are speaking at different levels),
wherein the sound components of the at least one first sound source are speech components of a first person out of a plurality of persons speaking in the environment (Vinton, col 5, ln 24-27, Table I includes information for the relative loudness of speech in three programs like those that may be broadcast to television receivers. In Newscast 1, two people are speaking at different levels),
wherein the other sound components of the one or more other sound sources are other speech components of one or more other persons out of the plurality of persons speaking in the environment (Vinton, col 5, ln 59 – col 6, ln 4, Table III includes information for the relative loudness of different sounds in three different scenes of one or more motion pictures. In Scene 1, people are speaking on the deck of a ship. Background sounds include the lapping of waves and a distant fog horn at levels significantly below the speech level. The scene also includes a blast from the ship's horn, which is substantially louder than the speech. In Scene 2, people are whispering and a clock is ticking in the background. The voices in this scene are not as loud as normal speech and the loudness of the clock ticks is even lower. In Scene 3, people are shouting near a machine that is making an even louder sound. The shouting is louder than normal speech), wherein the audio signal simultaneously comprises the speech components of the first person and the other speech components of the one or more other persons speaking in the environment (Vinton, col 5, ln 24-32, Table I includes information for the relative loudness of speech in three programs like those that may be broadcast to television receivers. In Newscast 1, two people are speaking at different levels. In Newscast 2, a person is speaking at a low level at a location with other sounds that are occasionally louder than the speech. Music is sometimes present at a low level. In Commercial, a person is speaking at a very high level and music is occasionally even louder; Vinton, col 5, ln 54-65, Table III includes information for the relative loudness of different sounds in three different scenes of one or more motion pictures. In Scene 1, people are speaking on the deck of a ship. Background sounds include the lapping of waves and a distant fog horn at levels significantly below the speech level. The scene also includes a blast from the ship's horn, which is substantially louder than the speech; [i.e., All of these sounds, e.g., at the deck of the ship, and people speaking are happening simultaneously]),
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Vinton in the method of Meléndez in view of Paulus because this would reduce the effects of very undesirable background sound effects such as wind, distant traffic, or gently flowing water which need not have the same loudness as speech (Vinton, col 2, ln 55-57).
Claim 47 is rejected under 35 U.S.C. 103 as being unpatentable over Meléndez in view of Paulus, and further in view of Ward et al. JP 2020173486 A (Ward)
Regarding Claim 47, Meléndez discloses a method of for providing an estimate of a loudness of signal components of interest of an audio signal (Meléndez, page 274, 2nd col, 1st para, In this section, we detail the models that we propose for the task of relative music loudness estimation: the TCN and the CNN-TCN, which is the combination of a CNN front-end with a TCN), wherein the method comprises:
receiving a plurality of samples of the audio signal (Meléndez, page 274, 2nd col, 2nd para, 3.1. Feature generation We use audio at 8000 samples per second with 16 bits per sample and normalized to have a maximum amplitude value of 1. From the audio data, we compute the power spectrogram with a Hanning window of length 512 samples (64 ms) and a hop size of 128 samples (16 ms). We then apply a Mel filter bank with Nmels = 128 filters to obtain the Mel-spectrogram; Meléndez, page 275, 1st col, Fig. 2, “xt - Input”; Meléndez, page 275, 1st col, 4th para, Fig. 2 is a simplified representation of a TCN model where xt ∈ RNmels is the time-frame t of the input features), and
estimating the loudness of the signal components of interest of the audio signal (Meléndez, page 274, 2nd col, 1st para, In this section, we detail the models that we propose for the task of relative music loudness estimation: the TCN and the CNN-TCN, which is the combination of a CNN front-end with a TCN),
wherein a neural network receives as input values the plurality of samples of the audio signal or a plurality of derived values being derived from the plurality of samples of the audio signal (Meléndez, page 275, 1st col, Figure 2 caption, Figure 2: An example of a TCN’s receptive field used to classify a single time-frame. The architecture includes 2 residual blocks with dilations d = [1, 2] and non-causal filters of length L = 3. This network’s receptive field is equal to 7 time-frames; [i.e., TCN stands for “Temporal Convolutional Network” as “neural network”]), and
wherein the neural network determines at least one output value from the plurality of input values, such that the at least one output value indicates the estimate of the loudness of the signal components of interest of the audio signal (Meléndez, page 275, 1st col, 4th para, Fig. 2 is a simplified representation of a TCN model where xt ∈ RNmels is the time-frame t of the input features, yt ∈ RNtcn is the output of the last residual block for that time-frame and yt ∈ R3 is the vector that carries the probability of the 3 relative music loudness estimation classes for that time-frame; [“xt …the input features” as “input values the plurality of samples of the audio signal”]).
Meléndez does not specifically disclose wherein the method comprises modifying the audio signal depending on the estimate of the loudness of the signal components of interest of the audio signal to acquire an audio output signal.
However, Paulus, in the same field of endeavour, discloses wherein the method comprises modifying the audio signal depending on the estimate of the loudness of the signal components of interest of the audio signal to acquire an audio output signal (Paulus, para 0140, The output loudness estimator 740 may be configured to determine the loudness compensation value. The object-based audio decoding unit 750 may be configured to determine a modified audio signal from an audio signal, being input to the decoder, by applying the rendering information R. Applying the loudness compensation value on the modified audio signal to compensate a total loudness change caused by the rendering is not shown in FIG. 7).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Paulus in the method of Meléndez because this would enable the volume levels of the audio tracks of various programs to be normalized based on various aspects, such as the peak signal level or the loudness level, in TV and radio broadcast (Paulus, para 0004).
Meléndez in view of Paulus do not specifically disclose non-transitory digital storage medium having stored thereon a computer program for performing, and when the computer program is run by a computer or signal processor.
However, Ward, in the same field of endeavour, discloses:
non-transitory digital storage medium having stored thereon a computer program for performing (In certain embodiments, a non-transitory computer-readable storage medium containing software instructions that, when executed by one or more processors, triggers the execution of any of the methods described herein), and
when the computer program is run by a computer or signal processor (In certain embodiments, a device having a processor and configured to perform any of the methods described herein).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Ward in the method of Meléndez in view of Paulus because this would provide loudness adjustment for downmixed audio content and audio content coded for a reference speaker configuration is downmixed to downmix audio content coded for a specific speaker configuration (Ward, Abstract).
Claims 32-33 are rejected under 35 U.S.C. 103 as being unpatentable over Meléndez in view of Paulus, further in view of Zhang, and further in view of Jia et al., "Hierarchical Regulated Iterative Network for Joint Task of Music Detection and Music Relative Loudness Estimation," in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1-13, 2021 (Jia).
Regarding Claim 32, Meléndez in view of Paulus disclose the apparatus according to claim 29.
Meléndez in view of Paulus does not specifically disclose wherein at least one layer of the two or more hidden layers is a fully connected layer.
However, Jia, in the same field of endeavor discloses wherein at least one layer of the two or more hidden layers is a fully connected layer (Jia, 8th page, 3rd para, For HRIN-cr, we design one recurrent unit architecture as 2-layer bidirectional GRU with the hidden size of 50 and no dropout. We design one convolutional unit as three 1-D convolutional layers. Each convolutional layer is followed by a batch normalization layer [51], a ReLU activation function [52], and a max-pooling layer).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Jia in the method of Meléndez in view of Paulus because this would solve the practical requirement of the music copyright management problem which is the estimation of music relative loudness that is mostly ignored in existing music detection works through a study of the joint task of music detection and music relative loudness estimation (Jia, Abstract).
Regarding Claim 33, Meléndez in view of Paulus disclose the apparatus according to claim 29.
Meléndez in view of Paulus do not specifically disclose wherein the hidden layers comprise at least one convolutional layer, at least one pooling layer, and at least one fully connected layer.
However, Jia, in the same field of endeavor discloses wherein the hidden layers comprise at least one convolutional layer, at least one pooling layer, and at least one fully connected layer (Jia, 8th page, 3rd para, We design one convolutional unit as three 1-D convolutional layers. Each convolutional layer is followed by a batch normalization layer [51], a ReLU activation function [52], and a max-pooling layer; Jia, 5th page, 1st col, 2nd para, Then o1[t] is passed into the subsequent fully-connected layer with sigmoid as the ultimate activation function).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Jia in the method of Meléndez in view of Paulus because this would solve the practical requirement of the music copyright management problem which is the estimation of music relative loudness that is mostly ignored in existing music detection works through a study of the joint task of music detection and music relative loudness estimation (Jia, Abstract).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Meléndez in view of Paulus, further in view of Vinton, and further in view of Azizi et al. Pat No. EP 2101411 A1 (Azizi).
Regarding Claim 7, Meléndez in view of Paulus disclose the apparatus according to claim 5.
wherein the neural network is configured to determine the at least one output value from the plurality of input values (Meléndez, page 275, 1st col, 4th para, Fig. 2 is a simplified representation of a TCN model where xt ∈ RNmels is the time-frame t of the input features, yt ∈ RNtcn is the output of the last residual block for that time-frame and yt ∈ R3 is the vector that carries the probability of the 3 relative music loudness estimation classes for that time-frame; [“xt …the input features” as “input values the plurality of samples of the audio signal”]).
Meléndez in view of Paulus do not specifically disclose wherein the sound components of the at least first sound source are sound components of at least one non-human sound source out of a plurality of non-human sound sources in an environment, wherein the other sound components of the one or more other sound sources are other sound components of one or more other non-human sound source out of the plurality of non-human sound sources, wherein the audio signal simultaneously comprises the sound components of the at least one first non-human sound source and the other sound components of the one or more other non-human sound sources in the environment, and wherein an influence of the other sound components of the one or more other non-human sound sources on the estimate of the loudness of the sound components of the at least one first non-human sound source is reduced or not present.
However, Vinton, in the same field of endeavour, discloses:
wherein the sound components of the at least first sound source are sound components of at least one non-human sound source out of a plurality of non-human sound sources in an environment (Vinton, col 5, ln 61-65, In Scene 1, people are speaking on the deck of a ship. Background sounds include the lapping of waves and a distant fog horn at levels significantly below the speech level. The scene also includes a blast from the ship's horn, which is substantially louder than the speech),
wherein the other sound components of the one or more other sound sources are other sound components of one or more other non-human sound source out of the plurality of non-human sound sources (Vinton, col 5, ln 61-65, In Scene 1, people are speaking on the deck of a ship. Background sounds include the lapping of waves and a distant fog horn at levels significantly below the speech level. The scene also includes a blast from the ship's horn, which is substantially louder than the speech),
wherein the audio signal simultaneously comprises the sound components of the at least one first non-human sound source and the other sound components of the one or more other non-human sound sources in the environment (Vinton, col 5, ln 61-65, In Scene 1, people are speaking on the deck of a ship. Background sounds include the lapping of waves and a distant fog horn at levels significantly below the speech level. The scene also includes a blast from the ship's horn, which is substantially louder than the speech),
wherein an influence of the other sound components of the one or more other non-human sound sources on the estimate of the loudness of the sound components of the at least one first non-human sound source is reduced or not present (Vinton, col 2, ln 43-48, measure or estimate the loudness of speech in the selected interval... The selected interval should contain representative speech but not contain other types of audio material that would distort the loudness measurement).
Meléndez in view of Paulus and Vinton do not specifically disclose wherein such that the at least one output value indicates the estimate of the loudness of the sound components of the at least one first non-human sound source.
However, Azizi, in the same field of endeavour, discloses such that the at least one output value indicates the estimate of the loudness of the sound components of the at least one first non-human sound source (Azizi, 5th page, 4th para, an auditory habits determination unit 28 is provided, detecting the reacting of the user when an announcement signal is output together with an entertainment signal. It may happen that the user cannot correctly follow the announcement as the entertainment signal output together with the announcement signal is such that the level difference of the two signals is too small for a correct recognizability of the speech content of the announcement sound signal. As a reaction, the user may correct the sound signal output in such a way that either the volume of the entertainment is decreased and/or the volume of the announcement sound signal is increased. This user reaction is detected by the auditory habits determination unit. The auditory habits determination unit may comprise an artificial neural network, artificial meaning to distinguish this neural network from natural neural networks as they occur in brains. Neural networks are a technique for processing a plurality of inputs; Azizi, 4th page, 3rd-4th para, According to another aspect of the invention, the latter relates to a sound signal adjusting system being configured to adjust the loudness of the announcement sound signal relative to the loudness of the entertainment sound signal. The system comprises an entertainment loudness determination unit determining the loudness of the entertainment sound signal… The gain offset determination units may comprise gain offset tables containing the self-adaptive gain offset values in dependence on the loudness of the entertainment sound signal and in dependence on the volume level set by the user. As mentioned above, two entertainment gain offset tables may be provided, one having the self-adaptive offset values for a music content, the other having the self-adaptive gain offset values for the speech content. In the same way, two different announcement gain offset tables may be provided, one for the speech content of the entertainment signal and one for a music content of the entertainment signal; [i.e., “two entertainment gain offset tables may be provided, one having the self-adaptive offset values for a music content, the other having the self-adaptive gain offset values for the speech content” indicating how the loudness of the nonhuman sound (“music” being an example nonhuman sound here) computed]).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Azizi in the method of Meléndez in view of Paulus and Vinton because this would introduce a method for adjusting a loudness of an announcement sound signal relative to a loudness of an entertainment sound signal, and also the method has the benefit of being used in ear audio systems in combination with navigation systems (Azizi, Abstract).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Meléndez in view of Paulus, further in view of Vinton, further in view of Jia, further in view of Azizi, and further in view of Ward et al., "Estimating the loudness balance of musical mixtures using audio source separation." In Proceedings of the 3rd Workshop on Intelligent Music Production (WIMP 2017), 2017 (Ward II).
Regarding Claim 8, Meléndez in view of Paulus disclose the apparatus according to claim 5,
Furthermore, Meléndez in view of Paulus disclose:
wherein the neural network is configured to determine the at least one output value from the plurality of input values (Meléndez, page 275, 1st col, 4th para, Fig. 2 is a simplified representation of a TCN model where xt ∈ RNmels is the time-frame t of the input features, yt ∈ RNtcn is the output of the last residual block for that time-frame and yt ∈ R3 is the vector that carries the probability of the 3 relative music loudness estimation classes for that time-frame; [“xt …the input features” as “input values the plurality of samples of the audio signal”]).
Furthermore, Vinton discloses:
wherein the sound components of the at least one first sound source is a singing of one or more singers in the environment (Vinton, col 9, ln 60 – col 10, ln 6, The average squared l.sub.2-norm of the weighted spectral flux exploits the fact that speech normally has a rapidly varying spectrum. Speech signals usually have one of two forms: a tone-like signal referred to as voiced speech, or a noise-like signal referred to as unvoiced speech... Non-speech signals like music can also have rapid spectral changes but these changes are usually less frequent. Even vocal segments of music have less frequent changes because a singer will usually sing at the same frequency for some appreciable period of time; Vinton, col. 13, ln 21-25, Techniques that use the previously described features can detect speech in many types of audio material; however, these techniques will often make false detections in highly rhythmic audio material like so called "rap" and many instances of pop music),
Meléndez in view of Paulus, Vinton, Jia and Azizi do not specifically disclose wherein the other sound components of the one or more other sound sources are sound components of accompanying musical instruments, which accompany the singing of the one or more singers in the environment, wherein the audio signal simultaneously comprises the signing of the one or more singers and the sound components of the accompanying musical instruments, wherein the neural network is configured to determine the at least one output value from the plurality of input values, such that the at least one output value indicates the estimate of the loudness of the singing, wherein an influence of the sound components of accompanying musical instruments on the estimate of the loudness of the singing is reduced or not present.
However, Ward II, in the same field of endeavour discloses: wherein the other sound components of the one or more other sound sources are sound components of accompanying musical instruments, which accompany the singing of the one or more singers in the environment (Ward II, 1st page, 1st col, 2nd para, Loudness features in particular have received a great deal of interest in previous studies investigating the level-balancing and prioritization of instruments within a mix [2–6], especially for addressing common assumptions of knowledge-driven automatic mixers),
wherein the audio signal simultaneously comprises the signing of the one or more singers and the sound components of the accompanying musical instruments (Ward II, 2nd page, 4th para, Figure 1 shows, for each source, the estimated probability distribution, with a supplementary boxplot overlay, of the relative-to-mix loudness levels across the 100 songs of the DSD100 dataset. Note that it is possible for individual sources to be measured as louder than the mix, as background elements can lower the integrated mix loudness, relative to the loudest components. It can be seen that the vocals show the least inter-song variation, as the mix engineers generally prioritise this instrument group in the mix),
such that the at least one output value indicates the estimate of the loudness of the singing (Ward II, 2nd page, 4th col, Figure 1 shows, for each source, the estimated probability distribution, with a supplementary boxplot overlay, of the relative-to-mix loudness levels across the 100 songs of the DSD100 dataset),
wherein an influence of the sound components of accompanying musical instruments on the estimate of the loudness of the singing is reduced or not present (Ward II, 3rd page, 1st para, The underestimation of source-to-mix loudness is likely caused by the presence of cross-source interference that reduces the relative gating threshold used by ITU-RBS.1770, thereby lowering the absolute programme loudness. This effect was not observed for the mixture, however, as the sum of interferers contributes little additional increase in mix energy).
Claims 12 are rejected under 35 U.S.C. 103 as being unpatentable over Meléndez in view of Paulus in view of Rauhala Pat No. US 20130144615 A1 (Rauhala).
Regarding Claim 12, Meléndez in view of Paulus disclose the apparatus according to claim 1.
Meléndez in view of Paulus do not specifically disclose wherein the apparatus is configured to determine and output at least one other output value indicating an estimate of a partial loudness of the speech components of the audio signal, and wherein the partial loudness of the speech components of the audio signal depends on the loudness of the speech components of the audio signal and on the loudness of background components of the audio signal.
However, Rauhala, in the same field of endeavor, discloses:
wherein the apparatus is configured to determine and output at least one other output value indicating an estimate of a partial loudness of the speech components of the audio signal (Rauhala, para 0070, In some embodiments the loudness balancer 199 further comprises a speech detector 101, which is configured to analyse the frames to determine if the frame if speech frame or a non-speech frame. In such embodiments the speech detector is configured to analyse the audio signal to determine when frames are speech and non-speech or silent. In some embodiments for example where the audio signal is a downlink signal (an example of which is a cellular or mobile call downlink to the apparatus) where speech may be present it is known that typical speech can be divided into speech frames comprising energy partials or formants where harmonic relations can be found and silent portions),
wherein the partial loudness of the speech components of the audio signal depends on the loudness of the speech components of the audio signal and on the loudness of background components of the audio signal (Rauhala, para 0070, a loudness estimate is to be generated according to some embodiments the loudness estimate is determined in relation to the speech frames and not the whole audio signal as the whole audio signal may provide a low estimate. In some embodiments the speech detector 101 is inactive or not present; [i.e., “partial loudness of the speech components of the audio signal” as “the loudness estimate is determined in relation to the speech frames and not the whole audio signal”]).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Rauhala in the method of Meléndez in view of Paulus because this would enable using a loudness model to estimate a perceived loudness of the signal gain or similar digital signal processing parameters that can be adaptively adjusted (Rauhala, paras 0005).
Claims 17-19 is rejected under 35 U.S.C. 103 as being unpatentable over Meléndez in view of Paulus, and further in view of Zhang Pat App No. CN 111491176 A (Zhang).
Regarding Claim 17, Meléndez in view of Paulus disclose the apparatus according to claim 1.
wherein the neural network has been trained using a plurality of data training items (Meléndez, page 276, 2nd col, 2nd para, we train a set of TCNs through a grid search over the hyper-parameters described in Section 3.2: (1) the number of filters Ntcn, (2) the filter length L and (3) the dropout rate dr of the 1D-convolutional layers. (2) allows us to modify the receptive field of the TCN without affecting the model’s architecture; Meléndez, page 276, 1st col, 2nd para, This dataset comes with 10 predefined splits containing approximately 15% Foreground Music, 35% Background Music and 50% No Music. During training, we use nine of them: eight for the training set and one for the development set. The tenth split constitutes the testing set).
Meléndez in view of Paulus do not specifically disclose wherein each of the plurality of data training items comprises one of a plurality of audio training signal portions and one or more reference loudness values.
However, Zhang, in the same field of endeavor, discloses:
wherein each of the plurality of data training items comprises one of a plurality of audio training signal portions and one or more reference loudness values (Zhang, 6th page, 11th para, the large amount of training original audio input to the pre-constructed neural network model to obtain the training original audio prediction loudness information; according to the prediction loudness information and the actual loudness information of the training original audio ).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Zhang in the method of Meléndez in view of Paulus because this would enable to improve the accuracy of the original loudness information and the explanation loudness information (Zhang, 6th page, 11th para).
Regarding Claim 18, Meléndez in view of Paulus, and further in view of Zhang disclose the apparatus according to claim 17,
wherein the neural network has been trained depending on a loss function,
wherein, to determine a return value of the loss function during training, the neural network is configured to determine one or more loudness value estimates of the audio training signal portion for each of one or more data training items of the plurality of data training items (Meléndez, page 273, 2nd col, 2nd para, We train 40 TCN and 80 CNN-TCN models with two grid searches over several hyper-parameters, and compare them among themselves and with the two best algorithms submitted to the tasks of music detection and relative music loudness estimation in MIREX 2019; Meléndez, page 276, 1st col, 5th para – 2nd col, 2nd para, Originally, M1 and M2 are two-neuron output regression algorithms that adapt to classification through a set of thresholds .. We train them for 100 epochs using the ADAM optimizer with learning rate lr = 0.001, and the categorical cross-entropy loss function… The training process has two steps: first, we train a set of TCNs through a grid search over the hyper-parameters described in Section 3.2: (1) the number of filters Ntcn, (2) the filter length L and (3) the dropout rate dr of the 1D-convolutional layers. (2) allows us to modify the receptive field of the TCN without affecting the model’s architecture. With (1) and (3) we experiment with the learning capacity of the network and its regularization, respectively. We train 40 models using the following set of values for each hyper-parameter), and
Furthermore, Zhang discloses:
wherein the neural network has been trained depending on the loss function such that a return value of the loss function depends on the one or more loudness value estimates of the audio training signal portion and on the one or more reference loudness values of each of the one or more data training items (Zhang, 4th page, 3rd – 4th para, the original video original audio and explanation audio as input parameters, input to the pre-trained deep learning model, according to the output result of the deep learning model, determining the loudness difference between the original audio and the audio. wherein the machine network model is obtained based on a large number of original audio and corresponding explanation audio training. Illustratively, the deep learning model may be a twinborn network model. Alternatively, optionally, the original audio and explanation audio as input parameters, respectively input to each trained machine learning model to obtain the original loudness information and the loudness information. wherein the machine learning model is obtained according to a large amount of audio training sample and loudness information corresponding to the audio training sample; [i.e., “the loudness difference between the original audio and the audio” indicating a “loss”] ).
Regarding Claim 19, Meléndez in view of Paulus, further in view of Zhang disclose the apparatus according to claim 18.
Furthermore, Zhang discloses:
wherein one of the one or more reference loudness values of a data training item of the one or more data training items indicates a loudness of the signal components of interest of the audio training signal portion of the data training item, and wherein one of the one or more loudness value estimates of the data training item indicates an estimate of said loudness of the signal components of interest of the audio training signal portion of the data training item by the neural network (Zhang, 6th page, 11th para, the large amount of training original audio input to the pre-constructed neural network model to obtain the training original audio prediction loudness information; according to the prediction loudness information and the actual loudness information of the training original audio, optimizing the model parameter of the neural network model, obtaining the trained original audio model. correspondingly, the original audio as the input parameter, input to the original audio model, obtaining original loudness information corresponding to the original audio
wherein one of the one or more reference loudness values of a data training item of the one or more data training items indicates a loudness of the other signal components of the audio training signal portion of the data training item, and wherein one of the one or more loudness value estimates of the data training item indicates an estimate of said loudness of the other signal components of the audio training signal portion of the data training item by the neural network; and/or
wherein one of the one or more reference loudness values of a data training item of the one or more data training items indicates a loudness of the entire audio training signal portion of the data training item, and wherein one of the one or more loudness value estimates of the data training item indicates an estimate of said loudness of the entire audio training signal portion of the data training item by the neural network; and/or
wherein one of the one or more reference loudness values of a data training item of the one or more data training items indicates a loudness of the audio training signal portion of the data training item when speech is present, and wherein one of the one or more loudness value estimates of the data training item indicates an estimate of said loudness of the audio training signal portion of the data training item by the neural network when speech is present, and/or
wherein one of the one or more reference loudness values of a data training item of the one or more data training items indicates a partial loudness of the signal components of interest the audio training signal portion of the data training item, and wherein one of the one or more loudness value estimates of the data training item indicates an estimate of said partial loudness of the signal components of interest of the audio training signal portion of the data training item by the neural network.
Claims 20-21 are rejected under 35 U.S.C. 103 as being unpatentable over Meléndez in view of Paulus in view of Zhang, and further in view of Jia et al., "Hierarchical Regulated Iterative Network for Joint Task of Music Detection and Music Relative Loudness Estimation," in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1-13, 2021 (Jia).
Regarding Claim 20, Meléndez in view of Paulus and Zhang disclose the apparatus according claim 18.
Furthermore, Zhang discloses:
wherein Loss indicates the return value of the Loss function, wherein estimatei indicates one of the one or more loudness value estimates of an i-th data training item of the one or more data training items, wherein referencei indicates one of the one or more reference loudness values of the i-th data training item of the one or more data training items (Zhang, 6th page, 3rd – 8th para, the operation of determining the original video of the original audio and the loudness difference between the audio is refined as " according to the original audio, determining the original loudness information of the original audio, and according to the explanation audio, determining the explanation loudness information of the audio; determining the loudness difference between the original loudness information and the explanation loudness information, so as to improve the determining mechanism of loudness difference. …Specifically, the large amount of training original audio and training audio as the training sample, the pre-constructed neural network model for training, obtaining the prediction loudness information of each training sample; according to the prediction loudness information of each training sample and the corresponding actual loudness information, optimizing and adjusting the model parameter of the neural network model, so as to realize the model training of the neural network model).
Meléndez in view of Paulus and Zhang do not specifically disclose wherein the loss function is defined according to
L
o
s
s
=
1
N
∑
i
=
1
N
e
s
t
i
m
a
t
e
i
-
r
e
f
e
r
e
n
c
e
i
p
wherein Loss indicates the return value of the Loss function, wherein estimatei indicates one of the one or more loudness value estimates of an i-th data training item of the one or more data training items, wherein referencei indicates one of the one or more reference loudness values of the i-th data training item of the one or more data training items, wherein p ≥ 1, and wherein N ≥1
However, Jia, in the same field of endeavour, discloses wherein the loss function is defined according to
L
o
s
s
=
1
N
∑
i
=
1
N
e
s
t
i
m
a
t
e
i
-
r
e
f
e
r
e
n
c
e
i
p
wherein Loss indicates the return value of the Loss function,
wherein estimatei indicates one of the one or more loudness value estimates of an i-th data training item of the one or more data training items,
wherein referencei indicates one of the one or more reference loudness values of the i-th data training item of the one or more data training items,
wherein p ≥ 1, and wherein N ≥1 (Jia, 7th page, 3rd para, We use mean squared error to ensure this equality and … loss:
PNG
media_image2.png
60
370
media_image2.png
Greyscale
where N means that we have N samples and subscript i denotes the i th sample. Ti is the time length of the i th sample and t represents the tth time-step).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Jia in the method of Meléndez in view of Paulus and Zhang because this would solve the practical requirement of the music copyright management problem which is the estimation of music relative loudness that is mostly ignored in existing music detection works through a study of the joint task of music detection and music relative loudness estimation (Jia, Abstract).
Regarding Claim 21, Meléndez in view of Paulus in view of Zhang disclose the apparatus according to claim 18.
Meléndez in view of Paulus in view of Zhangdo not specifically disclose wherein the neural network has been trained by iteratively adjusting the plurality of weights of the plurality of neural nodes of the neural network, wherein, in each iteration step of a plurality of iteration steps, the plurality of weights of the plurality of neural nodes of the neural network has been adjusted depending on one or more errors returned by the loss function in response to receiving the one or more data training items.
However, Jia, in the same field of endeavour, discloses the wherein the neural network has been trained by iteratively adjusting the plurality of weights of the plurality of neural nodes of the neural network,
wherein, in each iteration step of a plurality of iteration steps, the plurality of weights of the plurality of neural nodes of the neural network has been adjusted depending on one or more errors returned by the loss function in response to receiving the one or more data training items (Jia, page 3, 2nd col, 2nd para, In the above works, none of them has applied the hierarchical classification to the joint task of music detection and music relative loudness estimation. In this work, we reformulate this joint task as Hierarchical Event Detection and Localization (HEDL)
problem. Based on this, we propose the Hierarchical Regulated
Iterative Network (HRIN) to solve this problem. Specifically,
we design HRIN to capture relationships between event classes
and to regulate the predictions to obey the hierarchical structure; Jia, 4th page, 8th para – 2nd col, 6th para, We propose the Hierarchical Regulated Iterative Network (HRIN), a two-output deep neural network specifically designed
to solve the Hierarchical Event Detection and Localization
(HEDL) problem… In the rest of this subsection, we give the mathematical description of one iterative block (the one at time-step t) only; other iterative blocks can be computed in the same way using the same network module with the same parameters… In the first path, x[t] is first computed by a recurrent unit whose basic computation cell is GRU [44]. A GRU cell has two inputs and two outputs: it takes in the current external input x[t] and the last hidden state h[t − 1], and it exports the current output o[t] and the current hidden state h[t]. Inside a GRU cell, there are four computation steps: first, compute the reset gate r[t] using weight matrix Wr and bias vector br; second, compute the update gate z[t] using weight matrix Wz and bias vector bz; third, calculate the reset hidden state hˆ[t] using weight matrix Whˆ and bias vector bhˆ ; fourth, update the hidden state and obtain the current one h[t]. At last, the current output o[t] is computed from the current hidden state h[t] using weight matrix Wo and bias vector bo. To depict the GRU cell specifically for the first path, we add the superscript “1” for every notation. Let o1[t] denote the output of the first path’s recurrent unit with only one GRU cell and is given by:
PNG
media_image3.png
154
332
media_image3.png
Greyscale
PNG
media_image4.png
76
391
media_image4.png
Greyscale
; Jia, 4th page, 1st col., 8th para, HRIN propagates gradients from the two network outputs–each one corresponds to each hierarchical level. A corresponding loss function to each output is used for back-propagating the gradients from the event classes in the corresponding level; [i.e., for instance, compute the update gate z[t] using weight matrix Wz and bias vector bz adjusting model weights of a NN using an error or loss function]).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Jia in the method of Meléndez in view of Zhang because this would solve the practical requirement of the music copyright management problem which is the estimation of music relative loudness that is mostly ignored in existing music detection works through a study of the joint task of music detection and music relative loudness estimation (Jia, Abstract).
Claims 34 are rejected under 35 U.S.C. 103 as being unpatentable over Meléndez in view of Paulus in in view of Jia, and further view of Nielsen et al. Pat No. US 8175282 B2 (Nielsen) ).
Regarding Claim 34, Meléndez in view of Paulus, and further in view of Jia discloses the apparatus according to claim 29.
Furthermore Jia teaches:
wherein the apparatus is configured to employ linear activation in the output layer of the neural network (Jia, 8th page, 3rd para, We design one convolutional unit
as three 1-D convolutional layers. Each convolutional layer is followed by a batch normalization layer [51], and a max-pooling layer.).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Jia in the method of Meléndez in view of Paulus because this would solve the practical requirement of the music copyright management problem which is the estimation of music relative loudness that is mostly ignored in existing music detection works through a study of the joint task of music detection and music relative loudness estimation (Jia, Abstract).
Meléndez in view of Paulus in view of Jia do not specifically disclose linear activation in the output layer of the neural network.
However, Nielsen, in the same field of endeavor, discloses linear activation in the output layer of the neural network (Nielsen, col. 11, ln 43-49, To estimate the loudness value based on the feature set an artificial neural network is employed. The applied network comprises a multi-layer perceptron type having a tan-sigmoid activation function for the units in the single hidden layer and, moreover, it comprises a single output unit with a linear activation function ).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Nielsen in the method of Meléndez Meléndez in view of Paulus iand Jia because this would enable an automated loudness estimation of audio signals that is highly needed for different purposes such as automatic gain control in relation to broadcasting or reproduction of audio signals in a car (Nielsen, col. 1, ln 20-23).
Claims 40 is rejected under 35 U.S.C. 103 as being unpatentable over Meléndez in view of Paulus, and further in view of Dipling et al. Pat App No. EP 0624866 A2 (Dipling).
Regarding Claim 40, Meléndez in view of Paulus discloses the apparatus according to claim 35.
Meléndez in view of Paulus does not disclose wherein a first group of two or more of the plurality of spectral samples relate to a first group of frequency bands, which each exhibit a bandwidth that deviates by no more than 10 % from a predefined first bandwidth, and wherein a second group of two or more of the plurality of spectral samples relate to a second group of frequency bands, which each exhibit a higher center frequency than each frequency band of the first group of frequency bands, and which each exhibit a bandwidth being higher than the bandwidth of each frequency band of the first group.
However, Dipling, in the same field of endeavor, discloses:
wherein a first group of two or more of the plurality of spectral samples relate to a first group of frequency bands, which each exhibit a bandwidth that deviates by no more than 10 % from a predefined first bandwidth (Dipling, 6th page, 3rd para, The deviation of the bandwidths from the frequency group widths according to Zwicker is sufficiently small even with a relatively coarse frequency grid of 512 discrete frequencies (example in FIG.9 for 44.1 kHz sampling frequency) ),
wherein a second group of two or more of the plurality of spectral samples relate to a second group of frequency bands, which each exhibit a higher center frequency than each frequency band of the first group of frequency bands, and which each exhibit a bandwidth being higher than the bandwidth of each frequency band of the first group (Dipling, 2nd page, 2nd para, with increasing the center frequency also increases its bandwidth).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Dipling in the method of Meléndez in view of Paulus because this would enable to adapt the frequency analysis to the properties of the human ear as a sound receiver which is expedient and advantageous and it is an essential finding of psychoacoustics that the ear jointly evaluates sound events that fall within a certain bandwidth, the so-called frequency group width Dipling, 2nd page, 2nd para
Claims 41 are rejected under 35 U.S.C. 103 as being unpatentable over Meléndez in view of Paulus, further in view of Ward et al., "An efficient time-varying loudness model," in 2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, pp. 1-4. IEEE, 2013 (Ward III), and further in view of Dipling.
Regarding Claim 41, Meléndez in view of Paulus discloses the apparatus according to claim 40.,Meléndez does not specifically disclose wherein a third group of two or more of the plurality of spectral samples relate to a third group of frequency bands, which each exhibit a higher center frequency than each frequency band of the second group of frequency bands, which each exhibit a bandwidth being higher than the bandwidth of each frequency band of the second group.
However, Ward III, in the same field of endeavor, discloses:
wherein a third group of two or more of the plurality of spectral samples relate to a third group of frequency bands, which each exhibit a higher center frequency than each frequency band of the second group of frequency bands, which each exhibit a bandwidth being higher than the bandwidth of each frequency band of the second group (Ward III, 2nd page, 1st col – 2nd col, 1st para, Cassidy and Smith [6] proposed the Hopping Goertzel DFT (HGDFT) to address the redundancy that occurs when parallel FFTs are used to compute the spectrogram.…The multi-resolution DFT of the TVLM comprises six Hanning windowed segments of durations 64, 32, 16, 8, 4 and 2 ms used to compute the power spectrum in frequency bands 20 to 80 Hz, 80 to 500 Hz, 500 to 1250 Hz, 1250 to 2540 Hz, 2540 to 4050 Hz and 4050 to 15 000 Hz, respectively….Nb corresponds to the window length in band b. The HGDFT is simply the SGDFT when y[n] is computed every M samples, with M > 1, here set to 1 ms (to the nearest sample)).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Ward III in the method of Meléndez in view of Paulus because this would enable to reduce the influence of spectral leakage on loudness estimates at low frequencies windowing is performed efficiently in the frequency domain by convolving adjacent spectral samples with the DFT of the Hanning window (Ward III, 2nd page, 3rd para).
Meléndez in view of Paulus and Ward III do not specifically disclose wherein the bandwidth of each frequency band of the third group deviates less from an equivalent rectangular bandwidth than the bandwidth of each frequency band of the second group.
However, Dipling, in the same field of endeavor, discloses:
wherein the bandwidth of each frequency band of the third group deviates less from an equivalent rectangular bandwidth than the bandwidth of each frequency band of the second group (Dipling, 6th page, 3rd para, The deviation of the bandwidths from the frequency group widths according to Zwicker is sufficiently small even with a relatively coarse frequency grid of 512 discrete frequencies (example in FIG.9 for 44.1 kHz sampling frequency)).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Dipling in the method of Meléndez in view of Paulus and Ward III because this would enable to adapt the frequency analysis to the properties of the human ear as a sound receiver which is expedient and advantageous and it is an essential finding of psychoacoustics that the ear jointly evaluates sound events that fall within a certain bandwidth, the so-called frequency group width (Dipling, 2nd page, 2nd para).
Claims 44 are rejected under 35 U.S.C. 103 as being unpatentable over Meléndez in view of Paulus, and further in view of Rauhala Pat App No. US 20180269841 A1 (Rauhala II).
Regarding Claim 44, Meléndez in view of Paulus disclose the
Meléndez in view of Paulus do not specifically disclose wherein the signal processor is configured to modify the audio
However, Rauhala II, in the same field of endeavour, discloses wherein the signal processor is configured to modify the audio input signal depending on the estimate of the loudness of the speech components of the audio input signal and depending on an estimation of the loudness of the background components of the audio input signal to acquire the audio output signal (Rauhala II, para 0057, The implemented program codes comprise a loudness balancer for processing an audio signal and thus balance the loudness to the desired level; Rauhala II, para 0003, it can be common experience that speech audio during a phone call may have a significantly lower perceived loudness than a preceding music audio signal, which causes the user to increase the volume level in order to increase the loudness of the voice. However when the phone call ends and the user returns to listening to music with a higher loudness, this event can startle the user and require the user to reduce the volume ).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Rauhala II in the method of Meléndez in view of Paulus because this would enable processing an audio signal based on an estimated loudness and also introduce a loudness balancer for processing an audio signal and thus balance the loudness to the desired level (Rauhala II, para 0057).
Claims 45 are rejected under 35 U.S.C. 103 as being unpatentable over Meléndez in view of Paulus, further in view of Rauhala II, and further in view of Rauhala.
Regarding Claim 45, Meléndez in view of Paulus and Rauhala II disclose the apparatus according to claim 44.
Meléndez in view of Paulus and Rauhala II do not specifically disclose wherein the apparatus for providing an estimate of a loudness of speech components of the audio
However, Rauhala, in the same field of endeavor, discloses:
wherein the apparatus for providing an estimate of a loudness of speech components of the audio input signal is an apparatus configured to determine and output at least one other output value indicating an estimate of a partial loudness of the speech components of the audio signal (Rauhala, para 0070, a loudness estimate is to be generated according to some embodiments the loudness estimate is determined in relation to the speech frames and not the whole audio signal as the whole audio signal may provide a low estimate. In some embodiments the speech detector 101 is inactive or not present; [i.e., “partial loudness of the speech components of the audio input signal” as “the loudness estimate is determined in relation to the speech frames and not the whole audio signal”]),
wherein the partial loudness of the speech components of the audio signal depends on the loudness of the speech components of the audio signal and on the loudness of background components of the audio signal (Rauhala, para 0070, a loudness estimate is to be generated according to some embodiments the loudness estimate is determined in relation to the speech frames and not the whole audio signal as the whole audio signal may provide a low estimate),
wherein the signal processor is configured to modify a level of the audio input signal depending on the partial loudness of the speech components of the audio signal (Rauhala, para 0028, The loudness estimator may comprise a signal processor configured to apply at least one loudness model to the first audio signal).
Therefore, it would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the method of Rauhala in the method of Meléndez in view of Paulus and Rauhala II because this would enable using a loudness model to estimate a perceived loudness of the signal gain or similar digital signal processing parameters that can be adaptively adjusted (Rauhala, paras 0005).
Allowable Subject Matter
Claims 13-16 are objected to as being dependent upon rejected base claims, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and also if all these claims overcome the 101 rejections. The reasons for allowance are that the prior art of record do not specifically teach the limitations as recited in the mentioned claims.
Claims 22-28 are objected to as being dependent upon rejected base claims, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The reasons for allowance are that the prior art of record do not specifically teach the limitations as recited in the mentioned claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MULUGETA T. DUGDA whose telephone number is (703)756-1106. The examiner can normally be reached Mon - Fri, 4:30am - 7:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras D. Shah can be reached at 571-270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MULUGETA TUJI DUGDA/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
04/05/2026
Prosecution Insights