DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
The present application is being examined under the claims filed 05/14/2026.
Claims 16-31 are pending.
Information Disclosure Statement
The information disclosure statement filed 02/17/2026 fails to comply with 37 CFR 1.98(a)(3)(i) because it does not include a concise explanation of the relevance, as it is presently understood by the individual designated in 37 CFR 1.56(c) most knowledgeable about the content of the information, of each reference listed that is not in the English language. It has been placed in the application file, but the information referred to therein has not been considered. None of the references provided contained English explanations and could not be considered.
The information disclosure statement filed 05/14/2026 fails to comply with 37 CFR 1.98(a)(2), which requires a legible copy of each cited foreign patent document; each non-patent literature publication or that portion which caused it to be listed; and all other information or that portion which caused it to be listed. It has been placed in the application file, but the information referred to therein has not been considered. None of the foreign patent references listed were present in the application file and could not be considered.
Response to Amendment
This Office Action is in response to Applicant’s communication filed 05/14/2026 in response to office action mailed 01/14/2026. The Applicant’s remarks and any amendments to the claims or specification have been considered with the results that follow.
Response to Arguments
Regarding 35 U.S.C. 112(b)
In Remarks page 7-8, Argument 1
(Examiner summarizes Applicant’s argument) Applicant argues that the claims have been amended thus obviating the rejections under 35 U.S.C. 112.
Examiner’s response to Argument 1
While some of the rejections under 35 U.S.C. 112(b) have been overcome by the amendments, the rejections regarding the term “adjacent” remain. While the new amendments do specify a point to which frequencies are compared, it remains unclear which frequencies would be adjacent to others. For example, the question remains whether a frequency band with frequency labeled 445 hz would be adjacent to a frequency band with frequency labeled 450 hz. It remains unclear to what degree a frequency must be close to the other to satisfy the term “adjacent”. Traditionally, the term “adjacent” would refer to an ordered set of discrete values or contiguous intervals, however no such ordered set is defined by the claims. Accordingly, the claims remain indefinite under 35 U.S.C. 112(b).
Regarding 35 U.S.C. 101
In Remarks page 8-9, Argument 2
(Examiner summarizes Applicant’s arguments) Applicant argues that the office mischaracterized the following limitations as mental processes, stating that they could not be performed via an opinion.
predict a first set of output variables representing a time predicted frequency band of a current time frame given coefficients of one or several previous time frames, and
predict a second set of output variables representing a frequency predicted frequency band given coefficients of one or several frequency bands that are adjacent to and lower than the frequency predicted frequency band
Examiner’s response to Argument 2
Applicant’s arguments are not convincing. Predicting the output variables as claimed amounts to predicting certain kinds of numerical values based on given data. Examiner maintains that the human mind is capable of making numerical predictions as it is a matter of forming an opinion about what those numbers ought to be, based on the available data.
In Remarks page 9, Argument 3,
(Examiner’s response to Argument 3) Applicant argues that the office mischaracterized the output stage limitations as mere data gathering, stating that the limitation recites an improvement to technology when the claim is considered as a whole. Applicant recites portions of the specification which states the potential uses for the invention.
Examiner’s response to Argument 4
The claim is not directed to a technological improvement because the components or steps for how the invention would realize the alleged improvements are not present in the claims. The outputting step merely provides the values output from a neural network, but does not provide the details for how the output values were produced (e.g. any particular neural network architecture or training step), nor the details of how the output is used to improve technology (see MPEP 2106.05(a) “That is, the claim must include the components or steps of the invention that provide the improvement described in the specification.”). Consider Recentive Analytics v. Foxcorp (Recentive). In Recentive, the type of output was specified however the details of the model were not provided in the claims and the court concludes that (page 18) “Today, we hold only that patents that do no more than claim the application of generic machine learning to new data environments, without disclosing improvements to the machine learning models to be applied, are patent ineligible under § 101.”
Further, consider ex parte Desjardins (Desjardins). The claims in Desjardins were found to be eligible because they recited specific machine learning training that improved over ordinary machine learning training. By contrast, claim 1 of the instant application merely generically recites inputting and outputting of data with neural networks without the details of how these models operate or are trained.
Example 48 claim 2 of the July 2024 guidance was also deemed eligible for at least the following reason
(page 24) While on their own steps (b)-(e) recite judicial exceptions, steps (f) and (g) are directed to creating a new speech signal that no longer contains extraneous speech signals from unwanted sources. The claimed invention reflects this technical improvement by including these features.
By contrast, the outputting step of the instant application merely recite outputting data from a neural network (see also Example 47 claim 2 element f in the July 2024 examples, which was deemed to be mere instructions to apply). Mere inputting and outputting of data in a neural network environment is not sufficient to integrate the judicial exception nor amount to significantly more as evidenced by the examples and case law provided.
In Remarks page 9-10, Argument 5
(Examiner summarizes Applicant’s arguments) Applicant argues that the neural network of claim 26, as used in conjunction with the output stage recited in claim 16, may be used in general audio coding and would resolve spectral holes and quantization errors.
Examiner’s response to Argument 5
Similar to the arguments related to claim 16 above, Claim 26 does not include the components or steps necessary for filling spectral holes nor cleaning up quantization errors. That is, a person having ordinary skill in the art would recognize the additional elements as recited in claim 26 as generic machine learning components, which would not evidently provide the improvements applicant alleges.
In Remarks pages 10-11, Argument 6
(Examiner summarizes Applicant’s arguments) Applicant argues that Vasquez fails to teach the specific architecture of the “output stage” as recited in the claims and instead that the office action merely points to a general conclusory statement of Vasquez.
Examiner’s response to Argument 6
Examiner disagrees. The steps provided by Vasquez already recited to teach other portions of the claims show that the output of the model would be “a set of frequency coefficients representing a specific frequency band of said current time frame, based on said first and second set of output variables, said specific frequency band being at least one of the time predicted and frequency predicted frequency band”. Vasquez accomplishes this by using a neural network to predict the first and second output variables, and produce a spectrogram (i.e. a set of frequency coefficients representing a specific frequency band of a specific timeframe) as output. “Existing generative models for audio have predominantly focused on modelling these time-domain signals directly. We instead model spectrograms, which are two-dimensional time-frequency representations which contain information about how the frequency content of an audio signal varies through time”.
Moreover, according to Applicant’s specification: “Based on the above, it is therefore an object of the present invention to provide an improved generative model for general media, and in particular general audio, i.e. not only specific categories of audio, like speech or piano music, but audio in general.” That is, the model of Applicant’s disclosure is configured to produce audio as output. This is the same as what Vazquez does (page 1 abstract) “We apply our model to a variety of audio generation tasks, including unconditional speech generation, music generation, and text-to-speech synthesis—showing improvements over previous approaches in both density estimates and human judgments.” The rejection is maintained accordingly.
In Remarks page 11, Argument 7
(Examiner summarizes Argument 7) Applicant argues that neither Vasquez nor Martinez Ramirez teach the plurality of bins as recited in amended claim 16, and the cited portion of Martinez Ramirez merely states “that an optimal filter bank decomposition is learned, and makes no mention of provision of the frequency coefficients that correspond to a plurality of bins of a specific frequency band, as recited in amended claim 16.”
Examiner’s response to Argument 7
Examiner disagrees. The filter bank representation of Martinez Ramirez are frequency coefficients that correspond to a plurality of bins. Filter banks are collections of filters which decompose an input signal into a plurality of frequency ranges (which can be interpreted as bins). In fact, Martinez Ramirez explicitly states (page 300 column 1 paragraph 2) “From the feature map matrix X1 (see Fig. 2c), the filters W1 are actively acting as a filter bank or frequency selectors[*Examiner notes: mapped to bins], since X1 correspond to the decomposition of the input data into different frequencies.”
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 16-31 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Regarding Claim 16
The term “adjacent” in claim 16 is a relative term which renders the claim indefinite. The term “adjacent” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term “adjacent” indicates discrete values or a range with a clear ordering relative to a particular point. However, neither the claim nor the specification provide a standard range or set of discrete values to ascertain what is meant by “adjacent”.
Regarding Claim 17
The term “adjacent” in claim 17 is a relative term which renders the claim indefinite. The term “adjacent” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term “adjacent” indicates discrete values or a range with a clear ordering relative to a particular point. However, neither the claim nor the specification provide a standard range or set of discrete values to ascertain what is meant by “adjacent”.
Regarding Claim 20
The term “adjacent” in claim 20 is a relative term which renders the claim indefinite. The term “adjacent” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term “adjacent” indicates discrete values or a range with a clear ordering relative to a particular point. However, neither the claim nor the specification provide a standard range or set of discrete values to ascertain what is meant by “adjacent”.
Regarding Dependent Claims
Claims
17-31 are dependent upon claim 16
21 and 26-31 are dependent upon claim 20
and are therefore similarly rejected for including the deficiencies of claim 16, 21, and 23 respectively.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 16-31 are rejected under 35 U.S.C. 101 for containing an abstract idea without significantly more.
Regarding Claim 16:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes, the claim is to a machine.
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
a time predicting portion including […] predict a first set of output variables representing a time predicted frequency band of a current time frame given coefficients of one or several previous time frames — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to formulating an opinion about a time of a frequency band based on given data.
and a frequency predicting portion including […] predict a second set of output variables representing a frequency predicted frequency band given coefficients of one or several frequency bands that are adjacent to and lower than the frequency predicted frequency band in said current time frame — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to formulating an opinion of a frequency based on given data.
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, the claim does not recite additional elements that integrate the judicial exception into a practical application. The additional elements:
A computer implemented neural network system for predicting frequency coefficients of a media signal, the neural network system comprising: […] at least one neural network trained to — This limitation is directed to merely applying an abstract idea using a generic computer with a neural network as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
an output stage configured to provide a set of frequency coefficients representing a specific frequency band of said current time frame, based on said first and second set of output variables, said specific frequency band being at least one of the time predicted and frequency predicted frequency band, and wherein a) said first set of output variables, predicted by the time predicting portion, is used as input variables to the frequency predicting portion, or b) said second set of output variables, predicted by the frequency prediction portion, is used as input variables to the time predicting portion — This limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g)).
wherein the specific frequency band comprises a plurality of bins, and wherein the set of frequency coefficients comprises coefficients for the plurality of bins of the specific frequency band — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the frequency coefficients to bins of a specific frequency band.
Step 2B – Does the claim recite additional elements that amount to significantly more than the abstract idea itself?
No, the claim does not recite additional elements which amount to significantly more than the abstract idea itself. The additional elements as identified in step 2A prong 2:
A computer implemented neural network system for predicting frequency coefficients of a media signal, the neural network system comprising: […] at least one neural network trained to — Using a generic computer and neural network as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
an output stage configured to provide a set of frequency coefficients representing a specific frequency band of said current time frame, based on said first and second set of output variables, said specific frequency band being at least one of the time predicted and frequency predicted frequency band, and wherein a) said first set of output variables, predicted by the time predicting portion, is used as input variables to the frequency predicting portion, or b) said second set of output variables, predicted by the frequency prediction portion, is used as input variables to the time predicting portion — This limitation is recited at a high level of generality and amounts to mere data gathering of transmitting and receiving data over a network, which is well-understood, routine, and conventional activity (see MPEP 2106.05(d) II.), which cannot amount to significantly more than the judicial exception.
wherein the specific frequency band comprises a plurality of bins, and wherein the set of frequency coefficients comprises coefficients for the plurality of bins of the specific frequency band — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Regarding Claim 17
Claim 17 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 16 which included an abstract idea (see rejection for claim 16). The claim recites the additional limitations:
Step 2A Prong 2:
wherein a) said first set of output variables, predicted by the time predicting portion, is used as input variables to the frequency predicting portion, and said time predicted frequency band is adjacent to said frequency predicted frequency band in said current time frame — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the particular data operated on by the predictions.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein a) said first set of output variables, predicted by the time predicting portion, is used as input variables to the frequency predicting portion, and said time predicted frequency band is adjacent to said frequency predicted frequency band in said current time frame — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 18
Claim 18 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 16 which included an abstract idea (see rejection for claim 16). The claim recites the additional limitations:
Step 2A Prong 2:
wherein b) said second set of output variables, predicted by the frequency prediction portion, is used as input variables to the time predicting portion, and said time predicted frequency band and said frequency predicted frequency band are a same frequency band in a previous time frame and current time frame respectively — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the particular data operated on by the predictions.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein b) said second set of output variables, predicted by the frequency prediction portion, is used as input variables to the time predicting portion, and said time predicted frequency band and said frequency predicted frequency band are a same frequency band in a previous time frame and current time frame respectively — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 19
Claim 19 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 16 which included an abstract idea (see rejection for claim 16). The claim recites the additional limitations:
Step 2A Prong 2:
wherein said first set of output variables, predicted by the time predicting portion, are used as input variables to the frequency predicting portion — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the particular data operated on by the predictions.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein said first set of output variables, predicted by the time predicting portion, are used as input variables to the frequency predicting portion — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 20
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 16 which included an abstract idea (see rejection for claim 16). The claim recites the additional limitations:
Step 2A Prong 1:
predict an intermediate set of output variables representing the current time frame, given a first set of input variables representing a preceding time frame of the media signal — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to formulating an opinion about a time using given data.
predict said first set of output variables, wherein variables in the intermediate set are formed by mixing variables in said intermediate set representing said time predicted frequency band and a plurality of frequency bands adjacent to said time predicted frequency band — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to evaluating data using known data processing.
Step 2A Prong 2:
wherein the time predicting portion includes: a time predicting recurrent neural network comprising a plurality of neural network layers, said time predicting recurrent neural network being trained to — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the neural network configuration.
and a band mixing neural network trained to — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the neural network configuration.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein the time predicting portion includes: a time predicting recurrent neural network comprising a plurality of neural network layers, said time predicting recurrent neural network being trained to — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
and a band mixing neural network trained to — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 21
Claim 21 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 20 which included an abstract idea (see rejection for claim 20). The claim recites the additional limitations:
Step 2A Prong 1:
predict said first set of input variables given frequency coefficients of a preceding time frame of said media signal — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to formulating an opinion of what input variables should be based on given data.
Step 2A Prong 2:
wherein the time predicting portion further includes: an input stage comprising a neural network trained to — This limitation is directed to merely applying an abstract idea using a generic computer with a neural network as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein the time predicting portion further includes: an input stage comprising a neural network trained to — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 22
Claim 22 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 19 which included an abstract idea (see rejection for claim 19). The claim recites the additional limitations:
Step 2A Prong 1:
predict said second set of output variables, given a sum of said first set of output variables and a second set of input variables representing lower frequency bands of the current time frame — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to formulating an opinion on what the output variables should be based on given data.
Step 2A Prong 2:
wherein the frequency predicting portion includes: a frequency predicting recurrent neural network comprising a plurality of neural network layers, said frequency predicting neural network being trained to — This limitation is directed to merely applying an abstract idea using a generic computer with a neural network as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein the frequency predicting portion includes: a frequency predicting recurrent neural network comprising a plurality of neural network layers, said frequency predicting neural network being trained to — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 23
Claim 23 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 22 which included an abstract idea (see rejection for claim 22). The claim recites the additional limitations:
Step 2A Prong 2:
wherein the frequency predicting portion further includes: one or several output layers trained to — This limitation is directed to merely applying an abstract idea using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
provide said set of frequency coefficients based on said second set of output variables — This limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g)).
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein the frequency predicting portion further includes: one or several output layers trained to — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
provide said set of frequency coefficients based on said second set of output variables — This limitation is recited at a high level of generality and amounts to mere data gathering of transmitting and receiving data over a network, which is well-understood, routine, and conventional activity (see MPEP 2106.05(d) II.), which cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 24
Claim 24 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 16 which included an abstract idea (see rejection for claim 16). The claim recites the additional limitations:
Step 2A Prong 1:
wherein each frequency coefficient is represented by a set of distribution parameters, wherein said set of distribution parameters are configured to parametrize a probability distribution of the coefficient, wherein said specific frequency band of said current time frame is obtained by sampling the probability distribution of each frequency coefficient — This limitation is directed to the abstract idea of a mathematical process, and mathematical calculations in particular (MPEP 2106.04(a)(2) I. C.). The claim describes the mathematical operations of parametrizing and sampling a probability distribution.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 25
Claim 25 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 16 which included an abstract idea (see rejection for claim 16). The claim recites the additional limitations:
Step 2A Prong 2:
wherein plurality of bins are a time-to-frequency transform of the media signal — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the frequency coefficients to filterbank representations or bins of a time-to-frequency transform.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein plurality of bins are a time-to-frequency transform of the media signal — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 26
Claim 26 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 20 which included an abstract idea (see rejection for claim 20). The claim recites the additional limitations:
Step 2A Prong 1:
predict a set of conditioning variables given conditioning information describing the target media signal, the conditioning information comprising quantized frequency coefficients describing the target media signal — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to formulating an opinion on what conditioning variables should be based on given data.
combine said first set of input variables with at least a subset of said set of conditioning variables — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to performing a judgement of how to arrange input variables and conditioning variables together.
Step 2A Prong 2:
A generative model for generating a target media signal, comprising: a neural network system according to claim 20 — This limitation is directed to merely applying an abstract idea using a generic computer with a neural network as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
a conditioning neural network trained to — This limitation is directed to merely applying an abstract idea using a generic computer with a neural network as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
said time predicting recurrent neural network being configured to — This limitation is directed to merely applying an abstract idea using a generic computer with a neural network as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
A generative model for generating a target media signal, comprising: a neural network system according to claim 20 — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
a conditioning neural network trained to — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
said time predicting recurrent neural network being configured to — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 27
Claim 27 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 26 which included an abstract idea (see rejection for claim 26). The claim recites the additional limitations:
Step 2A Prong 1:
predict said second set of output variables, given a sum of said first set of output variables and a second set of input variables representing lower frequency bands of the current time frame — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to formulating an opinion on what the input and output variables should be based on given data.
combine said sum with at least a subset of said set of conditioning variables — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to performing a judgement of how to arrange a sum and conditioning variables together.
Step 2A Prong 2:
wherein said first set of output variables, predicted by the time predicting portion, are used as input variables to the frequency predicting portion — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the particular data operated on by the predictions.
wherein the neural network system includes a frequency predicting recurrent neural network comprising a plurality of neural network layers — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the architecture of the neural network system.
said frequency predicting neural network being trained to — This limitation is directed to merely applying an abstract idea using a generic computer with generic machine learning training as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
and wherein said frequency predicting recurrent neural network is configured to — This limitation is directed to merely applying an abstract idea using a generic computer with a neural network as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein said first set of output variables, predicted by the time predicting portion, are used as input variables to the frequency predicting portion — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
wherein the neural network system includes a frequency predicting recurrent neural network comprising a plurality of neural network layers — Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
said frequency predicting neural network being trained to — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
and wherein said frequency predicting recurrent neural network is configured to — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 28
Claim 28 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 26 which included an abstract idea (see rejection for claim 26). The claim recites the additional limitations:
Step 2A Prong 2:
wherein the conditioning information includes at least one of a set of distorted frequency coefficients, a set of perceptual model coefficients, and a spectral envelope — This limitation is directed to merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) as it merely limits the field of the conditioning information.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
wherein the conditioning information includes at least one of a set of distorted frequency coefficients, a set of perceptual model coefficients, and a spectral envelope —Merely limiting a judicial exception to a particular field of use (see MPEP 2106.05(h)) cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 29
Claim 29 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 29 which included an abstract idea (see rejection for claim 29). The claim recites the additional limitations:
Step 2A Prong 1:
predict a set of frequency coefficients representing this frequency band — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.). The limitation is directed to a mental process because it amounts to formulating an opinion on what conditioning variables should be based on given data.
Step 2A Prong 2:
A method for obtaining an enhanced media signal using a generative model according to claim 26, comprising the steps of: a) providing conditioning information to the conditioning neural network — This limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g)).
b) for each frequency band of a current time frame, using said frequency predicting recurrent neural network to — This limitation is directed to merely applying an abstract idea using a generic computer with a neural network as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
and providing said set of frequency coefficients to the frequency predicting recurrent neural network as said second set of input variables — This limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g)).
c) providing the predicted sets of frequency coefficients representing all frequency bands of the current frame to the time predicting RNN as said first set of input variables — This limitation is directed to mere data gathering and outputting which has been recognized by the courts (as per Ultramercial, 772 F.3d at 715, 112 USPQ2d at 1754) as insignificant extra-solution activity (see MPEP 2106.05(g)).
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
A method for obtaining an enhanced media signal using a generative model according to claim 26, comprising the steps of: a) providing conditioning information to the conditioning neural network — This limitation is recited at a high level of generality and amounts to mere data gathering of transmitting and receiving data over a network, which is well-understood, routine, and conventional activity (see MPEP 2106.05(d) II.), which cannot amount to significantly more than the judicial exception.
b) for each frequency band of a current time frame, using said frequency predicting recurrent neural network to — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
and providing said set of frequency coefficients to the frequency predicting recurrent neural network as said second set of input variables — This limitation is recited at a high level of generality and amounts to mere data gathering of transmitting and receiving data over a network, which is well-understood, routine, and conventional activity (see MPEP 2106.05(d) II.), which cannot amount to significantly more than the judicial exception.
c) providing the predicted sets of frequency coefficients representing all frequency bands of the current frame to the time predicting RNN as said first set of input variables — This limitation is recited at a high level of generality and amounts to mere data gathering of transmitting and receiving data over a network, which is well-understood, routine, and conventional activity (see MPEP 2106.05(d) II.), which cannot amount to significantly more than the judicial exception.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 30
Claim 30 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 26 which included an abstract idea (see rejection for claim 26). The claim recites the additional limitations:
Step 2A Prong 2:
A decoder comprising a generative model according to claim 26 — This limitation is directed to merely applying an abstract idea using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
A decoder comprising a generative model according to claim 26 — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 31
Claim 31 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 26 which included an abstract idea (see rejection for claim 26). The claim recites the additional limitations:
Step 2A Prong 2:
A computer program product comprising computer readable program code portions which, when executed by a computer, implement a generative model according to claim 26 — This limitation is directed to merely applying an abstract idea using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.04(d)).
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2.
Step 2B:
The additional elements as identified in step 2A prong 2:
A computer program product comprising computer readable program code portions which, when executed by a computer, implement a generative model according to claim 26 — Using a generic computer as a tool (see MPEP 2106.05(f)(2), 2106.05(d)) cannot amount to significantly more than the judicial exception itself.
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
20-21, 25-28, and 30
Claims 16-17, 19-22, 24, 26-28, and 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Vasquez et al. “MelNet: A Generative Model for Audio in the Frequency Domain” herein referred to as Vasquez in view of Martinez Ramirez et al. “END-TO-END EQUALIZATION WITH CONVOLUTIONAL NEURAL NETWORKS” herein referred to as Martinez Ramirez.
Regarding Claim 16
Vasquez teaches:
A computer implemented neural network system for predicting frequency coefficients of a media signal, the neural network system comprising:
(page 1 column 1 paragraph 2) “We introduce a generative model for audio which captures longer-range dependencies than existing end-to-end models. We primarily achieve this by modelling 2D time-frequency representations such as spectrograms rather than 1D time-domain waveforms (Figure 1).”
a time predicting portion including at least one neural network trained to predict a first set of output variables representing a time predicted frequency band of a current time frame given coefficients of one or several previous time frames
(page 3 column 2 section 4.1 paragraph 1) “The time-delayed stack utilizes multiple layers of multidimensional RNNs to extract features[*Examiner notes: first set of output variables] from x<i,∗, the two-dimensional region consisting of all frames preceding xij.”
and a frequency predicting portion including at least one neural network trained to predict a second set of output variables representing a frequency predicted frequency band given coefficients of one or several frequency bands that are adjacent to and lower than the frequency predicted frequency band
(page 4 column 1 paragraph 4) “The frequency-delayed stack is a one-dimensional RNN which runs forward along the frequency axis. Much like existing one-dimensional autoregressive models (language models, waveform models, etc.), the frequency-delayed stack operates on a one-dimensional sequence (a single frame) and estimates the distribution for each element conditioned on all preceding elements. The primary difference is that it is also conditioned on the outputs of the time-delayed stack, allowing it to use the full two-dimensional context x<ij.”; [*Examiner notes: The output of the frequency-delayed stack is mapped to the second set of output variables]
an output stage configured to provide a set of frequency coefficients representing a specific frequency band of said current time frame, based on said first and second set of output variables, said specific frequency band being at least one of the time predicted and frequency predicted frequency band,
(page 8 column 1 section 7.1.3) “To show that MelNet can model audio modalities other than speech, we apply the model to the task of unconditional music generation. We utilize the MAESTRO dataset [19], which consists of over 172 hours of solo piano performances. The samples demonstrate that MelNet learns musical structures such as melody and harmony. Furthermore, generated samples often maintain consistent tempo and contain interesting variation in volume, timbre, and rhythm.”
and wherein a) said first set of output variables, predicted by the time predicting portion, is used as input variables to the frequency predicting portion, or b) said second set of output variables, predicted by the frequency prediction portion, is used as input variables to the time predicting portion.
(page 4 column 1 paragraph 4) “The frequency-delayed stack is a one-dimensional RNN which runs forward along the frequency axis. Much like existing one-dimensional autoregressive models (language models, waveform models, etc.), the frequency-delayed stack operates on a one-dimensional sequence (a single frame) and estimates the distribution for each element conditioned on all preceding elements. The primary difference is that it is also conditioned on the outputs of the time-delayed stack, allowing it to use the full two-dimensional context x<ij.”
Vasquez does not explicitly teach:
wherein the specific frequency band comprises a plurality of bins, and wherein the set of frequency coefficients comprises coefficients for the plurality of bins of the specific frequency band
However, Martinez Ramirez teaches:
wherein the specific frequency band comprises a plurality of bins, and wherein the set of frequency coefficients comprises coefficients for the plurality of bins of the specific frequency band
(page 296 column 2) “Given an arbitrary EQ configuration, our task is to train a deep neural network to learn the specific transformation. In this way, an optimal filter bank decomposition and its latent representation are learned from the input data, and these are transformed and decoded to obtain an audio signal that matches the target”; (page 300 column 1 paragraph 2) “From the feature map matrix X1 (see Fig. 2c), the filters W1 are actively acting as a filter bank or frequency selectors[*Examiner notes: mapped to bins], since X1 correspond to the decomposition of the input data into different frequencies.”
Vasquez, Martinez Ramirez, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the time and frequency predictors of Vasquez with the filter bank “bins” representation of Martinez Ramirez because (Martinez Ramirez page 302 column 2 last paragraph) “Finally, it is worth noting the immense benefit that generative music could obtain from deep learning architectures for intelligent music production. Our implementation could be used in the field of deep neural networks applied to generative music and automatic mixing production systems.”
Regarding Claim 17
Vasquez in view of Martinez Ramirez teaches:
The neural network system according to claim 16
(see rejection of claim 16)
And Vasquez further teaches
wherein a) said first set of output variables, predicted by the time predicting portion, is used as input variables to the frequency predicting portion, and said time predicted frequency band is adjacent to said frequency predicted frequency band in said current time frame.
(page 4 column 1 paragraph 4) “The frequency-delayed stack is a one-dimensional RNN which runs forward along the frequency axis. Much like existing one-dimensional autoregressive models (language models, waveform models, etc.), the frequency-delayed stack operates on a one-dimensional sequence (a single frame) and estimates the distribution for each element conditioned on all preceding elements. The primary difference is that it is also conditioned on the outputs of the time-delayed stack, allowing it to use the full two-dimensional context x<ij[*Examiner notes: includes adjacent].”
Regarding Claim 19
Vasquez in view of Martinez Ramirez teaches:
The neural network system according to claim 16,
(see rejection of claim 16)
And Vasquez further teaches:
wherein said first set of output variables, predicted by the time predicting portion, are used as input variables to the frequency predicting portion.
(page 4 column 1 paragraph 4) “The frequency-delayed stack is a one-dimensional RNN which runs forward along the frequency axis. Much like existing one-dimensional autoregressive models (language models, waveform models, etc.), the frequency-delayed stack operates on a one-dimensional sequence (a single frame) and estimates the distribution for each element conditioned on all preceding elements. The primary difference is that it is also conditioned on the outputs of the time-delayed stack, allowing it to use the full two-dimensional context x<ij.”
Regarding Claim 20
Vasquez in view of Martinez Ramirez teaches:
The neural network system according claim 16
(see rejection of claim 16)
wherein the time predicting portion includes: a time predicting recurrent neural network comprising a plurality of neural network layers, said time predicting recurrent neural network being trained to predict an intermediate set of output variables representing the current time frame, given a first set of input variables representing a preceding time frame of the media signal,
(page 3 column 2 section 4.1) “The time-delayed stack utilizes multiple layers of multidimensional RNNs to extract features from x<i,∗, the two dimensional region consisting of all frames preceding xij .”
Martinez Ramirez further teaches:
and a band mixing neural network trained to predict said first set of output variables, wherein variables in the intermediate set are formed by mixing variables in said intermediate set representing said time predicted frequency band and a plurality of frequency bands adjacent to said frequency band.
(page 302 column 1 last paragraph) “Possible applications for this architecture are within the fields of automatic mixing and audio effect modeling. For example, style-learning of a specific sound engineer could be explored, where the model is trained with several tracks equalized by the engineer and finds a generalization from the engineer’s EQ practices.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to combine Vasquez with Martinez Ramirez for the same reasons given in claim 1 above.
Regarding Claim 21
Vasquez in view of Martinez Ramirez teaches:
The neural network system according to claim 20
(see rejection of claim 20)
And Vasquez further teaches:
wherein the time predicting portion further includes: an input stage comprising a neural network trained to predict said first set of input variables given frequency coefficients of a preceding time frame of said media signal.
(page 3 section 4.1 paragraph 1) “The time-delayed stack utilizes multiple layers of multidimensional RNNs to extract features from x<i,∗, the two-dimensional region consisting of all frames preceding xij . Each multidimensional RNN is composed of three one-dimensional RNNs: one which runs forwards along the frequency axis, one which runs backwards along the frequency axis, and one which runs forwards along the time axis. Each RNN runs along each slice of a given axis, as shown in Figure 2.”; Figure 3
PNG
media_image1.png
458
607
media_image1.png
Greyscale
Regarding Claim 22
Vasquez in view of Martinez Ramirez teaches:
The neural network system according to claim 19,
(see rejection of claim 19)
And Vasquez further teaches:
wherein the frequency predicting portion includes: a frequency predicting recurrent neural network comprising a plurality of neural network layers, said frequency predicting neural network being trained to predict said second set of output variables, given a sum of said first set of output variables and a second set of input variables representing lower frequency bands of the current time frame.
(page 4 figure 3 caption) “The outputs of these functions are projected (by the matrices Wtl and Wfl) and summed with the layer inputs to form residual blocks.”
PNG
media_image2.png
296
313
media_image2.png
Greyscale
Regarding Claim 24
Vasquez in view of Martinez Ramirez teaches:
The neural network system according to claim 16
(see rejection of claim 16)
And Vasquez further teaches:
wherein each frequency coefficient is represented by a set of distribution parameters, wherein said set of distribution parameters are configured to parametrize a probability distribution of the coefficient, wherein said specific frequency band of said current time frame is obtained by sampling the probability distribution of each frequency coefficient.
(page 2 column 2 section 3) “We use an autoregressive model which factorizes the joint distribution over a spectrogram x as a product of conditional distributions”; (page 6 column 2 section 6.2) “To sample from the multiscale model we iteratively sample a value for xg conditioned on x<g using the learned distributions defined by the estimated network parameters”
Regarding Claim 26
Vasquez in view of Martinez Ramirez teaches:
A generative model for generating a target media signal, comprising: a neural network system according to claim 20
(see rejection of claim 20)
Vasquez further teaches:
and a conditioning neural network trained to predict a set of conditioning variables given conditioning information describing the target media signal, the conditioning information comprising quantized frequency coefficients describing the target media signal,
(page 4 column 2 section 4.4) “To incorporate conditioning information into the model, conditioning features z are simply projected onto the input layer along with the inputs x, altering Equations 7 and 9:”; Equations 13 and 14; [*Examiner notes: The broadest reasonable interpretation of a neural network includes calculations involving matrix multiplications. Equations 13 and 14 provide conditioning operations involving matrix multiplications which could be interpreted as a conditioning neural network]
said time predicting recurrent neural network being configured to combine said first set of input variables with at least a subset of said set of conditioning variables
(page 3 column 2 last paragraph) “To ensure the output htij [l] is only a function of frames which lie in the context x<ij , the inputs to the time-delayed stack are shifted backwards one step in time:”; Equations 6 and 7; [*Examiner notes: Equations 6 and 7 show how the conditioning information is incorporated into the model]
Regarding Claim 27
Vasquez in view of Martinez Ramirez teaches:
The generative model according to claim 26
(see rejection of claim 26)
Vasquez further teaches:
wherein said first set of output variables, predicted by the time predicting portion, are used as input variables to the frequency predicting portion,
(page 4 column 1 paragraph 4) “The frequency-delayed stack is a one-dimensional RNN which runs forward along the frequency axis. Much like existing one-dimensional autoregressive models (language models, waveform models, etc.), the frequency-delayed stack operates on a one-dimensional sequence (a single frame) and estimates the distribution for each element conditioned on all preceding elements. The primary difference is that it is also conditioned on the outputs of the time-delayed stack, allowing it to use the full two-dimensional context x<ij.”
wherein the neural network system includes a frequency predicting recurrent neural network comprising a plurality of neural network layers,
(page 4 column 1 section 4.2) “The frequency-delayed stack is a one-dimensional RNN which runs forward along the frequency axis.”
said frequency predicting neural network being trained to predict said second set of output variables, given a sum of said first set of output variables and a second set of input variables representing lower frequency bands of the current time frame,
(page 4 column 1 first paragraph) “The primary difference is that it is also conditioned on the outputs of the time-delayed stack, allowing it to use the full two-dimensional context x<ij .” Figure 2
PNG
media_image3.png
195
586
media_image3.png
Greyscale
and wherein said frequency predicting recurrent neural network is configured to combine said sum with at least a subset of said set of conditioning variables.
(page 4 column 1 paragraph 4) “The frequency-delayed stack is a one-dimensional RNN which runs forward along the frequency axis. Much like existing one-dimensional autoregressive models (language models, waveform models, etc.), the frequency-delayed stack operates on a one-dimensional sequence (a single frame) and estimates the distribution for each element conditioned on all preceding elements. The primary difference is that it is also conditioned on the outputs of the time-delayed stack, allowing it to use the full two-dimensional context x<ij.”; [*Examiner notes: the sum and the conditioning variables are both inputs to the recurrent neural network. Thus they are combined during RNN processing.]
Regarding Claim 28
Vasquez in view of Martinez Ramirez teaches:
The generative model according to claim 26
(see rejection of claim 26)
Vasquez further teaches:
wherein the conditioning information includes at least one of a set of distorted frequency coefficients, a set of perceptual model coefficients, and a spectral envelope.
(page ) “Reshaping, upsampling, and broadcasting can be used as necessary to ensure the conditioning features have the same time and frequency shape as the input spectrogram, e.g. a
one-hot vector representation for speaker ID would first be broadcast along both the time and frequency axes.”; [*Examiner notes: conditioning by reshaping, upsampling, and broadcasting amounts to distorting the frequency coefficients.]
Regarding Claim 30
Vasquez in view of Martinez Ramirez teaches:
comprising a generative model according to claim 26.
(see rejection of claim 26)
Vasquez further teaches:
A decoder
(page 14 column 1 last paragraph) “All models have 8-layer autoregressive decoders”
Claims 18, 23, 29, and 31 are rejected under 35 U.S.C. 103 as being unpatentable over Vasquez in view of Martinez Ramirez, and further in view of NPL reference Adavanne et al. “Stacked Convolutional and Recurrent Neural Networks for Bird Audio Detection” herein referred to as Adavanne.
Regarding Claim 18
Vasquez in view of Martinez Ramirez teaches:
The neural network system according to claim 16
(see rejection of claim 16)
Vasquez further teaches:
and said time predicted frequency band and said frequency predicted frequency band are a same frequency band in a previous time frame and current time frame respectively
(page 2 figure 1 ) “Spectrogram and waveform representations of the same four-second audio signal. The waveform spans nearly 100,000 timesteps whereas the temporal axis of the spectrogram spans roughly 400.”; Figure 1
PNG
media_image4.png
202
337
media_image4.png
Greyscale
Vasquez in view of Martinez Ramirez does not explicitly teach:
wherein said second set of output variables, predicted by the frequency prediction portion, is used as input variables to the time predicting portion,
However, Adavanne teaches:
wherein said second set of output variables, predicted by the frequency prediction portion, is used as input variables to the time predicting portion,
(page 1730 column 2 paragraph 1) “The feature maps from the individual CNN s are merged using an elementwise multiplication operation and fed to bi-directional gated recurrent unit (GRU) layers followed by fully-connected time distributed dense layers.”
Vasquez, Martinez Ramirez, Adavanne, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the time and frequency predictors of Vasquez in view of Martinez Ramirez by switching the order of the RNNs so that the output of the frequency prediction neural network is used as an input to the time prediction neural network as taught by Adavanne because (Adavanne page 1732 column 2 section V) “A stacked convolutional and bidirectional recurrent neural network architecture (CBRNN) was proposed for bird audio detection task. Two kinds of features and their combination were studied and the best result on test data was achieved using the log mel-band energy feature. The proposed novel domain adaptation was shown to consistently perform better than having no adaptation. The data augmentation method studied was not helpful and gave comparable results as without augmentation. The proposed method achieved an area under curve measure of 88.1 % on the unseen evaluation data, and 95.5% on the development data.”
Regarding Claim 23
Vasquez in view of Martinez Ramirez teaches:
The neural network system according to claim 22
(see rejection of claim 22)
Vasquez in view of Martinez Ramirez does not explicitly teach:
wherein the frequency predicting portion further includes: one or several output layers trained to provide said set of frequency coefficients based on said second set of output variables
However, Adavanne teaches:
wherein the frequency predicting portion further includes: one or several output layers trained to provide said set of frequency coefficients based on said second set of output variables
(page 1730 column 1 paragraph 1) “This stacked neural network is built by stacking layers of CNN, RNN and FC followed by a single node output layer producing outputs in the range of [0, 1].”
Vasquez, Martinez Ramirez, Adavanne, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the time and frequency predictors of Vasquez in view of Martinez Ramirez with the output layer of Adavanne because (Adavanne page 1732 column 2 section V) “A stacked convolutional and bidirectional recurrent neural network architecture (CBRNN) was proposed for bird audio detection task. Two kinds of features and their combination were studied and the best result on test data was achieved using the log mel-band energy feature. The proposed novel domain adaptation was shown to consistently perform better than having no adaptation. The data augmentation method studied was not helpful and gave comparable results as without augmentation. The proposed method achieved an area under curve measure of 88.1 % on the unseen evaluation data, and 95.5% on the development data.”
Regarding Claim 29
Vasquez in view of Martinez Ramirez teaches:
A method for obtaining an enhanced media signal using a generative model according to claim 26, comprising the steps of:
(see rejection of claim 26)
Vasquez further teaches:
a) providing conditioning information to the conditioning neural network
(page 4 column 2 section 4.4) “To incorporate conditioning information into the model, conditioning features z are simply projected onto the input layer along with the inputs x”
b) for each frequency band of a current time frame, using said frequency predicting recurrent neural network to predict a set of frequency coefficients representing this frequency band, and providing said set of frequency coefficients to the frequency predicting recurrent neural network as said second set of input variables,
(page 4 column 1 section 4.2) “The frequency-delayed stack is a one-dimensional RNN which runs forward along the frequency axis. Much like existing one-dimensional autoregressive models (language models, waveform models, etc.), the frequency-delayed stack operates on a one-dimensional sequence (a single frame) and estimates the distribution for each element conditioned on all preceding elements […] At each layer, the frequency-delayed stack takes two inputs: the the previous-layer outputs of the frequency-delayed stack, hf[l − 1], and the current-layer outputs of the time-delayed stack ht[l].”
Vasquez in view of Martinez Ramirez does not explicitly teach:
c) providing the predicted sets of frequency coefficients representing all frequency bands of the current frame to the time predicting RNN as said first set of input variables.
However, Adavanne teaches:
c) providing the predicted sets of frequency coefficients representing all frequency bands of the current frame to the time predicting RNN as said first set of input variables.
(page 1730 column 2 paragraph 1) “The feature maps from the individual CNN s are merged using an elementwise multiplication operation and fed to bi-directional gated recurrent unit (GRU) layers followed by fully-connected time distributed dense layers.”
Vasquez, Martinez Ramirez, Adavanne, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the time and frequency predictors of Vasquez in view of Martinez Ramirez by switching the order of the RNNs so that the output of the frequency prediction neural network is used as an input to the time prediction neural network as taught by Adavanne because (Adavanne page 1732 column 2 section V) “A stacked convolutional and bidirectional recurrent neural network architecture (CBRNN) was proposed for bird audio detection task. Two kinds of features and their combination were studied and the best result on test data was achieved using the log mel-band energy feature. The proposed novel domain adaptation was shown to consistently perform better than having no adaptation. The data augmentation method studied was not helpful and gave comparable results as without augmentation. The proposed method achieved an area under curve measure of 88.1 % on the unseen evaluation data, and 95.5% on the development data.”
Regarding Claim 31
Vasquez in view of Martinez Ramirez teaches:
a generative model according to claim 26
(see rejection of claim 26)
Vasquez in view of Martinez Ramirez does not explicitly teach:
A computer program product comprising computer readable program code portions which, when executed by a computer, implement
However, Adavanne teaches:
A computer program product comprising computer readable program code portions which, when executed by a computer, implement
(page 1732 column 2 last paragraph) “Part of the computations leading to these results were performed on a TITAN-X GPU donated by NVIDIA.”
Vasquez, Martinez Ramirez, Adavanne, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the time and frequency predictors of Vasquez in view of Martinez Ramirez by implementing it on the computer-readable medium of Adavanne because (Adavanne page 1732 column 2 last paragraph) “Part of the computations leading to these results were performed on a TITAN-X GPU donated by NVIDIA.” That is, the NVIDIA chip can be used to perform computations related to neural network methods.
Claim 25 is rejected under 35 U.S.C. 103 as being unpatentable over Vasquez in view of Martinez Ramirez, and further in view of NPL reference Cheuk et al. “nnAudio: An on-the-Fly GPU Audio to Spectrogram Conversion Toolbox Using 1D Convolutional Neural Networks” herein referred to as Cheuk.
Regarding Claim 25
Vasquez in view of Martinez Ramirez teaches:
The neural network system according to claim 16
(see rejection of claim 16)
Vazquez in view of Martinez Ramirez does not explicitly teach:
wherein plurality of bins are a time-to-frequency transform of the media signal
[*Examiner notes: Martinez Ramirez does teach using fast fourier transform (a form of time-to-frequency transform), however Martinez Ramirez does not go into much detail about how it is used]
However, Cheuk teaches:
wherein plurality of bins are a time-to-frequency transform of the media signal
(page 161983 column 2 last paragraph) “Therefore, the audio waveforms consist of discrete data points. The Discrete Fourier Transform can be used to convert this discrete signal from the time domain to the frequency domain. Equation (1) shows the mathematical expression for the discrete Fourier transform [41], where X[k] is the output in the frequency domain; and x[n] is the nth sample of the audio input in the time domain.”
Vasquez, Martinez Ramirez, Cheuk, and the instant application are analogous because they are all directed to neural networks.
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the present invention to modify the time and frequency predictors as taught by Vasquez in view of Martinez Ramirez by using the time-to-frequency transform for the bins as taught by Cheuk because (Cheuk page 161983 column 2 last paragraph) “When recording audio using any computer or mobile device, the analogue signal is converted to a digital signal before storing the data. Therefore, the audio waveforms consist of discrete data points. The Discrete Fourier Transform can be used to convert this discrete signal from the time domain to the frequency domain.”
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ezra J Baker whose telephone number is (703)756-1087. The examiner can normally be reached Monday - Friday 10:00 am - 8:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/E.J.B./Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights