Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
(b) CROSS-REFERENCES TO RELATED APPLICATIONS: See 37 CFR 1.78 and MPEP § 211 et seq.
The parent application of the instant invention, has now pass to issue, as a US Patent. Please update the specification. Correction is require.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-15 of USPatent 12,159,645 and claim 1-15 of USPatent 12,170,096; each, respectfully, in further view of the combination of Paraskevopoulos (20200335086) in view of Sargsyan et al (20200066296). Although the claims at issue are not identical, they are not patentably distinct from each other because the alternative language in the disclosed steps are not necessary to realize the functionality of the instant claims. As to the extra claim steps of details toward the type of transform/inverse transform, as well as the detailed frequency ranges, and differing applications to the end user, found in the claims in the instant invention, the combination of Paraskevopoulos (20200335086) in view of Sargsyan et al (20200066296) teaches ( Paraskevopoulos (20200335086), transforming the audio signal data into spectrograms – para 0009, 0010, as well as using the ‘general approach’ may be adapted to produce synthesized audio signals –para 0027; examiner notes that it is very old and notoriously well known, in the art of signal transformation, to use short-time Fourier transforms, and it’s inverses, in generating spectrograms from audio signals and likewise, generating audio signals from spectrograms -- see prior art listing at the end of the office action); further, Sargsyan et al (20200066296), as sending to the user to review/hear the processed audio signal – para 0025, in an end-user service, or in para 0026, showing the ability of the user to use the application to cancel noise in their audio/video before uploading; and further, Sargsyan et al (20200066296), teaching the expansion of low bitrate audio of 8kHz up to 44.1 kHz – para 0027.
Claim feature mapping, is provided in the table below. Examiner notes, that the detailed mapping of the USPatents, are provided for claims 1-10 of the instant invention. Claims 11-20, are similar in scope and content to claims 1-10, and are therefore, mapped similarly to the mapping provided against claims 1-10.
A copy of claims 11-20 of the instant invention are provided in the table below; examiner guides the reader to track these claim features that are also provided in claims 1-10 of the instant invention, and to follow that particular mapping to the USPatents provided.
18/932072
12,159,645
12,170,096
1. A non-transitory medium with instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising: receiving input that is indicative of an instruction to train a generative adversarial network to facilitate upsampling to a given sampling rate; acquiring a plurality of discrete audio signals that have the given sampling rate; applying a transform to each audio signal of the plurality of discrete audio signals, so as to produce a plurality of magnitude spectrograms; and training the generative adversarial network with the plurality of magnitude spectrograms, such that the generative adversarial network learns a characteristic of the plurality of magnitude spectrograms, wherein using the characteristic, the generative adversarial network is able to facilitate upsampling of a first discrete audio signal by altering a magnitude spectrogram that is produced for the first discrete audio signal to produce another magnitude spectrogram to which an inverse transform can be applied to produce a second discrete audio signal having the given sampling rate.
2. The non-transitory medium of claim 1, wherein the transform is a short-time Fourier transform (STFT), and wherein the inverse transform is an inverse short-time Fourier transform (ISTFT).
3. The non-transitory medium of claim 1, wherein the operations further comprise: storing the generative adversarial network in a database that includes a plurality of generative adversarial networks, each of which is associated with a different sampling rate.
4. The non-transitory medium of claim 3, wherein the database is queryable by sampling rate.
5. The non-transitory medium of claim 3, wherein the operations further comprise: examining, in response to said receiving, the database to confirm that none of the plurality of generative adversarial networks are associated with the given sampling rate.
6. The non-transitory medium of claim 1, wherein the operations further comprise: receiving second input that is indication of a selection of the plurality of discrete audio signals.
7. The non-transitory medium of claim 1, wherein the operations further comprise: causing display of a notification that specifies the generative adversarial network has been trained.
8. The non-transitory medium of claim 7, wherein said causing comprises: posting the notification to an interface that is accessible to a computing device and via which an individual provides the instruction.
9. The non-transitory medium of claim 1, wherein the generative adversarial network includes a pair of neural networks that are trained using the plurality of magnitude spectrograms.
10. A method for training a neural network to upsample discrete audio signals to a given sampling rate, the method comprising: acquiring a plurality of discrete audio signals that have the given sampling rate; applying a Fourier transform to each audio signal of the plurality of discrete audio signals, so as to produce a plurality of magnitude spectrograms; and training the neural network with the plurality of magnitude spectrograms, such that the neural network learns a characteristic of the plurality of magnitude spectrograms; wherein when a magnitude spectrogram produced for an audio signal that has a sampling rate less than the given sampling rate is provided to the neural network as input, the neural network adjusts the characteristic to produce another magnitude spectrogram to which an inverse Fourier transform can be applied to produce another audio signal that has the given sampling rate.
11. The method of claim 10, further comprising: storing the neural network in a database that includes one or more neural networks, each of which is associated with a different sampling rate.
12. The method of claim 11, wherein neural networks in the database are sharable across multiple users of a computer program through which media content can be generated or manipulated.
13. The method of claim 10, further comprising: populating a data structure with information so as to programmatically associate the neural network with the given sampling rate.
14. The method of claim 10, further comprising: receiving input that is provided by an individual through an interface and that is indicative of an instruction to train the neural network.
15. The method of claim 10, wherein the given sampling rate is at least 40,000 hertz.
16. The method of claim 10, wherein the neural network is trained to take, as input, magnitude spectrograms that are associated with discrete audio signals that have a second given sampling rate.
17. The method of claim 16, wherein the given sampling rate is at least double the second given sampling rate.
18. A method comprising: acquiring a plurality of magnitude spectrograms, each of which corresponds to a different one of a plurality of discrete audio signals that have a given sampling rate; training a generative model with the plurality of magnitude spectrograms, such that the generative model learns to how to manipulate magnitude spectrograms that are produced for discrete audio signals with sampling rates less than the given sampling rate to mimic one or more characteristics of the plurality of magnitude spectrograms; and storing the generative model is a database that includes one or more other generative models, each of which is associated with a different sampling rate.
19. The method of claim 18, further comprising: receiving input that is indicative of a request to further train the generative model and that specifies one or more discrete audio signals to be used for training; applying a transform to the one or more discrete audio signals to produce one or more magnitude spectrograms; and providing the one or more magnitude spectrograms to the generative model as additional data for training.
20. The method of claim 18, wherein the database is browsable and/or queryable through an interface via which an individual is able to select a discrete audio signal for upsampling.
1. A method for training a generative adversarial network, the method comprising: receiving, by a processor, input that is provided through an interface and that specifies a desired sampling rate; examining, by the processor, a database that includes multiple generative adversarial networks associated with different sampling rates, so as to confirm that none of the multiple generative adversarial networks are associated with the desired sampling rate; obtaining, by the processor, a series of discrete audio signals having the desired sampling rate; applying, by the processor, a transform to each discrete audio signal in the series of discrete audio signals to produce a series of magnitude spectrograms; and training, by the processor, the generative adversarial network with the series of magnitude spectrograms, such that the generative adversarial network learns one or more characteristics of the series of magnitude spectrograms at the desired sampling rate, wherein using the one or more characteristics, the generative adversarial network is able to facilitate upsampling of a first discrete audio signal to the desired sampling rate by altering a magnitude spectrogram that is generated for the first discrete audio signal and provided as input to produce another magnitude spectrogram to which an inverse transform can be applied to produce a second discrete audio signal having the desired sampling rate.
a database that includes multiple generative adversarial networks associated with different sampling rates (claim 1)
4. The method of claim 3, wherein the database is queryable by sampling rate.
so as to confirm that none of the multiple generative adversarial networks are associated with the desired sampling rate; (claim 1).
2. The method of claim 1, further comprising: receiving, by the processor, second input that is provided through the interface and that is indicative of a selection of the series of discrete audio signals; wherein said obtaining comprises acquiring the series of discrete audio signals from a second database responsive to receiving the second input.
5. The method of claim 1, further comprising: causing, by the processor, transmission of a notification to a computing device that specifies the generative adversarial network has been trained.
6. The method of claim 5, wherein the computing device is associated with an individual who either indicated an interest in employing the generative adversarial network while producing a media compilation or provided the input through the interface.
wherein each of the multiple models is associated with a different pair of sampling rates, each pair of sampling rates being defined by a lower sampling rate at which magnitude spectrograms are input and a higher sampling rate at which magnitude spectrograms are output for transformation into audio signals at the higher sampling rate (claim 7)
1. A method for implement-ing a generative adversarial network, the method comprising: receiving, by a processor, input that is indicative of a selection of a first discrete audio signal to be upsampled from a first sampling rate to a second sampling rate; examining, by the processor, a database that includes multiple generative adversarial networks associated with different sampling rates, so as to identify the generative adversarial network that is associated with the second sampling rate; applying, by the processor, a transform to the first discrete audio signal to produce a first magnitude spectrogram; providing, by the processor, the first magnitude spectrogram to the generative adversarial network as input so as to produce a second magnitude spectrogram, wherein the second magnitude spectrogram is generated from the first magnitude spectrogram by the generative adversarial network by adjusting a characteristic learned, during training, from analysis of multiple magnitude spectrograms, each of which is associated with a different discrete audio signal having the second sampling rate; and applying, by the processor, an inverse transform to the second magnitude spectrogram to produce a second discrete audio signal having the second sampling rate.
5. The method of claim 1, wherein the transform is a short-time Fourier transform (STFT).
6. The method of claim 5, wherein the inverse transform is an inverse short-time Fourier transform (ISTFT).
a database that includes multiple generative adversarial networks associated with different sampling rates, (claim 1)
2. The method of claim 1, wherein the generative adversarial network includes a pair of neural networks that are trained using the multiple magnitude spectrograms that correspond to different discrete audio signals that have the second sampling rate.
3. The method of claim 1, further comprising: receiving, by the processor, second input indicative of a selection of the second sampling rate.
2. The method of claim 1, wherein the generative adversarial network includes a pair of neural networks that are trained using the multiple magnitude spectrograms that correspond to different discrete audio signals that have the second sampling rate.
14. The method of claim 9, further comprising: transmitting, by the processor, the third and fourth discrete audio signals to a computing device for review by an individual.
2. The method of claim 1, wherein the generative adversarial network includes a pair of neural networks that are trained using the multiple magnitude spectrograms that correspond to different discrete audio signals that have the second sampling rate.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Paraskevopoulos (20200335086) in view of Sargsyan et al (20200066296).
As per claim 18, Paraskevopoulos (20200335086) teaches a method comprising:
acquiring a plurality of magnitude spectrograms, each of which corresponds to a different one of a plurality of discrete audio signals that have a given sampling rate (para 0027, as obtaining data samples that are spectrograms; and using each spectrogram as a discrete/separate representation -- Paraskevopoulos (20200335086), fig. 1, subblock 110, and para 0027 -0028));
training a generative model with the plurality of magnitude spectrograms, such that the generative model learns to how to manipulate magnitude spectrograms that are produced for discrete audio signals with sampling rates less than the given sampling rate to mimic one or more characteristics of the plurality of magnitude spectrograms data (as, using a GAN, abstract, operating on spectrograms to generate a representative set of spectrograms – para 0027; and during GAN fine-tuning, maximizing the spectrograms – para 0034; and providing the series of magnitude spectrograms using batches of real and generated spectrograms (para 0034));
and storing the generative model is a database that includes one or more other generative models, each of which is associated
Paraskevopoulos (20200335086) teaches a desired sampling rate (as, changing the sampling rate, by possibly, oversampling – para 0006). Although Paraskevopoulos (20200335086) discusses the altering of sampling rate of the data (including spectrograms), and altering the sampling rate feature – see para 0029, wherein upsampling layers is used to generate more realistic samples, Paraskevopoulos (20200335086) does not explicitly teach closer tracking/examination of the sampling rate to judge the category pertaining to a particular sampling rate and result; Sargsyan et al (20200066296) teaches the use of upsampling and downsampling layers (para 0038) and altering the sampling rate to maximize signal (noise reduction – para 0040, 0041-0043). Therefore, it would have been obvious to one of ordinary skill in the art of neural networks to expand upon the layered network functionality of Paraskevopoulos (20200335086) with multi-layer upsampling/downsampling as taught by Sargsyan et al (20200066296), because it would advantageously improve upon perception quality of the audio (para 0042 – improvement in the PESQ).
As per claim 19, Paraskevopoulos (20200335086) in view of Sargsyan et al (20200066296) teaches the method of claim 18, further comprising:
receiving input that is indicative of a request to further train the generative model and that specifies one or more discrete audio signals to be used for training (Paraskevopoulos (20200335086), as, using the GAN to determine the emotional/behavior of the user – para 0007, 0010; during a computer-human or human-human interaction - para 0007, 0010);
applying a transform to the one or more discrete audio signals to produce one or more magnitude spectrograms; and providing the one or more magnitude spectrograms to the generative model as additional data for training (Paraskevopoulos (20200335086), teaching that, when the GAN is initialized, calling for a fine tuning process – para 0031 – the device is notified that real and generated spectrograms are forwarded to the discriminator D).
As per claim 20, Paraskevopoulos (20200335086) in view of Sargsyan et al (20200066296) teaches the method of claim 18, wherein the database is browsable and/or queryable through an interface via which an individual is able to select a discrete audio signal for upsampling (examiner notes that the claim language is in the alternative -- (Sargsyan et al (20200066296), extracting features according to “SR” – Sampling Rate – see para 0072) .
Allowable Subject Matter
Claims 1-17 are allowable over the prior art of record. As per independent claims 1,10, the claim elements in combinations, toward the generative models confirming that none of the networks are associated with the sampling rate, along with first and second upsampling rates, and the relationships between first and second spectrograms (or more), are not explicitly taught by the prior art of record. Representative prior art, Paraskevopoulos (20200335086) teaches obtaining data samples that are spectrograms, using GAN’s, abstract, operating on spectrograms to generate a representative set of spectrograms – (para 0027) and providing the series of magnitude spectrograms using batches of real and generated spectrograms (para 0034) to generate synthesized spectrograms – abstract, para 0029, 0034); further adapting the general approach to produce synthesized audio signal for the class of signals – para 0027 – examiner notes, that by definition, in producing synthesized audio signals from a spectrogram, an inverse transform is applied to the spectrogram, to convert frequency domain data back to discrete time based data; being represented by the synthesize spectrograms -- abstract, para 0029, 0034). Although Paraskevopoulos (20200335086) discusses the altering of sampling rate of the data (including spectrograms), and altering the sampling rate feature – see para 0029, wherein upsampling layers is used to generate more realistic samples, Paraskevopoulos (20200335086) does not explicitly teach closer tracking/examination of the sampling rate to judge the category pertaining to a particular sampling rate and result. Sargsyan et al (20200066296) teaches the use of upsampling and downsampling layers (para 0038) and altering the sampling rate to maximize signal (noise reduction – para 0040, 0041-0043). Sargsyan et al (20200066296), further teaches the discovery of different SR (sampling rates) – para 0072, selects the calculated features for that particular sampling rate, that maximizes the NoiseModelApproximation – para 0072)). Kawahara et al (20110015931) teaches the use of short term fourier transforms to generate spectrograms (para 0082) as well as, an inverse fourier transform operating on the same parameters and hence, short term as well – para 0137). Germain et al (20160007130) teaches an inverse short term fourier transform operating on spectrograms (para 0040). Arik (20190355347) teaches the spectrogram output of a neural network to be inverse transformed to generate a synthesized waveform (para 0047). Higurashi (20190392802) teaches a neural network generating a synthesized spectrogram and comparing to audio data based spectrogram (para 0057). Thomson et al (20200175961), and related apps, teaching the use of upsampling of audio magnitudes and gan’s/neural networks (para 0249) Uhle et al (20110191101) teaches sampling rate variation for the neural network measurement of snr using spectrograms (para 0088, 0094, reflecting back on para 0048). Huang et al (20190043507) teaches preprocessing according to sampling rate – para 0037, and using machine learning, neural networks, to extract a speaker selection – para 0041, wherein the features can also be spectrograms (para 0060). However, none of the prior art of record explicitly teaches the claim limitations of independent claims 1,10, as noted above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Please see related art listed on the PTO-892 form.
The following references were found, to meet applicant’s specification, (and as discussed above):
Arik (20190355347) teaches the spectrogram output of a neural network to be inverse transformed to generate a synthesized waveform (para 0047).
Higurashi (20190392802) teaches a neural network generating a synthesized spectrogram and comparing to audio data based spectrogram (para 0057)
Thomson et al (20200175961), and related apps, teaching the use of upsampling of audio magnitudes and gan’s/neural networks (para 0249)
Uhle et al (20110191101) teaches sampling rate variation for the neural network measurement of snr using spectrograms (para 0088, 0094, reflecting back on para 0048).
Huang et al (20190043507) teaches preprocessing according to sampling rate – para 0037, and using machine learning, neural networks, to extract a speaker selection – para 0041, wherein the features can also be spectrograms (para 0060).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael Opsasnick, telephone number (571)272-7623, who is available Monday-Friday, 9am-5pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Mr. Richemond Dorvil, can be reached at (571)272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/Michael N Opsasnick/Primary Examiner, Art Unit 2658 06/05/2024
Prosecution Insights