DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 5 and 15 are objected to because of the following informalities. The term "fitness" should be "fineness." Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1- 3, 5-13 and 15- 20 are rejected under 35 U.S.C. 103 as being unpatentable over US20200058316 (Short), hereinafter US’316, in view of US10068557 (Engel), hereinafter US’557.
Regarding claim 1, US’316 discloses ‘A sound generation method realized by a computer, the sound generation method comprising: receiving a first feature amount sequence in which a musical feature amount changes over time (US’316, ¶[0247]:"the multi-channel super-resolution method ... may have as an output, a set of parameters describing individual oscillator components ... each parameter may contain information ... frequency, amplitude ... amplitude modulation, frequency modulation, and the phase of the oscillator..." frequency, amplitude, phase, amplitude modulation, and frequency modulation are musical feature amounts represented as a sequence of parameters; ¶[0246]:" "knowledge of the behavior of that oscillator in previous sample windows may be used ... if the oscillator peak belongs with a tracklet of data that may have been falling in frequency, it may be likely that the frequency may continue falling.", teaches the oscillator parameters are tracked across successive sample windows and therefore change over time; " Fig. 1:"Detect Component Frequencies, Amplitudes, Phase, Amplitude Modulations, and Frequency Modulations", teaches musical feature amounts including frequencies, amplitudes, and modulation information; Fig. 1:"Output: Any part of the Oscillator peaks as ... (2) Feature Vectors.", teaches feature vectors representing musical feature information; Fig. 3:"Create Two Sample Windows (A) and (B), where (B) lags (A).", teaches feature information that varies across successive time intervals and therefore changes over time);
US’316 does not expressly disclose ‘and using a trained model that has learned an input-output relationship between an input feature amount sequence in which the musical feature amount changes over time at a first fineness and a reference sound data sequence corresponding to an output feature amount sequence in which the musical feature amount changes over time at a second fineness that is higher than the first fineness, to process the first feature amount sequence, ‘thereby generating a sound data sequence corresponding to a second feature amount sequence in which the musical feature amount changes at the second fineness
However, US’557 discloses ‘and using a trained model that has learned an input-output relationship between an input feature amount sequence in which the musical feature amount changes over time at a first fineness and a reference sound data sequence corresponding to an output feature amount sequence in which the musical feature amount changes over time at a second fineness that is higher than the first fineness, to process the first feature amount sequence (US’557, Abstract: The neural synthesizer model can be a neural synthesis autoencoder that includes an encoder model that learns embeddings descriptive of musical characteristics and an autoregressive decoder model that is conditioned on the embedding to autoregressively generate musical waveforms”, teaches a trained model that learns an input-output relationship between musical feature information and generated sound data; col. 8, lines 44-47:”The encoding can be referred to as a temporal encoding because the result is a sequence of hidden codes with separate dimensions for time and channel”, teaches an input feature amount sequence that changes over time, col. 8, lines 42-44:”The output feed into another l x l convolution before downsampling with average pooling to get the embedding or encoding Z”, generation of a pooled temporal encoding. The pooled temporal encoding corresponds to a first feature amount sequence having a first fineness because pooling reduces temporal resolution; col. 8, lines 47-48:“The time resolution depends on the stride of the pooling. The stride can be tuned…”, teaches that the temporal encoding may exist at different levels of temporal resolution and therefore different fineness levels); col. 8, lines 61-64:”Since the decoder does not downsample anywhere in the network, the temporal encodings can be upsampled to the original audio rate with nearest”, teaches an upsampled temporal encoding corresponding to a second feature amount sequence having a second fineness that is higher than the first fineness)
thereby generating a sound data sequence corresponding to a second feature amount sequence in which the musical feature amount changes at the second fineness (US’557, Abstract :”The neural synthesizer model can use deep neural networks to generate sounds at the level of individual samples", teaches generation of a sound data sequence); Abstract:” an autoregressive decoder model that is conditioned on the embedding to autoregressively generate musical waveforms”, teaches generation of sound data corresponding to the higher-resolution temporal representation).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the machine-learning-based sound generation system of US’316 to incorporate the multi-resolution feature representations of US’557 because it teaches that feature sequences may be represented at different levels of detail through pooling, downsampling, and upsampling operations, thereby allowing a trained model to efficiently learn relationships between lower-detail and higher-detail audio feature representations while generating audio output. Such modification would have improved feature representation efficiency and model performance while utilizing techniques for processing audio feature sequences at multiple fineness levels.
Regarding claim 2, US’316 (in view of US’557) discloses ‘The sound generation method according to claim 1, as discussed above.
US’316 (in view of US’557), US’557 further discloses ‘wherein the musical feature amount at each time point in the input feature amount sequence indicates a representative value of the musical feature amount within each prescribed time period including each time point (US’557, col. 1, lines 39-41:” encoder neural network is configured to receive an input audio waveform and, in response, provide an embedding descriptive of the input audio waveform”, generation of an embedding that represents musical information extracted from the input audio waveform; col. 8, lines 44-48:”The encoding can be referred to as a 'temporal encoding' because the result is a sequence of hidden codes with separate dimensions for time and channel. The time resolution depends on the stride of the pooling”, teaches a temporal representation associated with a period of time).
Regarding claim 3, US’316 (in view of US’557) discloses ‘The sound generation method according to claim 2, as discussed above.
US’316 (in view of US’557), US’557 further discloses ‘wherein the representative value indicates a statistical value of the musical feature amount within each prescribed time period in the output feature amount sequence (US’557, col. 8, lines 42-44:"The output feed into another 1×1 convolution before downsampling with average pooling to get the embedding or encoding Z", teaches generation of an embedding using an average pooling operation; col. 8, lines 44-48:"The encoding can be referred to as a 'temporal encoding' because the result is a sequence of hidden codes with separate dimensions for time and channel. The time resolution depends on the stride of the pooling”, teaches a temporal representation corresponding to a period of time).
Regarding claim 5, US’316 (in view of US’557) discloses ‘The sound generation method according to claim 1, as discussed above.
US’316 does not expressly disclose ‘wherein each of the first fineness and the second fitness indicates a frequency of change of the musical feature amount within a unit of time, or a content ratio of a high-frequency component of the musical feature amount within the unit of time.
However, US’557 discloses ‘wherein each of the first fineness and the second fitness indicates a frequency of change of the musical feature amount within a unit of time, or a content ratio of a high-frequency component of the musical feature amount within the unit of time (US’557, col. 8, lines 47-48:” The time resolution depends on the stride of the pooling. The stride can be tuned…”, the temporal encoding may be represented at different rates of change over time; col. 8, lines 43-44:”downsampling with average pooling to get the embedding or encoding Z”, reduction of temporal detail and reduction of higher-frequency temporal content; col. 8, lines 62-63:” the temporal encodings can be upsampled to the original audio rate…”, restoration of higher-frequency temporal content and a higher rate of change over time).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to utilize the tunable temporal resolutions taught by US’557 in the modified machine-learning-based sound generation system of US’316 (in view of US’557) because varying temporal resolution through pooling and upsampling is a technique for controlling the amount of temporal detail and high-frequency information represented in feature sequences while improving processing efficiency.
Regarding claim 6, US’316 (in view of US’557) discloses ‘The sound generation method according to claim 1, as discussed above.
US’316 further discloses ‘further comprising converting the sound data sequence representing a frequency-domain waveform into a time-domain waveform (US’316, ¶[0315]:”if time-domain data may be required, an inverse-FFT (iFFT) may be performed that may convert the frequency output to the time domain”, conversion from a frequency-domain representation into a time-domain waveform; ¶[0313]: "the selected oscillator peaks may be converted back to frequency or time-domain signal using single channel re-synthesis"; ¶[0089]:” the coherent structure reconstruction may refer to a method for creating a frequency domain or time domain signal that is composed of selected oscillator peaks", a time-domain signal from selected oscillator peaks).
Regarding claim 7, US’316 discloses ‘A training method realized by a computer, the training method comprising: extracting, from reference data representing a sound waveform, a reference sound data sequence in which a musical feature amount changes over time at a prescribed fineness and an output feature amount sequence which is a time series of the musical feature amount (US'316, ¶[0247]:” "the multi-channel super-resolution method ... may have as an output, a set of parameters describing individual oscillator components ... frequency, amplitude ... amplitude modulation, frequency modulation, and the phase of the oscillator..." teaches extraction of musical feature information from sound waveform data; ¶[0246]:"if the oscillator peak belongs with a tracklet of data that may have been falling in frequency, it may be likely that the frequency may continue falling", teaches a time-series of musical feature information; ¶[0111]:” At step 504 ... calculate the complex spectral phase evolution to generate high resolution frequencies for subsequent signal extraction", teaches extraction of a sound-data sequence from waveform information);
US’316 does not expressly disclose ‘generating, from the output feature amount sequence, an input feature amount sequence in which the musical feature amount changes over time at a lower fineness than the prescribed fineness; and constructing a trained model that has learned an input-output relationship between the input feature amount sequence and the reference sound data sequence by machine learning that uses the input feature amount sequence and the reference sound data sequence
However, US’557 disclose ‘generating, from the output feature amount sequence, an input feature amount sequence in which the musical feature amount changes over time at a lower fineness than the prescribed fineness; and constructing a trained model that has learned an input-output relationship between the input feature amount sequence and the reference sound data sequence by machine learning that uses the input feature amount sequence and the reference sound data sequence (US’557, col. 8, lines 42-44:"The output feed into another 1×1 convolution before downsampling with average pooling to get the embedding or encoding Z", teaches generation of a lower-fineness feature representation through downsampling; col. 8, lines 44-46:” The encoding can be referred to as a temporal encoding because the result is a sequence of hidden codes..." , teaches an input feature amount sequence; col. 1, lines 35-36:"a machine learned neural synthesizer model that includes an autoencoder model...", teaches a trained machine-learning model; col. 1, lines 49-54:” evaluating a loss function ... [and] adjusting one or more parameters of the autoencoder model to improve the loss function", teaches machine learning using input and output training data).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to train the model using input feature sequences having lower fineness and corresponding higher-fineness sound data sequences, as taught by US’557l to the machine-learning-based sound generation system of US’316, because such training enables the model to learn relationships between lower-resolution musical representations and higher-resolution audio representations for subsequent sound generation.
Regarding claim 8, US’316 (in view of US’557) discloses ‘The training method according to claim 7, as discussed above.
US’316 (in view of US’557), US’557 further discloses ‘wherein the generating of the input feature amount sequence is performed by extracting, as the musical feature amount at each time point in the input feature amount sequence, a representative value of the musical feature amount within each prescribed time period including each time point in the output feature amount sequence (US’557, col. 1, lines 39-41:” encoder neural network is configured to receive an input audio waveform and, in response, provide an embedding descriptive of the input audio waveform”, extraction of a representative feature value from musical audio information; col. 8, lines 42-44:"The output feed into another 1×1 convolution before downsampling with average pooling to get the embedding or encoding Z", teaches generation of an embedding from multiple feature values using average pooling; col. 8, lines 44-48:"The encoding can be referred to as a 'temporal encoding' because the result is a sequence of hidden codes with separate dimensions for time and channel. The time resolution depends on the stride of the pooling”, a temporal encoding corresponding to a temporal period of the musical signal)
Regarding claim 9, US’316 (in view of US’557) discloses ‘The training method according to claim 8, as discussed above.
US’316 (in view of US’557), US’557 further discloses ‘wherein the representative value indicates a statistical value of the musical feature amount within each prescribed time period in the output feature amount sequence (US’557, col. 8, lines 42-44:"The output feed into another 1×1 convolution before downsampling with average pooling to get the embedding or encoding Z", teaches generation of an embedding using an average pooling operation; col. 8, lines 44-48:"The encoding can be referred to as a 'temporal encoding' because the result is a sequence of hidden codes with separate dimensions for time and channel. The time resolution depends on the stride of the pooling”, teaches a temporal representation corresponding to a period of time).
Regarding claim 10, US’316 (in view of US’557) discloses ‘The training method according to claim 7, as discussed above.
US’316 further discloses ‘wherein the reference data represent the sound waveform in a time domain, and the reference sound data sequence represents the sound waveform in a frequency domain (US’316, ¶[0111]:“At step 504 , the single channel super resolution module 208 may be configured to calculate the complex spectral phase evolution to generate high resolution frequencies for subsequent signal extraction”, teaches generation of frequency-domain information from a sound waveform; (¶¶[0109]-[0111]) :[The signal may be received as a time-domain waveform and analyzed to determine spectral information], teaches extraction of frequency-domain data from time-domain waveform data).
Regarding claim 11, US’316 discloses ‘A sound generation (US’316, ¶[0338]:” a sound gathering device, such as a microphone, with a nearby processor for engaging in cooperative/distributed computing of source signal”) device comprising: at least one processor configured to receive a first feature amount sequence in which a musical feature amount changes over time (US’316, ¶[0247]:” frequency, amplitude ... amplitude modulation, frequency modulation, and the phase of the oscillator...", teaches musical feature amounts; ¶[0246]:” knowledge of the behavior of that oscillator in previous sample windows...", teaches feature information changing over time),
US’316 does not expressly disclose ‘and use a trained model that has learned an input-output relationship between an input feature amount sequence in which the musical feature amount changes over time at a first fineness and a reference sound data sequence corresponding to an output feature amount sequence in which the musical feature amount changes over time at a second fineness that is higher than the first fineness, to process the first feature amount sequence, thereby generating a sound data sequence corresponding to a second feature amount sequence in which the musical feature amount changes at the second fineness.
However, US’557 discloses ‘and use a trained model that has learned an input-output relationship between an input feature amount sequence in which the musical feature amount changes over time at a first fineness and a reference sound data sequence corresponding to an output feature amount sequence in which the musical feature amount changes over time at a second fineness that is higher than the first fineness, to process the first feature amount sequence (US’557, col. , lines :"The encoding can be referred to as a temporal encoding because the result is a sequence of hidden codes..." teaches a feature sequence; col. , lines :” downsampling with average pooling to get the embedding or encoding Z", first fineness, col. , lines :” The time resolution depends on the stride of the pooling”, teaches different fineness levels; col. , lines :”upsampling ... the embedding to an original resolution of the input audio waveform", teaches a second fineness higher than the first fineness; col. , lines :”a machine learned neural synthesizer model comprising an autoencoder model...", teaches a trained model),
thereby generating a sound data sequence corresponding to a second feature amount sequence in which the musical feature amount changes at the second fineness (US’557, col. 1, lines 61-63:”receive the first embedding and autoregressively generate a first audio waveform...", teaches generation of sound data; col. 2, lines 1-3:” receiving ... the first audio waveform as an output of the decoder neural network", generated sound output corresponding to the higher-resolution representation). (Claim 11 corresponds to claim 1)
Regarding claim 12, US’316 (in view of US’557) discloses ‘The sound generation device according to claim 11, as discussed above.
wherein the musical feature amount at each time point in the input feature amount sequence indicates a representative value of the musical feature amount within each prescribed time period including each time point. (Claim 12 corresponds to claim 2)
Regarding claim 13, US’316 (in view of US’557) discloses ‘The sound generation device according to claim 12, as discussed above.
US’316 (in view of US’557) US’557 further discloses ‘wherein the representative value indicates a statistical value of the musical feature amount within each prescribed time period in the output feature amount sequence (US’557, col. 8, lines 42-44:"The output feed into another 1×1 convolution before downsampling with average pooling to get the embedding or encoding Z", teaches generation of an embedding using an average pooling operation; col. 8, lines 44-48:"The encoding can be referred to as a 'temporal encoding' because the result is a sequence of hidden codes with separate dimensions for time and channel. The time resolution depends on the stride of the pooling”, teaches a temporal representation corresponding to a period of time).
Regarding claim 15, US’316 (in view of US’557) discloses ‘The sound generation device according to claim 11, as discussed above.
wherein each of the first fineness and the second fitness indicates a frequency of change of the musical feature amount within a unit of time, or a content ratio of a high-frequency component of the musical feature amount within the unit of time. (Claim 15 corresponds to claim 5)
Regarding claim 16, US’316 (in view of US’557) discloses ‘The sound generation device according to claim 11, as discussed above.
wherein the at least one processor is further configured to convert the sound data sequence representing a frequency-domain waveform into a time-domain waveform. (Claim 16 corresponds to claim 6)
Regarding claim 17, US’316 discloses ‘A training device (US’316, ¶[00363]:” speaker recognition training may be generated through device use”) comprising: at least one processor configured to extract, from reference data representing a sound waveform, a reference sound data sequence in which a musical feature amount changes over time at a prescribed fineness and an output feature amount sequence which is a time series of the musical feature amount,
generate, from the output feature amount sequence, an input feature amount sequence in which the musical feature amount changes over time at a lower fineness than the prescribed fineness,
and construct a trained model that has learned an input-output relationship between the input feature amount sequence and the reference sound data sequence by machine learning that uses the input feature amount sequence and the reference sound data sequence. (Claim 17 corresponds to claim 7)
Regarding claim 18, US’316 (in view of US’557) discloses ‘The training device according to claim 17, as discussed above.
wherein to generate the input feature amount sequence, the at least one processor is configured to extract, as the musical feature amount at each time point in the input feature amount sequence, a representative value of the musical feature amount within each prescribed time period including each time point in the output feature amount sequence. (Claim 18 corresponds to claim 8)
Regarding claim 19, US’316 discloses ‘A non-transitory computer readable medium storing a sound generation program that causes one or a plurality of computers to perform operations (US316’ ¶[0373]:” a computer processor … and a storage medium readable … computer program may be a set of … memory cards, or flash drives”) comprising: receiving a first feature amount sequence in which a musical feature amount changes over time;
and using a trained model that has learned an input-output relationship between an input feature amount sequence in which the musical feature amount changes over time at a first fineness
and a reference sound data sequence corresponding to an output feature amount sequence in which the musical feature amount changes over time at a second fineness that is higher than the first fineness to process the first feature amount sequence,
thereby generating a sound data sequence corresponding to a second feature amount sequence in which the musical feature amount changes at the second fineness. (Claim 19 corresponds to claim 1)
The rationale set forth for Claim 1 is equally applicable here because the claimed computer-readable medium merely stores instructions for performing the previously discussed method.
Regarding claim 20, US’316 discloses ‘A non-transitory computer readable medium storing a training program that causes one or a plurality of computers to perform operations (US316’ ¶[0373]:” a computer processor … and a storage medium readable … computer program may be a set of … memory cards, or flash drives”) comprising: extracting, from reference data representing a sound waveform (US'316, ¶[0111]:"generate high resolution frequencies for subsequent signal extraction", teaches extracting information from reference audio waveform data),
a reference sound data sequence in which a musical feature amount changes over time at a prescribed fineness and an output feature amount sequence that is a time series of the musical feature amount (US'316, ¶[0247]:” frequency, amplitude ... amplitude modulation, frequency modulation...", ; ¶[0246]:”falling in frequency ... continue falling", teaches a time-series of musical feature information);
US’316 does not expressly disclose ‘generating, from the output feature amount sequence, an input feature amount sequence in which the musical feature amount changes over time at a lower fineness than the prescribed fineness; and constructing a trained model that has learned an input-output relationship between the input feature amount sequence and the reference sound data sequence by machine learning that uses the input feature amount sequence and the reference sound data sequence.
However, US’557 discloses ‘generating, from the output feature amount sequence, an input feature amount sequence in which the musical feature amount changes over time at a lower fineness than the prescribed fineness (US’557, col. 8, lines 43-44:"downsampling with average pooling to get the embedding or encoding Z", teaches generation of a lower-fineness feature representation through downsampling; lower fineness);
and constructing a trained model that has learned an input-output relationship between the input feature amount sequence and the reference sound data sequence by machine learning that uses the input feature amount sequence and the reference sound data sequence (US’557, col. 1, lines 36-37: "a machine learned neural synthesizer model that includes a autoencoder model..."; col. 1, lines 53-54: "adjusting one or more parameters of the autoencoder model to improve the loss function", teaches machine-learning training of the model using training data).
The rationale set forth for Claim 1 is equally applicable here because the claimed computer-readable medium merely stores instructions for performing the previously discussed method.
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over US’316, in view of US’557, and in further view of US9105259 (Akazawa), hereinafter US’259.
Regarding claim 4, US’316 (in view of US’557) discloses ‘The sound generation method according to claim 1, as discussed above.
US’316 (in view of US’557) does not expressly disclose ‘further comprising presenting a reception screen in which the first feature amount sequence is displayed along a time axis, wherein the receiving of the first feature amount sequence is performed by input of a user via the reception screen.
However, US’259 discloses ‘further comprising presenting a reception screen in which the first feature amount sequence is displayed along a time axis, wherein the receiving of the first feature amount sequence is performed by input of a user via the reception screen (US’259, col. 1, lines 37-40:” displaying, on a display device, a musical note sequence image in which a musical note iconic image of each musical note is disposed in a musical score area where a time axis is set”, col. 2, lines 52-55:” “accepts an instruction from a user” and changes the display length or position “in a direction of the time axis.”, a reception screen displaying musical sequence information along a time axis and receiving user input through the screen)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to provide the musical feature sequence of US’316 (in view of US’557) through the timeline-based music editing interface of US’259 because such interfaces allow users to visually confirm and edit time-varying musical information in a user-friendly manner.
Regarding claim 14, US’316 (in view of US’557) discloses ‘The sound generation device according to claim 11, as discussed above.
wherein the at least one processor is further configured to present a reception screen in which the first feature amount sequence is displayed along a time axis,
and the at least one processor is configured to receive the first feature amount sequence through input of a user via the reception screen. (Claim 14 corresponds to claim 4)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICOLE K GILLESPIE whose telephone number is (571)482-4187. The examiner can normally be reached Monday-Friday 7:30-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Dedei K Hammond can be reached at (571)270-3819. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICOLE K GILLESPIE/Examiner, Art Unit 2837
/DEDEI K HAMMOND/Supervisory Patent Examiner, Art Unit 2837
Prosecution Insights