Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is sent in response to Applicant’s communication received on 7/11/2024 for the application number 18769406. The office hereby acknowledges receipt of the following placed of record in the file: Specification, Abstract, Oath/Declaration and claims.
Status of the claims
Claims 1-20 are presented for examination.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 7/11/2024 & 12/23/2024 were filed before the mailing date of the first office action. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claim(s) 1, 2, 4, 10, 11, 12, 14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Arik et al. (US 20190251952 B1) in view of Huang et al. (WO 2019227547 A1) and Rao et al. (US 12499904 B1).
Regarding claim 1, Arik teaches A voice mixing conversion system, comprising: a voice input unit configured to receive voice data and an unknown test audio file (Para 0072, “which receives as inputs, for a speaker, a training set of text-audio pairs and a corresponding speaker identifier” and para 0084 “Presented herein are speaker encoding embodiment methods to directly estimate the speaker embedding from audio samples of an unseen speaker.”); a memory configured to store a pre-training model (Para 0080, “the pre-trained multi-speaker model parameters”); reading the pre-training model, inputting the training audio file into the pre-training model, and training the pre-training model to become a trained model using the training audio file and a verifying audio file (Para 0080, “the pre-trained multi-speaker model parameters, including the speaker embedding parameters, may be fine-tuned (410/510) using a set of text-audio pairs 550 & 555 for a previously unseen speaker” and “During training, a set of cloning audio samples .sub.s.sub.i are randomly sampled for training speaker s.sub.i. During inference, .sub.s.sub.k, audio samples from the target speaker s.sub.k, is used to compute g(.sub.s.sub.k; 0)” where the audio samples from the target speaker correspond to the verifying audio file, generally the inference and training process is the same other than minor weight distribution differences, see para 180/210); and performing an inference via the trained model using a denoised and separated audio file to be processed and the verifying audio file to obtain an initial generated speech (Para 0087, “During training, a set of cloning audio samples .sub.s.sub.i are randomly sampled for training speaker s.sub.i. During inference, .sub.s.sub.k, audio samples from the target speaker s.sub.k, is used to compute g(.sub.s.sub.k; 0)” where elsewhere in para 0055, “A goal of voice conversion is to modify an utterance from source speaker to make it sound like the target speaker… Those models are typically trained with a large amount of audio pairs of target and source speakers.”);
Arik does not teach, a processor coupled to the memory and the voice input unit configured to perform the following steps: performing a data pre-processing on the voice data comprising: removing a plurality of silent segments in the voice data, merging and normalizing the voice data with the silent segments removed, and then performing a frequency sampling rate conversion on the merged and normalized voice data to generate a training audio file; performing a speech denoising and separation on the unknown test audio file, performing a noise reduction processing on the initial generated speech based on a noise threshold to generate a post-noise reduction speech.
However, Huang teaches a processor coupled to the memory and the voice input unit (Pg. 20 Para 2: “For example, the computer device 5 may further include an input-output device, a network access device, and a bus.”) configured to perform the following steps: performing a data pre-processing on the voice data (Pg. 23 Para 12: “A voice file preprocessing module, configured to preprocess the voice file to obtain audio data”) comprising: removing a plurality of silent segments in the voice data, merging and normalizing the voice data with the silent segments removed, and then performing a frequency sampling rate conversion on the merged and normalized voice data to generate a training audio file (Ph. 11 Para 4: “the audio data pair is normalized, and the audio data is divided into multiple voice frames to improve the efficiency of data processing. … If the frame energy of the voice frame is less than a preset frame energy threshold, mark the voice frame as a mute frame … the voice file is segmented according to the segmented frame to obtain a target file.” and “The specific sampling frequency can be set based on historical experience”); performing a speech denoising and separation on the unknown test audio file (Teaches the speech denoising and separation in the same quotation on Pg. 11 Para 4) performing a noise reduction processing on the initial generated speech based on a noise threshold to generate a post-noise reduction speech (Pg. 26, Para 5: “If the calculated frame energy is less than the frame energy threshold, the corresponding voice frame is marked as a mute frame.”, Arik already teaches the initial generated speech).
It would have been obvious to a person of ordinary skill in the art to modify Arik before the effective filing date in such a way as to incorporate the teachings of Huang in order to achieve the benefit of improving the efficiency of data processing (Pg. 11 Para 4) and ensuring that the silence is correctly identified and the audio can be accurately segmented (See Pg. 12 Para 1).
Arik modified by Huang does not teach calculating a first quality score and a second quality score, wherein in response to a number of the initial generated speech being 1, the first quality score is calculated based on the initial generated speech and the second quality score is calculated based on the post-noise reduction speech; and determining whether the first quality score is greater than the second quality score, and outputting the initial generated speech, otherwise outputting the post-noise reduction speech.
However, Rao teaches calculating a first quality score and a second quality score (Col. 11, Ln. 42-45, “The first ML audio quality assessment may include a first quality score for the first audio content and/or a first quality degradation reason for the first audio content.” And Col. 12, Ln. 38-40, “The second ML audio quality assessment may include a second quality score for the second audio content”), wherein in response to a number of the initial generated speech being 1, the first quality score is calculated based on the initial generated speech (Col. 11, Ln. 46-48, “In some examples, the first audio content may be input audio content 301 of FIG. 3, which is input to an audio enhancement ML model”), and the second quality score is calculated based on the post-noise reduction speech (Col. 11, Ln. 65-67, “the first audio content may be modified by an audio enhancement ML model to form the second audio content.” Where noise reduction comprises an audio enhancement.); and determining whether the first quality score is greater than the second quality score (Para 25, “For example, if the output ML quality score 351 is substantially higher than an input ML quality score 341, then this may be an indication that the audio enhancement ML model 320 is operating in a desired manner, and a use of the current audio enhancement functions may be propagated.”); and outputting the initial generated speech or the pre-noise reduction mixed audio file in a case that the first quality score is greater than the second quality score (Rao sends the chosen score, see para 25).
It would have been obvious to a person of ordinary skill in the art to modify Arik before the effective filing date in such a way as to incorporate the teachings of Rao in order to output the best possible audio in the event that an audio enhancement is in not producing a higher quality result (Para 25, “if the output ML quality score 351 is substantially higher than an input ML quality score 341, then this may be an indication that the audio enhancement ML model 320 is operating in a desired manner, and a use of the current audio enhancement functions may be propagated.”).
Regarding claim 2, Arik teaches a professional recording equipment coupled to the processor and the memory and configured to capture a plurality of voice signals of different people (Para 0084, " Presented herein are speaker encoding embodiment methods to directly estimate the speaker embedding from audio samples of an unseen speaker.” Where the presence of audio samples is understood to be able to originate from a microphone, and Para 0165, “Also, increasing the amount and diversity of speakers tends to enable a more meaningful speaker embedding space” which teaches that there can be a large amount of different speakers); wherein the processor forms the voice data according to the voice signals and stores the voice data in a voice database of the memory (Para 0113, “The speaker verification model may be trained on a multi-speaker dataset”, a multi-speaker dataset can comprise a voice database.).
Regarding claim 4, Arik teaches wherein the processor is further configured to read a speaker embedding vector of the training audio file via the pre-training model (Para 0080, “in one or more embodiments, the pre-trained multi-speaker model parameters, including the speaker embedding parameters, may be fine-tuned (410/510) using a set of text-audio pairs 550 & 555 for a previously unseen speaker.” And para 0076, “Having fine-tuned the speaker embedding parameters to produce a speaker embedding 330 for the new speaker”) and train with a multi-head attention mechanism and a multiple combination loss function to generate a generated audio file (Fig. 13 teaches multi-head attention mechanism, and multiple loss is supported by para 0205, “An L1 loss may be computed using the output mel-spectrograms, and a binary cross-entropy loss may be computed using the final-frame prediction. L1 loss was selected since it yielded the best result empirically. Other losses, such as L2, may suffer from outlier spectral features, which may correspond to non-speech noise.”).
Regarding claim 10, Arik teaches a communication interface coupled to the processor and configured to receive the unknown test audio file via a network (Para 0171 “For example, input data and/or output data may be remotely transmitted from one physical location to another. In addition, programs that implement various aspects of the disclosure may be accessed from a remote location (e.g., a server) over a network.”).
Claim 11 is analogous to claim 1 in that it recites substantially the same limitations. It is therefore rejected for the same reasons set forth above.
Claim 12 is analogous to claim 2 in that it recites substantially the same limitations. It is therefore rejected for the same reasons set forth above.
Claim 14 is analogous to claim 4 in that it recites substantially the same limitations. It is therefore rejected for the same reasons set forth above.
Claim 20 is analogous to claim 10 in that it recites substantially the same limitations. It is therefore rejected for the same reasons set forth above.
Claim 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Arik (US 20190251952 B1), Huang (WO 2019227547 A1) and Rao (US 12499904 B1) as above in claim 1 & 11, and further in view of Lanzi et al. (US 20250342823 A1).
Huang teaches removing the silent segments in a middle of the voice data so that the voice data becomes a plurality of first sub-audio files (Pg. 11 Para 4: “the audio data is divided into multiple voice frames to improve the efficiency of data processing”);
Arik modified by Huang and Rao does not teach merging the first sub-audio files sequentially to form a second sub-audio file after the silent segments at a beginning and an end of the first sub-audio files are removed; and remove the silent segments at a beginning and an end of the second sub-audio file.
However, Lanzi teaches merging the first sub-audio files sequentially to form a second sub-audio file after the silent segments at a beginning and an end of the first sub-audio files are removed (Para 0410, “Concatenation can then occur through stitching together the cleaned audio chunks to form a coherent audio stream without gaps”); and remove the silent segments at a beginning and an end of the second sub-audio file (Para 0410, “Audio refinement may involve applying Silero VAD again to queued audio to eliminate any residual noise or silence,”).
It would have been obvious to a person of ordinary skill in the art to modify Hillman Beauchesne before the effective filing date in such a way as to incorporate the teachings of Lanzi in order to further refine the data and thus the system as a whole (Para 0410, “to eliminate any residual noise or silence, refining the data further”).
Claim 13 is analogous to claim 3 in that it recites substantially the same limitations. It is therefore rejected for the same reasons set forth above.
Claim 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Arik (US 20190251952 B1), Huang (WO 2019227547 A1) and Rao (US 12499904 B1) as above in claim 1 & 11, and further in view of Zahedi et al. (US 20220240026 A1).
Arik modified by Huang, Rao does not teach wherein the processor is further configured to weight each of the initial generated speeches and each of the post-noise reduction speeches with different proportions of weights for mixing.
However, Zahedi teaches weighting each of the initial generated speeches and each of the post-noise reduction speeches with different proportions of weights for mixing (Para 5, “The output of the resulting enhancement system is composed of two components: the original microphone signal and a processed version, where noise is suppressed. These two components are then dynamically combined to produce an output signal that adapts to the situation: When there is a lot of noise (and therefore noise is interfering with speech intelligibility), the dynamic combination leans toward the noise-reduced component. When there is not much noise (and therefore noise is perceived as benign ambient sound), the dynamic combination leans toward the original unprocessed microphone signal.” This describes a weighted sum and para 0006 explicitly describes these as weights, “The sum of the weights of the linear combination may be equal to one.”)
It would have been obvious to a person of ordinary skill in the art to modify Arik before the effective filing date in such a way as to incorporate the teachings of Zahedi in order to allow more natural output. (Para 0005, Zahedi).
Claim 15 is analogous to claim 5 in that it recites substantially the same limitations. It is therefore rejected for the same reasons set forth above.
Claim 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Arik (US 20190251952 B1), Huang (WO 2019227547 A1) and Rao (US 12499904 B1) as above in claim 1 & 11, and further in view of Pu et al. (CN 114116424 A).
Arik modified by Huang, and Rao does not teach wherein the first quality score and the second quality score are both generated by mixing and calculating a subjective score and an objective score.
However, Pu teaches wherein the first quality score and the second quality score are both generated by mixing and calculating a subjective score and an objective score (Pg. 18 Para 3: “calculating the subjective score and objective score, determining the car audio quality score.” Where Pg. 18 Para 4: “subjective evaluation method and multi-dimensional objective evaluation are combined, fast and reliable giving audio quality score”).
It would have been obvious to a person of ordinary skill in the art to modify Arik before the effective filing date in such a way as to incorporate the teachings of Pu in order to improve the accuracy of the test audio (Pg. 19, Para 2).
Claim 16 is analogous to claim 6 in that it recites substantially the same limitations. It is therefore rejected for the same reasons set forth above.
Claim 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Arik (US 20190251952 B1), Huang (WO 2019227547 A1) and Rao (US 12499904 B1) as above in claim 1 & 11, and further in view of Song et al. (US 20240161727 A1)
Arik modified by Huang, and Rao does not teach wherein the pre-training model comprises a plurality of discriminators, and a plurality of feature layers are obtained via the discriminators.
However, Song teaches wherein the pre-training model comprises a plurality of discriminators (Para 00035, 107, decoder has plural discriminators.), and a plurality of feature layers are obtained via the discriminators. (Para 0107-0109, “In this way, frequency domain features of different resolutions in the predicted speech may be learnt, and then discrimination may be implemented.”).
It would have been obvious to a person of ordinary skill in the art to modify Arik before the effective filing date in such a way as to incorporate the teachings of Song in order to improve speech discrimination accuracy by learning a multitude of features (see para 108).
Claim 18 is analogous to claim 8 in that it recites substantially the same limitations. It is therefore rejected for the same reasons set forth above.
Claim 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Arik (US 20190251952 B1), Huang (WO 2019227547 A1) and Rao (US 12499904 B1) as above in claim 1 & 11, and further in view of Kumar et al. (US 20220101872 A1)
Arik modified by Huang, and Rao does not teach wherein when performing the frequency sampling rate conversion of the voice data, the processor is further configured to: upsample the second sub-audio file to 44100 HZ.
However, Kumar (US 20220101872 A1) teaches wherein when performing the frequency sampling rate conversion of the voice data, the processor is further configured to: upsample the second sub-audio file to 44100 HZ (Para 0041, “For example, the individual may be able to select an audio file that is representative of a recording of a speaker and then specify that the audio file should be upsampled to 44,100 Hz”).
It would have been obvious to a person of ordinary skill in the art to modify Arik before the effective filing date in such a way as to incorporate the teachings of Kumar in order to allow inputs/outputs in a wider variety of formats as 44,100 HZ is the standard sampling rate for compact disc (CD) audio (Para 0020, “while compact disc (CD) audio is normally sampled at a rate of 44,100 Hz.”).
Claim 19 is analogous to claim 9 in that it recites substantially the same limitations. It is therefore rejected for the same reasons set forth above.
Allowable Subject Matter
Claims 7, 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Su (CN 114242084 A) teaches wherein the subjective score is related to Perceptual Evaluation of Speech Quality (PESQ), and the objective score is related to Mel-Cepstral distortion (MCD). (Pg. 14 Para 2: “the embodiment of the invention uses the voice subjective quality evaluation PESQ-MOS (Mobility evaluation of speech-mean ) and average cepstrum distortion(Mel-cepstrum) is evaluation index” where “MCD is on the basis of speech data distortion measure, using distance criteria to measure the similar degree of speech before and after the steganography”. Here PESQ is a subjective quality evaluation, and MCD is based on specific measurements and can thus be interpreted as an objective score).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL ALAN FOSTER JR. whose telephone number is (571)272-8874. The examiner can normally be reached M - Th 8:00am - 6:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached at (571) 272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL A FOSTER JR/ Examiner, Art Unit 2654
/HAI PHAN/ Supervisory Patent Examiner, Art Unit 2654
Prosecution Insights