DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
2. The information disclosure statement (IDS) submitted on 09/05/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority Acknowledgment
3. Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application EP22162999.1 filed on 03/18/2022.
Claim Rejections - 35 USC § 101
4. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
5. Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. Claim 15 recites “15. A program having a program code for performing the method of claim 13, when the program is executed on a processor or a programmable hardware.” Claim 15 encompass nothing more than logic/software modules as per the specification. Page 28 of the Specification discloses “[0133] Examples may further be or relate to a (computer) program including a program code to execute one or more of the above methods when the program is executed on a computer, processor or other programmable hardware component. Thus, steps, operations or processes of different ones of the methods described above may also be executed by programmed computers, processors or other programmable hardware components. Examples may also cover program storage devices, such as digital data storage media, which are machine-, processor - or computer-readable and encode and/or contain machine-executable, processor-executable or computer-executable programs and instructions. Program storage devices may include or be digital storage devices, magnetic storage media such as magnetic disks and magnetic tapes, hard disk drives, or optically readable digital data storage media, for example. Other examples may also include computers, processors, control units, (field) programmable logic arrays ((F) PLAs), (field) programmable gate arrays ((F) PGAs), graphics processor units (GPU), application-specific integrated circuits (ASICs), integrated circuits (ICs) or system-on-a-chip (SoCs) systems programmed to execute the steps of the methods described above.”
Thus, claim 15 is directed to non-statutory subject matter because their scope includes a computer program embodiment, an abstract data structure which does not fall within one of the four statutory categories (i.e., it is directed to a program per se). See also MPEP § 2106.IV.B.1.a. Data structures not claimed as embodied in computer readable media are descriptive material per se and are not statutory because they are not capable of causing functional change in the computer. See, e.g., Warmerdam, 33 F.3d at 1361, 31 USPQ2d at 1760 (claim to a data structure per se held nonstatutory). Such claimed data structures do not define any structural and functional interrelationships between the data structure and other claimed aspects of the invention, which permit the data structure's functionality to be realized. In contrast, a claimed computer readable medium encoded with a data structure defines structural and functional interrelationships between the data structure and the computer software and hardware components which permit the data structure's functionality to be realized, and is thus statutory. Similarly, computer programs claimed as computer listings per se, i.e., the descriptions or expressions of the programs are not physical “things.” They are neither computer components nor statutory processes, as they are not “acts” being performed. Such claimed computer programs do not define any structural and functional interrelationships between the computer program and other claimed elements of a computer, which permit the computer program's functionality to be realized.
Claim Rejections - 35 USC § 102
6. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
7. Claims 1-2, 4-6 and 13-15 are rejected under 35 U.S.C. 102(a) (2) as being anticipated by Kim et al. (US 2023/0206896 A1.)
With respect to Claim 1, Kim et al. disclose
An apparatus for end-to-end text-to-speech synthesis, comprising: input interface circuitry configured to receive:
first input data indicative of a phoneme (Kim et al. [0087] describes receiving the divided input text, wherein the divided input text is/are syllable, character and/or phoneme); and
second input data indicative of a first target duration for the phoneme (Kim et al. [0033-0034 and 0125] describes a target of the duration of the synthesized voice);
processing circuitry configured to, using a trained machine-learning model:
map the phoneme to a state using an encoder sub-model of the trained machine-learning model (Kim et al. [0087] descries using a machine learning model to generate the character embedding based on the received phoneme(s));
estimate a second target duration for the phoneme based on the state (Kim et al. [0124] describes inputting the embedding of each of the plurality of phonemes to the artificial neural network duration prediction model to predict the duration of each of the plurality of phonemes);
determine an attention weight based on the first target duration and the second target duration (Kim et al. [0125] describes using the length regulator (i.e., hyperparameter to adjust the speech of voice) to generate a plurality of frames based on the predicted duration and the requirement frames, Fig. 10 element 1020 Length Regulator); and
map the state to audio data based on the attention weight using a decoder sub-model of the trained machine-learning model, wherein the audio data are indicative of an audio waveform representing speech (Kim et al. [0125] describes mapping the embedding for each of the plurality to phonemes based on the hyperparameter to audio data and generating the synthesized voice based on the voice data, Fig. 9 element 930 decoder, 720 vocoder, Fig. 10 element 930 decoder.)
With respect to Claim 2, Kim et al. disclose
wherein the first input data are indicative of a text to be converted into the speech, and wherein the processing circuitry is further configured to determine the phoneme based on the text (Kim et al. [0087] describes dividing the input text into the syllable, character and/or phoneme.)
With respect to Claim 4, Kim et al. disclose
wherein the second input data are indicative of a video depicting a speaking person, and wherein the processing circuitry is further configured to determine the first target duration based on the video to synchronize the speech to the video (Kim et al. [0007] describes determining the number of frames including a speaker’s mouth shape, and [0048] describes the speaker’s mouth shape and the synthesis voice included in each frame included in the video content may be synchronized.)
With respect to Claim 5, Kim et al. disclose
wherein, for determining the first target duration, the processing circuitry is configured to:
determine a gesture of the speaking person matching the phoneme based on the video (Kim et al. [0082] determining an image frame including the speaker’s mouth shape); and
determine the first target duration based on the determined gesture (Kim et al. [0082] determining a number of frames including the speaker’s mouth shape.)
With respect to Claim 6, Kim et al. disclose
wherein the processing circuitry is configured to:
determine a second phoneme matching a shape of lips of the speaking person based on the video (Kim et al. [0007-0008] describes determining a phoneme corresponding with the speaker’s mouth shape);
determine a correlation between the phoneme and the second phoneme (Kim et al. [0047-0048] determining a relationship between the phoneme in the input text and the phoneme corresponding with the speaker’s mouth shape); and
determine the first target duration based on the correlation (Kim et al. [0053 and 0071] determining the target duration (i.e. a number of frames) in the synthesized voice.)
With respect to Claim 13, Claim 13 recites the similar features as Claim 1, thus Claim 13 is rejected as the same ground as Claim 1.
With respect to Claim 14, Claim 14 is a non-transitory machine-readable medium claim for performing the method of Claim 13, thus Claim 14 is rejected as the same ground as Claim 13.
With respect to Claim 15, Claim 15 is a program claim for performing the method of Claim 13, thus Claim 15 is rejected as the same ground as Claim 13.
Claim Rejections - 35 USC § 103
8. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
9. Claim 3 is rejected under 35 U.S.C.103 as being unpatentable over Kim et al. (US 2023/0206896 A1) in view of Bruckert (6,029,131).
With respect to Claim 3, Kim et al. disclose all the limitations of 2 upon which Claim 3 depends. Kim et al. fail to explicitly teach
wherein the second input data are indicative of a stress to be given to the text or parts thereof, wherein the processing circuitry is further configured to determine the first target duration based on the stress.
However, Bruckert discloses
wherein the second input data are indicative of a stress to be given to the text or parts thereof, wherein the processing circuitry is further configured to determine the first target duration based on the stress (Bruckert Fig. 6A and 6C, Col. 7 lines 1-16 and lines 28-34 describes adjusting the time durations of the phonemes based on the stress time intervals.)
Kim et al. and Bruckert are analogous art because they are from a similar field of endeavor in the Speech Processing techniques and applications. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the steps of adjusting the duration of the synthesized speech as taught by Kim et al., using teaching of determining the stress time intervals as taught by Bruckert for the benefit of adjusting the time durations of the phonemes in the synthetic speech (Bruckert Fig. 6A and 6C, Col. 7 lines 1-16 and lines 28-34 describes adjusting the time durations of the phonemes based on the stress time intervals.)
10. Claim 7 is rejected under 35 U.S.C.103 as being unpatentable over Kim et al. (US 2023/0206896 A1) in view of Brochu (US 10,276,189 B1.)
With respect to Claim 7, Kim et al. disclose all the limitations of Claim 6 upon which Claim 7 depends. Kim et al. fail to explicitly teach
wherein the processing circuitry is configured to determine the correlation based on dynamic time warping.
However, Brochu teaches
wherein the processing circuitry is configured to determine the correlation based on dynamic time warping (Brochu Col. 10 lines 66-67, Col. 11 lines 1-19 describes using the dynamic time warping to measure similarity and/or distance between two time domain sequences in the video files and the audio files.)
Kim et al. and Brochu are analogous art because they are from a similar field of endeavor in the Speech Processing techniques and applications. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the steps of adjusting the duration of the synthesized speech as taught by Kim et al., using teaching of the dynamic time warping as taught by Brochu for the benefit of measuring similarity between two time domain sequences in the video files and the audio files (Brochu Col. 10 lines 66-67, Col. 11 lines 1-19 describes using the dynamic time warping to measure similarity and/or distance between two time domain sequences in the video files and the audio files.)
11. Claim 8 is rejected under 35 U.S.C.103 as being unpatentable over Kim et al. (US 2023/0206896 A1) in view of Saino et al. (US 2023/0098145 A1.)
With respect to Claim 8, Kim et al. disclose all the limitations of Claim 1 upon which Claim 8 depends. Kim et al. fail to explicitly teach
wherein the processing circuitry is configured to estimate the second target duration based on at least one of a differentiable function and a stochastic process.
However, Saino et al. teach
wherein the processing circuitry is configured to estimate the second target duration based on at least one of a differentiable function and a stochastic process (Saino et al. [0038] describes using a statistical estimation model to estimate a duration of each phoneme.)
Kim et al. and Saino et al. are analogous art because they are from a similar field of endeavor in the Speech Processing techniques and applications. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the steps of adjusting the duration of the synthesized speech as taught by Kim et al., using teaching of the statistical estimation model as taught by Saino et al. for the benefit of estimation the duration of each phoneme (Saino et al. [0038] describes using a statistical estimation model to estimate a duration of each phoneme.)
12. Claim 9 is rejected under 35 U.S.C.103 as being unpatentable over Kim et al. (US 2023/0206896 A1) in view of Elias et al. (US 2022/0301543 A1.)
With respect to Claim 9, Kim et al. disclose all the limitations of Claim 1 upon which Claim 9 depends. Kim et al. fail to explicitly teach
wherein the processing circuitry is configured to map the state to the audio data by resampling the state based on the attention weight.
However, Elias et al. teach
wherein the processing circuitry is configured to map the state to the audio data by resampling the state based on the attention weight (Elias et al. [0004] describes upsampling the sequence representation of an encoded text into an upsampled output specifying a number of frames using the interval representation matrix and the auxiliary attention context representation.)
Kim et al. and Elias et al. are analogous art because they are from a similar field of endeavor in the Speech Processing techniques and applications. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the steps of adjusting the duration of the synthesized speech as taught by Kim et al., using teaching of the upsampling as taught by Elias et al. for the benefit of generating one or more predicted mel-frequency spectrogram sequences for the encoded text sequence (Elias et al. [0004] describes upsampling the sequence representation of an encoded text into an upsampled output specifying a number of frames using the interval representation matrix and the auxiliary attention context representation and generating one or more predicted mel-frequency spectrogram sequences for the encoded text sequence.)
13. Claim 10 is rejected under 35 U.S.C.103 as being unpatentable over Kim et al. (US 2023/0206896 A1) in view of Zhang et al. (US 2022/0059072 A1.)
With respect to Claim 10, Kim et al. disclose all the limitations of Claim 1 upon which Claim 9 depends. Kim et al. fail to explicitly teach
wherein the processing circuitry is configured to determine the attention weight by estimating a probability that the state is aligned with a predefined frame for the audio waveform.
However, Zhang et al. teach
wherein the processing circuitry is configured to determine the attention weight by estimating a probability that the state is aligned with a predefined frame for the audio waveform (Zhang et al. [0048, 0053] and Fig. 4 describes estimating a probability that the speech frames of the speech are aligned with the characters of the text in order to determining an importance index of each weight in the first weight matrix, and generating a second weight matrix based on the first weight matrix.)
Kim et al. and Zhang et al. are analogous art because they are from a similar field of endeavor in the Speech Processing techniques and applications. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the steps of adjusting the duration of the synthesized speech as taught by Kim et al., using teaching of the probability as taught by Zhang et al. for the benefit of estimating how similar between the speech frames of the speech and the characters of the text in order to evaluate the accuracy of the speech model (Zhang et al. [0048, 0053] and Fig. 4 describes estimating a probability that the speech frames of the speech are aligned with the characters of the text in order to determining an importance index of each weight in the first weight matrix, and generating a second weight matrix based on the first weight matrix. Zhang et al. indicates that the greater the probability is higher accuracy of the speech model.)
Allowable Subject Matter
14. Claims 11 and 12 are objected as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the prior art(s) taken alone or in combination fail(s) to teach the following element(s) in combination with the other recited elements in the claim(s).
“wherein the processing circuitry is configured to determine a second attention weight based on the second target duration and determine the attention weight by modifying the second attention weight based on the first target duration.” as recited in Claim 11.
Claim 12 depends on Claim 11, and thus Claim 12 is objected to as being dependent upon an objected claim(s) by virtue of their dependency.
Conclusion
15. The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. See PTO-892.
a. Jia et al. (US 2023/0013777 A1.) In this reference, Jia et al. disclose a method for speech-to-speech translation. The S2ST in Jia et al. receives the context vector and the phoneme representation and generate a translated synthesized speech representation that corresponds to a translation of the utterance spoken in the different second language.
b. Zhang et al. (US 2022/0108680 A1.) In this reference, Zhang et al. disclose a method for text-to-speech using duration prediction. Zhang et al. controls the pace of synthesized audio on a per-word or per-phoneme level by modifying the predicted durations of each word or phoneme determined by a duration prediction neural network, while still maintaining the naturalness of the synthesized speech.
c. Gu et al. (US 2019/0371292 A1.) In this reference, Gu et al. disclose a method for synthesizing the speech. In this reference, Gu et al. predicts a time length of a state of each phoneme corresponding to a target text and synthesize speech corresponding to the target text according to the predicted time length.
16. Any inquiry concerning this communication or earlier communications from the examiner should be directed to THUYKHANH LE whose telephone number is (571)272-6429. The examiner can normally be reached Mon-Fri: 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew C. Flanders can be reached on 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THUYKHANH LE/Primary Examiner, Art Unit 2655