DETAILED ACTION
The Response filed on 04/02/2026 has been correspondingly accepted and considered in the office action. Claims 1-24 are pending and Claims 1, 8, 15, and 22 are independent.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The objection to Specification as Abstract exceeding 150 words has been withdrawn because the objection was issued incorrectly by the Examiner in the previous office action.
Claims 1-24 stand rejected under 35 U.S.C. § 103. Applicant’s arguments with respect to Claims 1-24 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. In order to expedite prosecution, and as to the material from the Specifications that are not in the Claim and are argued by the Applicant, please note Wang et al., (US Pat No. 8,224,652, hereinafter, Wang), Marple et al., (US Pat No. 7,877,259, hereinafter, Marple), Yoon et al., ("Speech gesture generation from the trimodal context of text, audio, and speaker identity." ACM Transactions on Graphics (TOG) 39.6 (2020): 1-16, hereinafter, Yoon), and Ren, Yi, et al., ("Fastspeech 2: Fast and high-quality end-to-end text to speech." arXiv preprint arXiv:2006.04558 (2020)).
For at least the supra provided reasons, Applicant's arguments have been fully considered but they are not persuasive.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7-12, 14 and 22-24 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Marple further in view of Yoon.
Regarding Claim 1,
Wang discloses a method for creating a script for rendering audio and video streams, comprising:
identifying at least one prosodic speech feature, in at least one of a received stream including at least audio or a received language model (Wang, Fig.2, col14, lls.8-65, "…receiving 200 a synchronized speech and video signal from a database 105 or live input via a video camera 110 or other audio/video input device..."; ), and at least one prosodic gesture in a received stream including at least video (Wang, Fig.2, col.14, lls.8-65, "…The synchronized speech and video signal is then processed to perform 205 motion analysis of a video portion of the synchronized speech and video signal to extract motion components of the synchronized speech and video signal for various body parts...e.g., mouth, nose, eyes, eyebrows, ears, face, head, fingers, hands, arms, legs...", "…The information extracted from the synchronized speech 60 and video signal, including the speech prosody information, speech acoustic features, speech duration characteristics, and body part motions...").
Wang does not explicitly disclose the text stream annotation to create a prosodic script, but Marple, in the analogous field of endeavor, discloses automatically temporally annotating an associated text stream with at least one prosodic speech symbol created from the identified at least one prosodic speech feature and at least one prosodic gesture symbol created from the identified at least prosodic gesture to create a prosodic script (Marple, Fig.13, col.2, lls.35-67, "…The invention also provides a method and system of speech synthesis which includes prosodic codes, or notations, for marking up text with expressive meaning to specify an appropriate prosody… "; Figs.11-12, col.11, lls.12-19, "…the invention provides and employs a graphical symbol set which can be employed to indicate, or provide a template for attractive, prosodic speech output which has one or another quite distinct style..."; Fig.13, col.12, lls.6-21, "…The method also includes a step 72 of electronically marking the text with electronic versions of one or more prosodic graphical symbols..."),
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified the trainable probabilistic animation synthesizer of Wang with prosodic codes/notation for marking up text of Marple with a reasonable expectation of success to communicate desired prosody to a speech synthesizer in a manner that permits control of the prosody of the output speech (Marple, col.2, lls. 35-67).
Neither Wang nor Marple explicitly discloses the rendering synchronized audio and video stream with concatenated features from text, speech, and gesture. However, Yoon, in the analogous field of endeavor, discloses when rendered, provides an audio and video stream comprising the at least one prosodic speech feature and the at least one prosodic gesture that are temporally aligned (Yoon, Fig.1, Introduction, "… an end-to-end gesture generation model that uses the multimodal context of text for speech content, audio for speech rhythm, and speaker identity (ID) for style variations…."; Processing Multimodal Data, "…Our model uses explicitly aligned speech and gesture because speech and gesture are synchronized temporally, allowing the network to concentrate on the translation from input speech to gestures."; Fig.2, 3.3 Gesture Generator, "…Encoded features of speech text, audio, and speaker ID are concatenated to form a concatenated feature vector fi for each time instant...").
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified taught by Wang in view of Marple with the multi-modal speech gesture generation of Yoon with a reasonable expectation of success to output gestures that are humanlike and that match with speech content and rhythm by incorporating a multimodal context and an adversarial training scheme (Yoon, Abstract).
Regarding Claim 2,
Wang in view of Marple further in view of Yoon discloses the method of claim 1, further comprising: converting a received stream including at least audio and/or a language model into a text stream to create the associated text stream (Wang, col.14, lls.47-51, "…the Animation Synthesizer also receives 210 or generates a text signal that corresponds to the speech portion of the synchronized speech and video signal by processing the speech signal using speech-to-text recognition techniques...");and
creating the at least one prosodic speech symbol and/or the at least one prosodic gesture symbol using stored, pre-determined symbols (Marple, Fig.13, col.2, lls.35-67, "…The invention also provides a method and system of speech synthesis which includes prosodic codes, or notations, for marking up text with expressive meaning to specify an appropriate prosody… "; Figs.11-12, col.11, lls.12-19, "…the invention provides and employs a graphical symbol set which can be employed to indicate, or provide a template for attractive, prosodic speech output which has one or another quite distinct style..."; col.16, lls.24-28, "…the pauses are part of the pace of speaking and may be determined by the overall speed and rhythmic variation of the speech..") .
Regarding Claim 3,
Wang in view of Marple further in view of Yoon discloses the method of claim 1, further comprising: rendering the prosodic script to create at least one predicted audio and video stream (Yoon, Fig.1, Introduction, "… an end-to-end gesture generation model that uses the multimodal context of text for speech content, audio for speech rhythm, and speaker identity (ID) for style variations…."; Fig.2, 3.3 Gesture Generator, "…Encoded features of speech text, audio, and speaker ID are concatenated to form a concatenated feature vector fi for each time instant..."); and
comparing prosodic speech features and/or prosodic gestures of the at least one predicted audio and video stream to prosodic speech features of a ground truth stream including at least audio and/or prosodic gestures of a ground truth stream including at least video to determine respective loss functions for training a system to create the prosodic script (Yoon, Fig.1: Losses, 3.4 Adversarial Scheme "… An adversarial scheme is applied in training the model to generate more realistic gestures. The adversarial scheme uses a discriminator…."; 4.2 Training Loss Function, "…The model is trained using the losses below. We use 𝐿𝐺 to train the encoders and gesture generator and 𝐿𝐷 to train the discriminator...").
Regarding Claim 4,
Wang in view of Marple further in view of Yoon discloses the method of claim 1, wherein the prosodic gestures are identified from movement of at least a portion of a body of a speaker of the received stream including at least video (Wang, Fig.2, col.14, lls.8-65, "…The synchronized speech and video signal is then processed to perform 205 motion analysis of a video portion of the synchronized speech and video signal to extract motion components of the synchronized speech and video signal for various body parts...e.g., mouth, nose, eyes, eyebrows, ears, face, head, fingers, hands, arms, legs...", "…The information extracted from the synchronized speech 60 and video signal, including the speech prosody information, speech acoustic features, speech duration characteristics, and body part motions...").
Regarding Claim 5,
Wang in view of Marple further in view of Yoon discloses the method of claim 4, wherein the portion of a body of a speaker comprises a face of the speaker and that at least one prosodic gesture comprises a change in at least a portion of the face of the speaker, including at least one of a head, mouth, forehead, ears, chin, or eyes of the speaker (Wang, Fig.2, col.14, lls.8-65, "…to extract motion components of the synchronized speech and video signal for various body parts...e.g., mouth, nose, eyes, eyebrows, ears, face, head, fingers, hands, arms, legs...").
Regarding Claim 7,
Wang in view of Marple further in view of Yoon discloses the method of claim 1, wherein the at least one prosodic speech feature comprises at least one of an emphasis, a duration, or a pitch of a temporal portion of the received stream including at least audio (Marple, col.2, lls.50-55, "…The prosody to impart can comprises one or more prosody elements selected from the group consisting of pace, intonation pattern, rhythm, musicality, amplitude, pauses for emphasis and breath, and formal and informal articulations of words and phrases...") .
Claim 8 is an apparatus claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale. Additionally,
Wang discloses a processor; and a memory accessible to the processor, the memory having stored therein at least one of programs or instructions executable by the processor to configure the apparatus to (Wang, Fig.3: General Purpose Computer, Processing Units 310, System Memory 320 Storage 360, GPU 315, col.15, ll.54 - col.16, ll.38.)
…
Rationale for combination is similar to that provided for Claim 1.
Claim 9 is an apparatus claim with limitations similar to the limitations of Claim 2 and is rejected under similar rationale.
Claim 10 is an apparatus claim with limitations similar to the limitations of Claim 3 and is rejected under similar rationale.
Claim 11 is an apparatus claim with limitations similar to the limitations of Claim 4 and is rejected under similar rationale.
Claim 12 is an apparatus claim with limitations similar to the limitations of Claim 5 and is rejected under similar rationale.
Claim 14 is an apparatus claim with limitations similar to the limitations of Claim 7 and is rejected under similar rationale.
Claim 22 is a method claim with limitations similar to the limitations of Claims 1, 2, and 4 and is rejected under similar rationale.
Rationale for combination is similar to that provided for Claim 1.
Claim 23 is a method claim with limitations similar to the limitations of Claims 1 and 2 and is rejected under similar rationale.
Claim 24 is a method claim with limitations similar to the limitations of Claim 4 and is rejected under similar rationale.
Claims 6, 13, and 15-21 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Marple further in view of Yoon further in view of Ren.
Regarding Claim 6,
Wang in view of Marple further in view of Yoon discloses the method of claim 1. Wang and Yoon disclose probabilistic (Wang, Fig.1: training module 100, col.5, lls.39-53) and adversarial training models (Yoon, Fig.1: Losses, 3.4 Adversarial Scheme, 4.2 Training Loss Function), respectively but does not explicitly discloses the model training with spectrogram.
However. Ren, in the analogous field of endeavor, discloses creating a spectrogram of the received stream including at least audio; rendering the spectrogram from the prosodic script to create a predicted spectrogram; and comparing the predicted spectrogram to the created spectrogram to determine a loss function for training a system to create the prosodic script (Ren, Fig.1: Mel-spectrogram Decoder, Abstract, "…we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference...").
It would have been obvious to a person of ordinary skill in the art to use the in multi-modal speech gesture generator with a prosodic script taught by Wang, Marple, and Yoon to improve the device with a reasonable expectation that this would result in the text to speech (TTS) model directly generating speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings by Wang in view of Marple further in view of Yoon and Ren to obtain the invention as specified in claim 6 (Ren, Abstract).
Claim 13 is an apparatus claim with limitations similar to the limitations of Claim 6 and is rejected under similar rationale.
Claim 15 is an system claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale. Additionally,
Ren discloses a spectral features module (Ren, Fig.1: Mel-spectrogram Decoder);
Wang discloses a streams to script module ; and an apparatus comprising a processor and a memory accessible to the processor, the memory having stored therein at least one of programs or instructions executable by the processor to configure the apparatus to (Wang, Fig.3: General Purpose Computer, Processing Units 310, System Memory 320 Storage 360, GPU 315, col.15, ll.54 - col.16, ll.38.)
a gesture features module (Wang, Fig.1: Feature Extraction Module 155, Motion Sequence Synthesis Module 160, Animation Generation Module, 175)
…
Rationale for combination is similar to that provided for Claim 6.
Claim 16 is an system claim with limitations similar to the limitations of Claim 2 and is rejected under similar rationale.
Claim 17 is an system claim with limitations similar to the limitations of Claim 3 and is rejected under similar rationale.
Claim 18 is an system claim with limitations similar to the limitations of Claim 4 and is rejected under similar rationale.
Claim 19 is an system claim with limitations similar to the limitations of Claim 5 and is rejected under similar rationale.
Claim 20 is an system claim with limitations similar to the limitations of Claim 6 and is rejected under similar rationale. Rationale for combination is similar to that provided for Claim 6.
Claim 21 is an system claim with limitations similar to the limitations of Claim 7 and is rejected under similar rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Aaron et al., (US Pat No. 8,886,538) discloses systems and methods for speech synthesis and, in particular, text-to-speech systems and methods for converting a text input to a synthetic waveform by processing prosodic and phonetic content of a spoken example of the text input to accurately mimic the input speech style and pronunciation.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JANGWOEN LEE whose telephone number is (703)756-5597. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, BHAVESH MEHTA can be reached at (571)272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JANGWOEN LEE/Examiner, Art Unit 2656
/BHAVESH M MEHTA/Supervisory Patent Examiner, Art Unit 2656
Prosecution Insights