DETAILED ACTION
This examination is in response to the communication filed on 06/16/2026. Claims 1-18 are currently pending, wherein new claims 14-18 have been added and claims 1 and 12-14 have been amended.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment/Arguments
Applicant’s amendments and arguments filed 06/16/2026, with respect to claim 1-18 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Objections
Applicant is advised that should claims 5, 6 or 7 be found allowable, claims 16, 17 and 18, respectively, will be objected to under 37 CFR 1.75 as being a substantial duplicate thereof. The Examiner notes that it appears that new claims 16-18 should have been dependent on new claim 14 rather than claim 1. When two claims in an application are duplicates or else are so close in content that they both cover the same thing, despite a slight difference in wording, it is proper after allowing one claim to object to the other as being a substantial duplicate of the allowed claim. See MPEP § 608.01(m).
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1 and 4-13 are rejected under 35 U.S.C. 103 as being unpatentable over Chronopoulou et al. “Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing” AWS AI Labs, Center for Information and Language Processing arXiv:2302.12979v1 [cs.CL] 25 Feb 2023 (herein “Chronopoulou) in view of Marchuk et al. (US 2022/0377407 A1; herein “Marchuk”), further in view of Fayan et al. (US 2003/0164845 A1; herein “Fayan”).
Regarding claim 1, Chronopoulou teaches a method of editing media, the method comprising:
receiving a media composition that includes video and audio, wherein the audio includes dialog in a first language spoken by a first voice (Page 1, section 1 teaches “Automatic Dubbing (AD) is a specialization of speech-to-speech translation in which speech from a video is translated to a new language and the new speech is overlayed on the original video” and Section 2 teaches “a source speech utterance is passed through an ASR system”, the source speech utterance is interpreted as a received media composition that includes dialog, e.g., speech utterances);
converting the spoken dialog in the first language to first language text using automatic speech recognition (Page 1, Section 2 teaches “a source speech utterance is passed through an ASR system, which provides a transcript in the source language” );
using machine-learning-based text-to-text translation to translate the first language text into second language text (Page 1, Section 2 teaches “The transcript (in a pure textual form) is translated to the target language using an MT model…” );
using text-to-speech synthesis on the second language text to generate a spoken dialog in the second language (Page 2, first column teaches “…the output is passed to a PA module and finally to a TTS synthesizer. The TTS model adjusts the speaking rate of each phrase to make the speech segment approximately fit the timing of the corresponding source phrase”); and
linking the spoken dialog in the second language to the media composition, such that the spoken dialog in the second language is in temporal synchrony with video of the media composition (Page 2, first column teaches “…the output is passed to a PA module and finally to a TTS synthesizer. The TTS model adjusts the speaking rate of each phrase to make the speech segment approximately fit the timing of the corresponding source phrase” adjusting the speech segment to fit the timing of the corresponding source phrase is interpreted as temporal synchronization of the TTS output with the original video).
Chronopoulou fails to explicitly disclose generating a language proxy and enabling an editor to edit the audio and video of the media composition by editing the language proxy.
Marchuk teaches method and system for synchronizing video to audio recorded in a distributed system that includes, inter alia, generating a language proxy by linking the spoken dialog in the second language to the media composition, such that the spoken dialog in the second language is in temporal synchrony with video of the media composition (¶[0019] teaches “The distributed network recording system… provides true synchronization between the audio recorded by the actor and the frames of a portion of the film content being dubbed…by sending a compressed, time-stamped proxy audio file of the original lossless recording to each user device participating in the recording session…The proxy audio file can be reviewed, edited, and manipulated by the participants in the recording session and final time synchronized edit information can be saved and associated with the original, lossless audio file to script the final audio edit for the dubbed film content” and ¶[0039] teaches “The proxy file creation application 336 may further create a proxy file of each dub activity”); and
enabling an editor to edit the media composition by editing the language proxy (¶[0019] teaches “…The proxy audio file can be reviewed, edited, and manipulated by the participants in the recording session and final time synchronized edit information can be saved and associated with the original, lossless audio file to script the final audio edit for the dubbed film content”).
Chronopoulou differs from the claimed invention, as defined in claim 1, in that Chronopoulou fails to disclose utilizing a language proxy for synchronizing the dubbed audio with the original media composition. Using a language or audio proxy for synchronizing dubbed audio in an original video is known in the art as evidenced by Marchuk. Therefore, it would have been obvious to one having ordinary skill in the art to have modified the automatic dubbing system of Chronopoulou to include utilizing a audio/language proxy for storing the synchronized dubbing audio as taught by Marchuk in order to allow an audio engineer to adjust and edit various aspects of an audio dub in the final recording. (Marchuk ¶[0026]).
Although the combination of Chronopoulou and Marchuk teaches the proxy audio file can be edited and then final time synchronized edit information can be saved and associated with the original audio, the combination of Chronopoulou and Marchuk fails to explicitly disclose editing the audio and video of the media composition by editing the language proxy.
Fayan teaches a retiming/synchronization process in which an editing system utilizing a retiming effect/process to retime and synchronize playback of both the audio and video data in a media clip (Fayan, ¶[004]). Fayan, paragraph [0005] teaches that “to allow synchronized playback, a retiming function that defines the rampable retiming effect is used to generate new audio and video samples at appropriate output times.” More specifically, ¶[0037] teaches “both the video and the audio are retimed to produce a new output pacing.” Under a broadest reasonable interpretation “enabling an editor to edit the audio and video of the media composition…” includes retiming the audio and video to keep them in synch. This interpretation is support by the ¶[0024] of the present application which states “…the video may be retimed to keep the video in sync with the translated dialog…”. Thus, Fayan teaches “editing the audio and video of the media composition” as claimed.
The combination of Chronopoulou and Marchuk differs from the claimed invention, as defined in claim 1, in that the combination fails to disclose editing, e.g., retiming, the video of the media composition using the language proxy. Retiming audio and/or video of a media composition to allow synchronized playback is known in the art as evidenced by Fayan. Therefore, it would have been obvious to one having ordinary skill in the art to have modified the automatic dubbing system taught by the combination of Chronopoulou and Marchuk to include utilizing retiming of video and audio as taught by Fayan in order to allow synchronized playback. (Fayan ¶[0005]).
Regarding claim 4, the combination of Chronopoulou, Marchuk and Fayan teaches all of the elements of claim 1 (see detailed element mapping above). In addition, Marchuk further teaches the spoken dialog in the second language is stored as a file and the language proxy retrieves audio from the file (Fig. 3, Audio storage server 338 and Proxy Creation 336; and corresponding description in ¶[0039]).
Chronopoulou differs from the claimed invention, as defined in claim 4, in that Chronopoulou fails to disclose utilizing a language proxy for synchronizing the dubbed audio with the original media composition. Using a language or audio proxy for synchronizing dubbed audio in an original video is known in the art as evidenced by Marchuk. Therefore, it would have been obvious to one having ordinary skill in the art to have modified the automatic dubbing system of Chronopoulou to include utilizing a audio/language proxy for storing the synchronized dubbing audio as taught by Marchuk in order to allow an audio engineer to adjust and edit various aspects of an audio dub in the final recording. (Marchuk ¶[0026])
Regarding claims 5 and 16, the combination of Chronopoulou, Marchuk and Fayan teaches all of the elements of claim 1 (see detailed element mapping above). In addition, Marchuk further teaches the spoken dialog in the second language is generated in real-time when required by an editor who is editing the media composition with dialog in the second language (¶[0005] teaches “Recording controls, playback/record status, audio channel configuration, volume, audio timeline, script edits, and other functions are synchronized across participants and may be controlled for all participants remotely by a designated user, typically a sound engineer, so that each participant sees and hears the section of the program being recorded and edited at the same time.” and ¶[0026] teaches “Editing task may be performed during the recording session…” during the recording session is interpreted as in real-time).
Chronopoulou differs from the claimed invention, as defined in claims 5 and 16, in that Chronopoulou fails to disclose utilizing a language proxy for synchronizing the dubbed audio with the original media composition in real-time. Using a language or audio proxy for synchronizing dubbed audio in an original video in real-time is known in the art as evidenced by Marchuk. Therefore, it would have been obvious to one having ordinary skill in the art to have modified the automatic dubbing system of Chronopoulou to include utilizing a audio/language proxy for generating the synchronized dubbing audio in real-time as taught by Marchuk in order to allow an audio engineer to adjust and edit various aspects of an audio dub in the final recording. (Marchuk ¶[0026])
Regarding claim 6 and 17, the combination of Chronopoulou, Marchuk and Fayan teaches all of the elements of claim 1 (see detailed element mapping above). In addition, Marchuk further teaches compositing over the video of the media composition a text translation in the second language of the spoken dialog in the first language (¶[0025] teaches “The relevant portion of the script that the actor is reading for dubbing may be presented in a script window 206… If the actor is dubbing a scene in a different language, e.g., for localization, the script may be presented in a foreign language with respect to the original language of the film.” ).
Chronopoulou differs from the claimed invention, as defined in claims 6 and 17, in that Chronopoulou fails to disclose an editing interface which overlays a text translation in the second language of the spoken dialog in the first language. Editing interfaces which overlay a text translation in a second language is known in the art as evidenced by Marchuk. Therefore, it would have been obvious to one having ordinary skill in the art to have modified the automatic dubbing system of Chronopoulou to include utilizing an editing interface as taught by Marchuk in order to allow an audio engineer to adjust and edit various aspects of an audio dub in the final recording. (Marchuk ¶[0026])
Regarding claims 7 and 18, the combination of Chronopoulou, Marchuk and Fayan teaches all of the elements of claim 1 (see detailed element mapping above). In addition, Chronopoulou further teaches compositing over the video of the media composition a text of the spoken dialog in the first language (¶[0025] teaches “The relevant portion of the script that the actor is reading for dubbing may be presented in a script window 206. If the actor is overdubbing their own original take, the script may be a portion of the original script…”).
Chronopoulou differs from the claimed invention, as defined in claims 7 and 18, in that Chronopoulou fails to disclose an editing interface which overlays a text of the spoken dialog in the first language. Editing interfaces which overlay a text/script in the original language is known in the art as evidenced by Marchuk. Therefore, it would have been obvious to one having ordinary skill in the art to have modified the automatic dubbing system of Chronopoulou to include utilizing an editing interface as taught by Marchuk in order to allow an audio engineer to adjust and edit various aspects of an audio dub in the final recording. (Marchuk ¶[0026])
Regarding claim 8, the combination of Chronopoulou, Marchuk and Fayan teaches all of the elements of claim 1 (see detailed element mapping above). In addition, Chronopoulou further teaches the spoken dialog in the second language is temporally synchronized by matching a plurality of sentence durations in spoken dialog in the second language to corresponding sentence durations in the dialog spoken in the first language (Page 1, Section 2 teaches “…The transcript (in a pure textual form) is translated to the target language using an MT model, then a prosodic alignment (PA) [1,6] segments the translated text into phrases and pauses that follow the phrase-pause arrangement of the original speech [10,6]…The TTS model adjusts the speaking rate of each phrase to make the speech segment approximately fit the timing of the corresponding source phrase” ).
Regarding claim 9, the combination of Chronopoulou, Marchuk and Fayan teaches all of the elements of claim 1 (see detailed element mapping above). In addition, Chronopoulou further teaches the media composition includes video that is synchronized with the spoken dialog in the first language (Page 1, Section 1 teaches “Automatic Dubbing (AD) is a specialization of speech-to-speech translation in which speech from a video is translated to a new language and the new speech is overlayed on the original video”).
Regarding claim 10, the combination of Chronopoulou, Marchuk and Fayan teaches all of the elements of claim 9 (see detailed element mapping above). In addition, Chronopoulou further teaches the spoken dialog in the second language is time-stretched to improve synchronization of the spoken dialog in the second language with the video (Page 1, Section 1 teaches “…modifying generated speech, via stretching/compressing phoneme durations [6] during synthesis with text-to-speech (TTS) and adding pauses [7, 8, 6], to make its timing more closely match that of the source speech, often referred to as prosodic alignment (PA).” ).
Regarding claim 11, the combination of Chronopoulou, Marchuk and Fayan teaches all of the elements of claim 9 (see detailed element mapping above). In addition, Chronopoulou further teaches the video is retimed to synchronize with the spoken dialog in the second language and the media composition is finished with spoken dialog in the second language (Page 1, Section 1 teaches “Automatic Dubbing (AD) is a specialization of speech-to-speech translation in which speech from a video is translated to a new language and the new speech is overlayed on the original video …modifying generated speech, via stretching/compressing phoneme durations [6] during synthesis with text-to-speech (TTS) and adding pauses [7, 8, 6], to make its timing more closely match that of the source speech, often referred to as prosodic alignment (PA).”).
Independent claims 12 and 13 recite computer program product comprising a non-transitory computer-readable memory, and system comprising a memory and processor of editing media which is configured to perform the method of claim 1. Therefore, the claims are rejected for the same reasoning as claim 1 above.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Chronopoulou, Marchuk and Fayan as applied to claim 1 above, and further in view of Di Gangi et al. (US 2025/0118336 A1; herein “Di Gangi”).
Regarding claim 2, the combination of Chronopoulou, Marchuk and Fayan teaches all of the elements of claim 1 (see detailed element mapping above). The combination of Chronopoulou, Marchuk and Fayan fails to disclose that the TTS uses voice cloning to generate the dubbed dialog/speech.
Di Gangi teaches an automatic video dubbing method and apparatus that automatically dubs a video signal into a different language than the original while preserving the vocal characteristics of original speakers in the corresponding audio. More specifically, Di Gangi teaches the text-to-speech synthesis on the second language text uses machine-learning-based voice cloning to generate dialog in the second language spoken by a voice that resembles the first voice (¶[0013] teaches “The TTs components need to pronounce the words correctly and the voice should sound somehow like the original. In some cases, a cloning of the original voice is required…TTS with the speaker voice can be achieved in at least two ways. The first approach…is to use speaker-adaptive TTS, which takes as an additional input a feature vector representing the speaker voice’s characteristics. The system learns to pronounce the text with a voice that reproduces the encoded one…”).
The combination of Chronopoulou, Marchuk and Fayan differs from the claimed invention, as defined in claim 2, in that the combination fails to disclose the TTS process include voice cloning. Automatic video dubbing which utilizes a voice cloning TTS process is known in the art as evidenced by Di Gangi. Therefore, it would have been obvious to one having ordinary skill in the art to have modified the automatic dubbing system taught by the combination of Chronopoulou, Marchuk and Fayan to include utilizing a voice cloning TTS process as taught by Marchuk in order to allow dubbing of a video signal into different target languages while controllably preserving vocal characteristics of original speakers in the corresponding audio. (Di Gangi, Abstract)
Allowable Subject Matter
Claim 3 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 3, although the combination of Chronopoulou, Marchuk and Fayan teaches all of the elements of claim 1 and relinking a version of the media composition that has been editing using the language proxy to source media of the media composition, wherein the source media includes the spoken dialog in the first language, the combination fails to disclose or suggest generating an edited version of the media composition with dialog spoken in the first language as claimed.
Claims 14 and 15 are allowed.
The following is an examiner’s statement of reasons for allowance:
Regarding new independent claim 14, the prior art of record fails to disclose or suggest relinking a version of the media composition that has been edited using the language proxy to source media of the media composition, wherein the source media includes the spoken dialog in the first language and generating an edited version of the media composition with dialog spoken in the first language in combination with the other limitations of the claim.
Regarding new claim 15, this claim depends from independent claim 14 and therefore are allowable over the prior art of record for at least those reasons presented above with respect to claim 14.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PENNY L CAUDLE whose telephone number is (703)756-1432. The examiner can normally be reached M-Th 8:00 am to 5:00 pm eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at 571-272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PENNY L CAUDLE/Examiner, Art Unit 2657
/HAI PHAN/Supervisory Patent Examiner, Art Unit 2654
Prosecution Insights