Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The abstract of the disclosure is objected to because “content,.” in line 2 appears to be a typo and should apparently be “content.” A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The disclosure is objected to because of the following informalities:
“200of” in paragraph [0032] appears to be a typo and should apparently be “200 of”
There needs to be a period at the end of paragraphs [0035] and [0041] instead of an interpunct.
“device530” in paragraph [0062] appears to be a typo and should apparently be “device 530”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5, 12, and 19 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 5, 12, and 19 recite the limitation “the audio token” in line 6. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-4, 8-11, 15-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nair et al., US Pub. 11514948 B1 (hereinafter Nair) in view of Cohen-Or et al., US Pub. 2025/0140257 A1 (hereinafter Cohen-Or).
Regarding Claim 1, the combined teaching of Nair and Cohen-Or discloses a method for processing video content, comprising: generating a set of audio tokens (Cohen-Or: [0212], where the transformer model may include a TransformerEncoder which encodes input audio into "tokens"; [0306], where the target audio data object can be first decomposed to time-stamped audio tokens) corresponding to first audio content of first video content (Nair: Fig. 7, element 710 and Col. 3, lines 45-49, where source video 110 may have an extracted audio portion spoken in a source language 112), the first audio content corresponding to first text content of a first language (Nair: Fig. 3 and Col. 8, lines 34-36, where the resulting cepstrum (or cepstral) may be used for voice generation and manipulation matching the same features of the voice captured in the source audio); generating, based on an audio feature representation corresponding to the set of audio tokens, second audio content corresponding to second text content, the second text content being generated by translating the first text content into a second language (Nair: Col. 8, lines 56-58, where audio extraction and conversion 222 may implement language converter 340, in various embodiments, to produce translated audio text 306 for spoken portions; Fig. 4 and Col. 9, lines 24-31, where generator network 410 may then create replacement audio using the voice features 404 and identified emotion 406 to produce a voice that mimics (in some scenarios indistinguishably mimics) the voice of the original speaker as if the original speaker spoke the translated audio text 402. The candidate replacement audio of generator network 410 may then be submitted to discriminator network 420); generating a second set of video frames based on a set of visual features corresponding to a first set of video frames of the first video content and the audio feature representation (Nair: Col. 8, lines 5-7, where frames, segments, or other or other portions of the source video that correspond to different respective speech may be identified; Col. 10, lines 6-15, where image modification for replacement audio 226 may implement face image data extract 510 to obtain the portions of image data from source video 506 (e.g., of individual frames in the source video for the portion of the source video being modified), in some embodiments. For example, face image data extraction 510 may identify the pixel boundaries of lips (of a speaking person) in a video frame. Generator network 520 may then create modifications to the extracted facial image data based on new audio portions); and generating second video content based on the second set of video frames and the second audio content (Nair: Fig. 7, element 740 and Col. 14, lines 13-18, where the translated audio portions and modified facial image data may be incorporated back into the source vice, updating the source video to be a translated version of the source video that includes the new audio portion(s) and the modified facial movement).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Nair in view of Cohen-Or such that the method for processing video content would be comprised of generating audio tokens in order to generate second audio content corresponding to a second text content. Nair’s cepstrum analysis produces an audio feature representation corresponding to the spoken audio content (the first text content) of the first language, and combining that with Cohen-Or’s token generation allows for an accurate representation of audio content that is suitable to use for translation/dubbing into a second language. Both Nair and Cohen-Or disclose techniques to improve lip dubbing, so combining elements of their teachings would ensure that the highest quality audio-visual synchronization occurs.
Regarding Claim 2, the combined teaching of Nair and Cohen-Or discloses the method of claim 1, wherein generating the second set of video frames based on the set of visual features corresponding to the first set of video frames of the first video content and the audio feature representation comprises: determining, for a first video frame of the first set of video frames and based on time information of the first video frame, a feature segment corresponding to the first video frame from the audio feature representation (Nair: Col. 8, lines 4-12, where time boundary recognition 314 may be implemented, in some embodiments. Frames, segments, or other or other portions of the source video that correspond to different respective speech may be identified, as discussed below with regard to FIG. 7. The amount of time that the speakers face is visible for speaking the dialogue may also be determined. In this way, the amount of time that replacement audio can occupy is determined from the time the speakers dialogue is visible); and generating, based on a first visual feature of the first video frame and the feature segment, a second video frame corresponding to the first video frame (Nair: Col. 10, lines 6-15, where image modification for replacement audio 226 may implement face image data extract 510 to obtain the portions of image data from source video 506 (e.g., of individual frames in the source video for the portion of the source video being modified), in some embodiments. For example, face image data extraction 510 may identify the pixel boundaries of lips (of a speaking person) in a video frame. Generator network 520 may then create modifications to the extracted facial image data based on new audio portions 502).
Regarding Claim 3, the combined teaching of Nair and Cohen-Or discloses the method of claim 1, wherein the second audio content is generated using an audio converter in a target model, the second set of video frames are generated using a video converter in the target model (Nair: Col. 6, lines 8-11, where translation processing may implement audio extraction and conversion 222 for received source videos in order to perform replacement audio generation 224, and image modification for replaced audio 226), and the audio converter and the video converter are co-trained (Nair: Col. 14, lines 39-43, where language-specific models for translation, new audio generation, and/or facial image data (e.g., from English to Spanish, from French to Chinese, etc.) may be retrained or modified (or replaced) once a threshold number of feedback has been collected for that specific language).
Regarding Claim 4, the combined teaching of Nair and Cohen-Or discloses the method of claim 1, wherein the second video content has a mouth shape change corresponding to the second audio content (Cohen-Or: [0038], where the processor is configured to receive an initial video data object V comprising a plurality of image frame data objects; receive a set of time-synchronized viseme parameters corresponding to target audio data object A'; and process the initial video data object V and the set of time-synchronized viseme parameters using a machine learning network to generate an output video data object V', wherein initial mouth regions in the initial video data object V have been replaced with replacement mouth regions generated based on the set of time-synchronized viseme parameters.).
Regarding Claim 8, the combined teaching of Nair and Cohen-Or discloses an electronic device, comprising: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions, when executed by the at least one processor (Nair: Fig. 10 and Col. 15, lines 4-14, where computer system 1000 includes one or more processors 1010 (any of which may include multiple cores, which may be single or multi-threaded) coupled to a system memory 1020 via an input/output (I/O) interface 1030. Computer system 1000 further includes a network interface 1040 coupled to I/O interface 1030. In various embodiments, computer system 1000 may be a uniprocessor system including one processor 1010, or a multiprocessor system including several processors 1010 (e.g., two, four, eight, or another suitable number). Processors 1010 may be any suitable processors capable of executing instructions), causing the electronic device to perform act comprising all of the claimed limitations (see rejection of Claim 1 for details).
Regarding Claim 9, the combined teaching of Nair and Cohen-Or discloses the electronic device of claim 8, wherein generating the second set of video frames based on the set of visual features corresponding to the first set of video frames of the first video content and the audio feature representation comprises all of the claimed limitations (see rejection of Claim 2 for details).
Regarding Claim 10, the combined teaching of Nair and Cohen-Or discloses the electronic device of claim 8, comprising all of the claimed limitations (see rejection of Claim 3 for details).
Regarding Claim 11, the combined teaching of Nair and Cohen-Or discloses the electronic device of claim 8, comprising all of the claimed limitations (see rejection of Claim 4 for details).
Regarding Claim 15, the combined teaching of Nair and Cohen-Or discloses a non-transitory computer-readable storage medium storing a computer program thereon, the computer program being executable by a processor (Nair: Col. 18, lines 10-16, where any or all of program instructions 1025 may be provided as a computer program product, or software, that may include a non-transitory computer-readable storage medium having stored thereon instructions, which may be used to program a computer system (or other electronic devices) to perform a process according to various embodiments) to perform acts comprising all of the claimed limitations (see rejection of Claim 1 for details).
Regarding Claim 16, the combined teaching of Nair and Cohen-Or discloses the non-transitory computer-readable storage medium of claim 15, wherein generating the second set of video frames based on the set of visual features corresponding to the first set of video frames of the first video content and the audio feature representation comprises all of the claimed limitations (see rejection of Claim 2 for details).
Regarding Claim 17, the combined teaching of Nair and Cohen-Or discloses the non-transitory computer-readable storage medium of claim 15, comprising all of the claimed limitations (see rejection of Claim 3 for details).
Regarding Claim 18, the combined teaching of Nair and Cohen-Or discloses the non-transitory computer-readable storage medium of claim 15, comprising all of the claimed limitations (see rejection of Claim 4 for details).
Claim(s) 5, 12, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nair in view of Cohen-Or, and further in view of Yang et al., “UniAudio: An Audio Foundation Model Toward Universal Audio Generation” (hereinafter Yang).
Regarding Claim 5, the combined teaching of Nair and Cohen-Or discloses the method of claim 1, wherein generating the set of audio tokens corresponding to the first audio content of the first video content comprises: extracting the first audio content from the first video content (Nair: Col. 7, lines 50-51, where received source video 302 may be provided to spoken audio extraction 310); and generating, using an audio tokenizer, a plurality of audio tokens corresponding to a plurality of segments of the first audio content (Cohen-Or: [0212], where the TransformerEncoder encodes input audio into “tokens”; [0306], where the target audio data object can be first decomposed to time-stamped audio tokens; [0029], where the voice-to-lips machine learning model architecture is configured for encoding an input audio set associated with an individual as audio tokens). However, it fails to explicitly disclose that the audio token is a universal audio token (UAT). Yang from a similar endeavor teaches tokenizing all audio types into a common representation (Yang: [Abstract], “the UniAudio system, which, unlike prior task-specific approaches, leverages LLM techniques to generate multiple types of audio (including speech, sounds, music, and singing)” and “UniAudio first tokenizes all types of target audio along with other condition modalities”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Nair in view of Cohen-Or, and further in view of Yang such that the audio tokens used to generate second audio content corresponding to a second text content would be universal audio tokens. This simplifies the processing of audio streams since it avoids the separate processing of speech and non-speech audio components.
Regarding Claim 12, the combined teaching of Nair and Cohen-Or discloses the electronic device of claim 8, and the combined teaching of Nair, Cohen-Or, and Yang comprises all of the claimed limitations (see rejection of Claim 5 for details).
Regarding Claim 19, the combined teaching of Nair and Cohen-Or discloses the non-transitory computer-readable storage medium of claim 15, and the combined teaching of Nair, Cohen-Or, and Yang comprises all of the claimed limitations (see rejection of Claim 5 for details).
Claim(s) 6-7, 13-14, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nair in view of Cohen-Or, and further in view of Santhar et al., US Pub. 12118323 B2 (hereinafter Santhar).
Regarding Claim 6, the combined teaching of Nair and Cohen-Or discloses the method of claim 1, wherein generating, based on the audio feature representation corresponding to the set of audio tokens, the second audio content comprises: generating, using an audio encoder, an audio feature representation corresponding to the set of audio tokens (Cohen-Or: [0212], where the transformer model may include a TransformerEncoder which encodes input audio into "tokens", along with a TransformerDecoder which attends to the audio tokens and previous lip landmarks to synthesize lip landmark sequences; [0306], where the target audio data object can be first decomposed to time-stamped audio tokens, which are mapped to phonemes and then corresponding visemes. Effectively, each time-stamped audio token can represent a mouth shape or a mouth movement that corresponds to the target audio data object). However, it fails to explicitly disclose processing, using an audio decoder, the audio feature representation to generate the second audio content. Santhar from a similar endeavor teaches processing, using an audio decoder, the audio feature representation to generate the second audio content (Santhar: Fig. 3 and Col. 5, lines 62-66, where the encoding vector can be fed into a decoder made of multiple nodes (e.g., recurrent neural networks, transformers, etc....). The output vector can be fed into a fully connected neural network trained to predict the most probable word or words based on the output vector).
It would have been obvious to one of ordinary skill in the art to modify Nair in view of Cohen-Or, and further in view of Santhar such that the generation of the second audio content requires processing the audio feature representation using an audio decoder. The decoder processes the audio feature representation to produce the translated text, and Santhar’s text-to-speech module 208 then converts the translated text to speech, generating the second audio content.
Regarding Claim 7, the combined teaching of Nair, Cohen-Or, and Santhar discloses the method of claim 6, wherein the audio decoder is configured to perform at least one of the following tasks: a first task, configured to translate the first text content into the second text content (Santhar: Col. 5, lines 40-42, where neural machine translation module 206 is a computer program that can translate text is a source language into text in a target language); a second task, configured to align a first duration of the first audio content and a second duration of the second audio content (Nair: Col. 8, lines 8-12, where the amount of time that the speakers face is visible for speaking the dialogue may also be determined. In this way, the amount of time that replacement audio can occupy is determined from the time the speakers dialogue is visible).
Regarding Claim 13, the combined teaching of Nair and Cohen-Or discloses the electronic device of claim 8, and the combined teaching of Nair, Cohen-Or, and Santhar comprises all of the claimed limitations (see rejection of Claim 6 for more details).
Regarding Claim 14, the combined teaching of Nair, Cohen-Or, and Santhar discloses the electronic device of claim 13, comprising all of the claimed limitations (see rejection of Claim 7 for details).
Regarding Claim 20, the combined teaching of Nair and Cohen-Or discloses the non-transitory computer-readable storage medium of claim 15, and the combined teaching of Nair, Cohen-Or, and Santhar comprises all of the claimed limitations (see rejection of Claim 6 for details).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Gupta et al., US Pub. 11551664 B2 teaches a method of translating audio into another language, and also teaches improved dubbing techniques for manipulating faces and mouths
Jones et al., GB 2601162 A teaches a video translation system that also manipulates the video for improved dubbing
Levine et al., US Pub. 2024/0087557 A1 teaches a method of translating and dubbing audio
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HANNA CHONG whose telephone number is (571)270-0520. The examiner can normally be reached Monday - Friday, 8 a.m. - 5 p.m. ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Nathan Flynn can be reached at (571) 272-1915. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HANNA CHONG/Examiner, Art Unit 2421
/NATHAN J FLYNN/Supervisory Patent Examiner, Art Unit 2421
Prosecution Insights