DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Preliminary Remarks
This is a reply to the application filed on 10/02/2025, in which, claims 1-20 remain pending in the present application with claims 1, 8, and 15 being independent claims.
When making claim amendments, the applicant is encouraged to consider the references in their entireties, including those portions that have not been cited by the examiner and their equivalents as they may most broadly and appropriately apply to any particular anticipated claim amendments.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on December 31, 2025 and March 10, 2026 are in compliance with the provisions of 37 CFR 1.97 and are being considered by the Examiner.
Claim Objections
Claim 14 is objected to for erroneous claim dependency. Apparatus claim 14 is recited to depend on method claim 1. It is believed that claim 14 is a dependent claim of apparatus claim 8 instead. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al. (US 20230118966 A1, hereinafter referred to as “Liu”) in view of Tzinis et al. (US 20220310113 A1, hereinafter referred to as “Tzinis”).
Regarding claim 1, Liu discloses a method to generate synchronized audio for a video (see Liu, paragraph [0045]: “The processor is further configured to select audio data based on the story text. The processor is further configured to output a story including the story text, the selected audio data, and a story video having content corresponding to the story text, wherein the story video includes the plurality of story images”), the method comprising:
receiving a text input describing one or more of a scene, an event, or a mood to be reflected in an audio track (see Liu, paragraph [0014]: “the user input 208 is a text input that includes a sentence. As can readily be appreciated, the user input 208 can also be a single word, a phrase, or multiple sentences”);
generating a latent audio representation via an audio generation model conditioned jointly (see Liu, paragraph [0016]: “The plurality of animated story images 130 may be concatenated to form a story 134, which includes the story text 118 and a story video including content corresponding to the story text 118 generated from the user input”) and text embeddings associated with the text input (see Liu, paragraph [0024]: “a sequence-to-sequence transformer model may be used to generate a new sentence or phrase from a text input”); and
decoding the latent audio representation to produce an audio track temporally aligned with the video and semantically consistent with the text input (see Liu, paragraph [0019]: “A language-text matching model including an image encoder and a text encoder can be used to project images and sentences, respectively, into vectors in a same learned latent space. In such a learned latent space, image vectors and text vectors are positioned closer when the corresponding image and text have similar semantic meaning”).
Regarding claim 1, Liu discloses all the claimed limitations with the exception of receiving the video comprising a sequence of frames; and on video embeddings associated with the sequence of frames.
Tzinis from the same or similar fields of endeavor discloses receiving the video comprising a sequence of frames (see Tzinis, paragraph [0044]: “a computing device can receive an audio waveform and a plurality of video frames associated with video content”); and
on video embeddings associated with the sequence of frames (see Tzinis, paragraph [0059]: “an audio-visual spatio-temporal attention scheme may be utilized by letting neural network 200 attend to local features of the visual embeddings for each separated source. In some embodiments, based on audio embedding 235 and local video embeddings 260 for the plurality of video frames 240, an audio-visual embedding 265 is generated”).
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings as in Tzinis with the teachings as in Liu. The motivation for doing so would ensure the system to have the ability to use the system and method disclosed in Tzinis to receive an audio waveform and plurality of video frames associated with video content and to generate an audio-visual embedding based on audio embedding and local video embeddings for the plurality of video frames thus receiving the video comprising a sequence of frames and generating audio-visual embedding based on video embeddings associated with the sequence of frames in order to produce an audio track that is temporally aligned with the video so that it can be used to temporally align audio track based on the embeddings.
Regarding claim 2, the combination teachings of Liu and Tzinis as discussed above also disclose the method of claim 1, wherein the audio track comprises one or more of instrumental music, ambient sound, or sound effects (see Liu, paragraph [0016]: “the audio track 136 is selected via input received from the user. In the depicted example, the audio track 136 is selected using a sentiment analysis model 138. The sentiment analysis model 138 can be implemented to analyze the content of the story text 118 and to select, or recommend, an appropriate audio track 136. In some implementations, the sentiment analysis model 138 recommends an audio genre”).
The motivation for combining the references has been discussed in claim 1 above.
Regarding claim 3, the combination teachings of Liu and Tzinis as discussed above also disclose the method of claim 1, further comprising: encoding the sequence of frames into video embeddings using a vision encoder (see Tzinis, paragraph [0047]: “The plurality of video frames 240 may be input to a video embedding network 245. Visual features may be extracted from the plurality of video frames”), and encoding the text input into text embeddings using a language encoder (see Liu, paragraph [0022]: “The language-text matching model 314 includes a text encoder 316 and an image encoder 318 for projecting the portion of the story text 308 related to the story image 302 that is to be generated and intermediate generated images, respectively, into vectors in a same learned latent space”).
The motivation for combining the references has been discussed in claim 1 above.
Regarding claim 4, the combination teachings of Liu and Tzinis as discussed above also disclose the method of claim 1, further comprising: concatenating the video embeddings and text embeddings into a multimodal embedding sequence (see Liu, paragraph [0016]: “The plurality of animated story images 130 may be concatenated to form a story 134, which includes the story text 118 and a story video including content corresponding to the story text 118 generated from the user input”), wherein the audio generation model is conditioned on the multimodal embedding sequence (see Liu, paragraph [0016]: “the audio track 136 is generated based on the story text 118 using a generative model”).
The motivation for combining the references has been discussed in claim 1 above.
Regarding claim 5, the combination teachings of Liu and Tzinis as discussed above also disclose the method of claim 1, wherein decoding the latent audio representation comprises applying a variational autoencoder trained to reconstruct audio signals from compressed latent representations (see Tzinis, paragraph [0034]: “given any input video, one or more audio sources that constitute an input mixture can be separated, and probability scores indicative of audio-visual correspondence for each separated source can be estimated. A higher probability score is indicative of separated sources which are apparent on-screen, and a lower probability score is indicative of separated sources which are not apparent on-screen. The separated audio sources, weighted by their estimated on-screen probabilities, can be summed together to reconstruct the on-screen mixture”).
The motivation for combining the references has been discussed in claim 1 above.
Regarding claim 6, the combination teachings of Liu and Tzinis as discussed above also disclose the method of claim 1, wherein the audio context comprises at least one of: audio infilling, audio extension, or audio replacement for one or more frames of the video (see Tzinis, paragraph [0066]: “the computing device can identify a portion of an image in the video content. For example, the video may capture a person playing a musical instrument in a crowded marketplace. Accordingly, the computing device can identify a portion of the video that corresponds to the person playing the musical instrument. Then, the computing device can determine that a particular audio source of the one or more estimated audio sources corresponds to a particular object in the identified portion of the video content”).
The motivation for combining the references has been discussed in claim 1 above.
Regarding claim 7, the combination teachings of Liu and Tzinis as discussed above also disclose the method of claim 1, wherein the semantic consistency is based on the correspondence between the generated audio and the text input using a contrastive audio-video-text pre-training model (see Liu, paragraph [0019]: “An example framework includes the use of a contrastive language-image pretraining model to provide guidance to the image generation process through user input”).
The motivation for combining the references has been discussed in claim 1 above.
Claim 8 is rejected for the same reasons as discussed in claim 1 above. In addition, the combination teachings of Liu and Tzinis as discussed above also disclose the apparatus comprising:
one or more processors (see Liu, paragraph [0033]: “The logic machine 502 may include one or more processors configured to execute software instructions”); and
one or more memories coupled with the one or more processors and storing processor-executable code (see Liu, paragraph [0034]: “Storage machine 504 includes one or more physical devices configured to hold instructions executable by the logic machine”).
Claim 9 is rejected for the same reasons as discussed in claim 2 above.
Claim 10 is rejected for the same reasons as discussed in claim 3 above.
Claim 11 is rejected for the same reasons as discussed in claim 4 above.
Claim 12 is rejected for the same reasons as discussed in claim 5 above.
Claim 13 is rejected for the same reasons as discussed in claim 6 above.
Claim 14 is rejected for the same reasons as discussed in claim 7 above.
Claim 15 is rejected for the same reasons as discussed in claim 1 above. In addition, the combination teachings of Liu and Tzinis as discussed above also disclose a non-transitory computer-readable medium having program code recorded thereon to generate synchronized audio for a video, the program code executed by one or more processors (see Liu, paragraph [0033]: “Processors of the logic machine 502 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing”).
Claim 16 is rejected for the same reasons as discussed in claim 2 above.
Claim 17 is rejected for the same reasons as discussed in claim 3 above.
Claim 18 is rejected for the same reasons as discussed in claim 4 above.
Claim 19 is rejected for the same reasons as discussed in claim 5 above.
Claim 20 is rejected for the same reasons as discussed in claim 6 above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIENRU YANG whose telephone number is (571)272-4212. The examiner can normally be reached Monday-Friday 10AM-6PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, THAI TRAN can be reached at 571-272-7382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
NIENRU YANG
Examiner
Art Unit 2484
/NIENRU YANG/Examiner, Art Unit 2484
/THAI Q TRAN/Supervisory Patent Examiner, Art Unit 2484
