Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-4, 6, 8, 10-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Tzinis (US 20220310113 A1, hereinafter “Tzinis”).
Regarding claim 1, Tzinis teaches obtaining a video that includes an audio portion; (see [0044]: receive audio waveform associated with video content)
separating the audio portion into speaker audio and non-speaker audio, wherein the speaker audio includes one or more people that are speaking; (see [0027], [0044]-[0046], [0069]: audio separation network 215 and output estimated audio sources 220, separating between speech and non-speech audio, )
determining a respective gain for the speaker audio; (see [0027], [0034], [0062]: by suppressing the audio source, it inherently determines the initial gain of the speaker audio])
separating the non-speaker audio into a plurality of channels, wherein each channel corresponds to a particular non-speaker audio source; (see [0044]-[0046], [0027]: audio separation network 215 may separate non-speech audio to its corresponding channel)
obtaining, with an on-screen classifier model, an indication of whether the particular non-speaker audio source for each channel is depicted in the video, wherein image embeddings for a plurality of video frames of the video and audio embeddings for the plurality of channels are provided as input to the on-screen classifier model; (see [0080]: on-screen classifier 375 identify whether one or more audio sources associated with the one or more audio tracks (e.g. 350a, 350b, 350c and 350d) correspond to objects in the video content.)
determining, with an audio-type classifier model, an auditory object classification for each channel; (see [0062]-[0066]: identify audio, such as human speech, vehicular traffic, and/or other sounds)
determining the respective gain for each channel based on the indication of whether the particular non-speaker audio source for the channel is depicted in the video and the auditory object classification for the channel; (see [0034], [0062], [0072]: separate audio sources, weighted by their estimated on-screen probabilities, based on whether non-speech audio source and auditory object classification is identified, inherently determining the gain for each channel before suppressing/adjusting channel)
modifying the speaker audio and each channel by applying the respective gain; (see [0062]: suppress audio source)
and after the modifying, mixing the modified speaker audio and the modified channels with the audio portion to generate a combined audio. (see [0062], [0072]: once the audio is modified, it can be mixed in to recreate a mixture of soundtracks.)
Regarding claim 2, Tzinis teaches providing a user interface that includes an identification of the speaker audio, the particular audio source for each channel, and options for modifying the respective gain for the speaker audio and the particular audio source for each channel; wherein determining the respective gain for the speaker audio and for each channel is further based on user input that modifies the respective gain of one or more of the speaker audio and one or more particular audio sources for the plurality of channels. (see [0118]: upon reception of responses to the requests, the first computing device can provide the requested output (e.g., using a user interface))
Regarding claim 3, Tzinis teaches identifying a plurality of people that are speaking in the speaker audio, wherein the user interface includes options for modifying the respective gain for each speaker audio source. (see [0069]: a user may indicate selection of one individual, and the computing device may enhance the audio track corresponding to the selected individual, and/or suppress the audio track corresponding to the other individual.)
Regarding claim 4, Tzinis teaches the user interface includes playback of separated audio for each of the speaker audio and the particular audio source for each channel. (see [0069]: a user may indicate selection of one individual, and the computing device may enhance the audio track corresponding to the selected individual, and/or suppress the audio track corresponding to the other individual.)
Regarding claim 6, Tzinis teaches the auditory object classification is one of: an enhancer type or a distractor type; and determining the respective gain for each channel based on the indication of whether the particular non-speaker audio source for the channel is depicted in the video and the auditory object classification for the channel comprises determining the respective gain to each channel such that a volume level of channels associated with the enhancer type is raised and a volume level of channels associated with the distractor type is lowered. (see [0099]: positive and negative labels, based on classification decision volume of sound may be minimized and/or eliminated by using a weight for each source signal)
Regarding claim 8, Tzinis teaches mixing at least a part of the audio portion in with the combined audio. (see [0039]: mixing combined audio)
Regarding claim 10, Tzinis teaches the separating is performed using an audio-separation model wherein the audio-separation model uses the image embeddings as a conditioning input, wherein the conditioning input provides cues to audio-separation model about audio sources present in the video. (see [0047]-[0048]: a visual embedding model… used to condition audio separation network 215, as indicated by a dashed arrow from video embedding network 245)
Regarding claims 11-15, the claimed limitations directly correspond to the method claims 1-5; therefore, is rejected for the significant similar reasons as claims 1-5 as discussed above .
Regarding claims 16-20, the claimed limitations are a system claim directly corresponding the method claims 1-5; therefore, is rejected for the significant similar reasons as claims 1-5 as discussed above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 5, 7, and 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tzinis (US 20220310113 A1, hereinafter “Tzinis”).
Regarding claim 5, Tzinis does not teach user interface includes an option to create event profiles for different types of events that increases gain for a first type of audio and decreases gain for a second type of audio, the method and further comprising: responsive to selection of an event profile, applying the gain for the first type of audio and decreasing gain for the second type of audio for the audio portion in the video.
However, it would have been obvious that the designer/a person of ordinary skill could have chosen such arrangement for an event profile for different types of events for ease and convenience based on the users' needs/preferences and no unexpected result is produced.
Regarding claim 7, Tzinis does not explicitly teach separating the audio portion into the plurality of channels is such that each of the plurality of channels is associated with a respective sound type and one or more of the plurality of channels is obtained by performing deduplication to combine two or more audio sources in the audio portion that are of a same sound type.
However, official notice is taken that it is well-known in the art to perform deduplication, and therefore it would have been obvious to a person of ordinary skill to have performed deduplication since it is just one of the many well-known ways that a designer can use to eliminate redundancies and combine two of the same sound types for audio enhancement.
Regarding claim 9, Tzinis does not explicitly mention mixing at least a part of higher-frequency portions of the audio portion in with the combined audio.
However, it would have been obvious that the designer/a person of ordinary skill could have chosen such arrangement for mixing at least a part of higher frequency portions of the audio portion based on the users' needs/preferences and no unexpected result is produced.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANNABELLE KANG whose telephone number is (571)270-3403. The examiner can normally be reached Monday-Thursday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vivian Chin can be reached at 571-272-7848. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANNABELLE KANG/ Examiner, Art Unit 2695
/PAUL KIM/ Primary Examiner, Art Unit 2695
Prosecution Insights