DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Preliminary Remarks
This is a reply to the application filed on 06/20/2025, in which, claims 1-20 remain pending in the present application with claims 1, 12, and 20 being independent claims.
When making claim amendments, the applicant is encouraged to consider the references in their entireties, including those portions that have not been cited by the examiner and their equivalents as they may most broadly and appropriately apply to any particular anticipated claim amendments.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 7-14, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Hadap et al. (US 11581020 B1, hereinafter referred to as “Hadap”) in view of Zhang et al. (CN 115497082 A, hereinafter referred to as “Zhang”).
Regarding claim 1, Hadap discloses an audio and video synchronization detection method, comprising:
extracting first image frames and first audio frames from a video (see Hadap, Column 2, lines 18-26: “The video synthesis system identifies a particular actor's face within a first frame of a particular shot (e.g., a sequence of frames between two cuts in the movie title). The video synthesis system then generates facial parameters (e.g., including facial shape, pose, and/or expression parameters) for a first three-dimensional (3D) model of the particular actor's face. Meanwhile, the video synthesis system also analyzes the dubbed audio and generates facial parameters (e.g., including facial expression parameters), which may be associated, among other regions of the face, with lip movements of the voice that spoke the dubbed audio”);
determining a respective target audio and video synchronization detection algorithm according to the respective target type (see Hadap, Column 11, lines 2-16: “the video synthesis system 101 may receive the video file 102 and at least one dubbed audio 108 that corresponds to the dialogue for one of the actors in a regional language (e.g., Hindi, instead of English). In some embodiments, the subtitles 109 may also be included as input. As described further herein, the video synthesis system 101 may identify, for a particular frame in a given shot (e.g., shot 104) in the video file 102, the particular person (e.g., the particular actor) for which the dubbed audio 108 applicable for. For example, the video synthesis system 101 may identify the face of the particular person in a first frame (e.g., frame A 106), and then may generate first facial parameters for a first three-dimensional model 116 (e.g., a 3DMM) for the face of the particular person in frame A 106 of the shot”); and
performing audio and video synchronization detection on the first image frames and the first audio frames based on the target audio and video synchronization detection algorithms (see Hadap, Column 2, lines 43-47: “The video synthesis system may perform a similar process for other frames of the particular shot (and similarly, for other shots of the movie title), thus automatically generating a synthesized video that synchronizes lips of the particular actor shown in the video to the dubbed audio”).
Regarding claim 1, Hadap discloses all the claimed limitations with the exception of obtaining a respective target type of each first image frame by identifying types of the first image frames.
Zhang from the same or similar fields of endeavor discloses obtaining a respective target type of each first image frame by identifying types of the first image frames (see Zhang, page 3: “A judging module, configured to determine that the text contained in the target text area in the multi-frame image satisfies the subtitle text condition, and the proportion of the number of images containing text in the target text area in the multi-frame image greater than the first threshold, it is determined that subtitles exist in the target video”).
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings as in Zhang with the teachings as in Hadap. The motivation for doing so would ensure the system to have the ability to use the system and method for judging subtitles in a video disclosed in Zhang to determine that the text contained in the target text area in the multi-frame image satisfies the subtitle text condition, and to determine that subtitles exist in the target video if the proportion of the number of images containing text in the target text area in the multi-frame image greater than the first threshold thus obtaining a respective target type of each first image frame by identifying types of the first image frames in order to determine the target type of the first image frame is a first type in response to the existence of the subtitle in a first image frame so that a respective target audio and video synchronization detection can be determined according to the respective target type.
Regarding claim 2, the combination teachings of Hadap and Zhang as discussed above also disclose the method of claim 1, wherein obtaining the respective target type of each first image frame by identifying the types of the first image frames, comprises:
determining whether each first image frame contains a subtitle by performing content recognition on the first image frame (see Zhang, page 4: “the method for judging subtitles in a video in the related art usually first recognizes the text in the video, and if it is determined that the text is recognized in a specified position below the video, it is considered that the video has subtitles”);
in response to the first image frame containing the subtitle, determining that the target type of the first image frame is a first type (see Zhang, page 7: “If it is determined that the text contained in the multi-frame image in the target text area satisfies the second subtitle text condition, and the ratio of the number of images containing text in the multi-frame image in the target text area is greater than the first threshold, then determine the target video Subtitles exist”); and
in response to the first image frame not containing the subtitle, determining that the target type of the first image frame is a second type (see Zhang, page 8: “If it is determined that the text contained in the target text area in the multi-frame image does not meet the second subtitle text condition, and/or the proportion of the number of images containing text in the target text area in the multi-frame image is greater than the first threshold, then Make sure subtitles do not exist in the target video”).
The motivation for combining the references has been discussed in claim 1 above.
Regarding claim 3, the combination teachings of Hadap and Zhang as discussed above also disclose the method of claim 2, wherein determining the respective target audio and video synchronization detection algorithm according to the respective target type, comprises:
in response to the target type being the first type, determining a subtitle-audio synchronization detection algorithm as the target audio and video synchronization detection algorithm (see Hadap, Column 18, lines 49-56: “The audio files 402 may include a dubbed audio 404 and subtitles 406, which may be similar to any of the dubbed audio and/or subtitles described herein (e.g., with respect to FIG. 2 ). It should be understood that techniques described herein, with respect to automatically synchronizing lips (and/or other facial expressions) in a frame to a dubbed audio, may be performed optionally with (or without) subtitles”); and
in response to the target type being the second type, determining a labial-sound synchronization detection algorithm as the target audio and video synchronization detection algorithm (see Hadap, Column 15, lines 38-41: “this method of selectively replacing (e.g., merging) facial parameters may enable automatic lip synchronization with the dubbed audio 204, while still keeping other aspects of the person shown in the video intact”).
The motivation for combining the references has been discussed in claim 1 above.
Regarding claim 7, the combination teachings of Hadap and Zhang as discussed above also disclose the method of claim 3, wherein in a case that the target audio and video synchronization detection algorithm is the labial-sound synchronization detection algorithm, performing the audio and video synchronization detection on the first image frames and the first audio frames based on the target audio and video synchronization detection algorithms, comprises:
obtaining one or more face identifications by performing face detection and tracking on the first image frames, and obtaining a plurality of image lists by dividing the first image frames into groups according to the face identifications (see Hadap, Column 9, lines 27-42: “the first source person may have a particular face expression (e.g., mouth open or closed, smiling, etc.), depending, for example, on the word being spoken at the point in time corresponding to frame A … the video synthesis system 101 may utilize a neural texture that incorporates characteristics of the neighboring frames (e.g., particular pixel color data and or three-dimensional structural data) of the shot to render an update for frame A that replaces the source actors with the target actors. This may be utilized, for example, to accurately render and/or blend a replacement 3D model of the replacement actor into the frame, so as to minimize artifacts and/or jitter in the frame”);
determining respective mouth region pictures corresponding to each image list according to first image frames in each image list (see Hadap, Column 12, lines 53-56: “a first frame may show the face 202 with the mouth open, and thus, the lips being separated from each other. A subsequent frame may show the face 202 with the mouth closed, and thus, the lip being pursed together”); and
performing the audio and video synchronization detection on the respective mouth region pictures corresponding to each image list and the first audio frames (see Hadap, Column 11, lines 33-36: “the dubbed audio 204 may be received in conjunction with any suitable metadata file that may be used to synchronize the dubbed audio 204 with the appropriate video file”).
The motivation for combining the references has been discussed in claim 1 above.
Regarding claim 8, the combination teachings of Hadap and Zhang as discussed above also disclose the method of claim 7, wherein performing the audio and video synchronization detection on the respective mouth region pictures corresponding to each image list and the first audio frames, comprises:
obtaining an audio feature sequence of the first audio frames by extracting audio features of the first audio frames (see Hadap, Column 11, lines 41-43: “the video synthesis system 206 may extract features based in part on the audio signal associated with the dubbed audio 204”);
obtaining a lip motion feature sequence corresponding to the image list by extracting lip motion features from the mouth region pictures corresponding to the image list (see Hadap, Column 12, lines 53-56: “a first frame may show the face 202 with the mouth open, and thus, the lips being separated from each other. A subsequent frame may show the face 202 with the mouth closed, and thus, the lip being pursed together”);
obtaining a labial-sound similarity corresponding to the image list according to the lip motion feature sequence corresponding to the image list and the audio feature sequence (see Hadap, Column 11, lines 56-62: “The video synthesis system 206 may then convert the phonemes to visemes. In some embodiments, a viseme may correspond to any of several speech sounds that look similar (e.g., the same) when lip reading. The video synthesis system 206 may then generate face expression parameters 210 based in part on the determined visemes”); and
performing the audio and video synchronization detection on the video according to the labial-sound similarities corresponding to the image lists (see Hadap, Column 34, lines 6-9: “the facial expressions (e.g., lip movements, etc.) of the target person may also be optionally synchronized to an audio file (e.g., a dubbed audio)”).
The motivation for combining the references has been discussed in claim 1 above.
Regarding claim 9, the combination teachings of Hadap and Zhang as discussed above also disclose the method of claim 8, wherein obtaining the lip motion feature sequence corresponding to the image list by extracting the lip motion features from the mouth area pictures corresponding to the image list, comprises:
obtaining a mouth region picture sequence by ranking the mouth region pictures corresponding to the image list according to timestamps (see Hadap, Column 21, lines 1-9: “As depicted in diagram 501, the third facial parameters for the second (e.g. modified) 3D model may correspond to the mouth of the particular person being open, for example as the mouth may voice a particular new phoneme (e.g., corresponding to the second language of the dubbed audio 511) at a particular point in time associated with the frame 523 of the video shot”); and
obtaining the lip motion feature sequence corresponding to the image list by extracting lip motion features of the mouth region picture sequence (see Hadap, Column 12, lines 53-56: “a first frame may show the face 202 with the mouth open, and thus, the lips being separated from each other. A subsequent frame may show the face 202 with the mouth closed, and thus, the lip being pursed together”).
The motivation for combining the references has been discussed in claim 1 above.
Regarding claim 10, the combination teachings of Hadap and Zhang as discussed above also disclose the method of claim 8, wherein obtaining the audio feature sequence of the first audio frames by extracting the audio features of the first audio frames, comprises:
obtaining a first audio frame sequence by ranking the first audio frames according to the timestamps of the first audio frames (see Hadap, Column 14, lines 33-36: “the dubbed audio 204 may be received in conjunction with any suitable metadata file that may be used to synchronize the dubbed audio 204 with the appropriate video file”); and
obtaining the audio feature sequence by extracting audio features of the first audio frame sequence (see Hadap, Column 14, lines 41-43: “the video synthesis system 206 may extract features based in part on the audio signal associated with the dubbed audio 204”).
The motivation for combining the references has been discussed in claim 1 above.
Regarding claim 11, the combination teachings of Hadap and Zhang as discussed above also disclose the method of claim 9, wherein before obtaining the audio feature sequence of the first audio frames by extracting the audio features of the first audio frames, the method further comprises:
extracting mouth key points from the mouth region pictures in the mouth region picture sequence, and obtaining a motion trajectory of key points by tracking the mouth key points (see Hadap, Column 9, lines 22-39: “utilizing the illustration above, suppose that shot 104 depicts the first source person walking in the foreground while speaking a message to the second source person, who may also be engaged in dialogue with the first source person. In this example, frame A 106 may capture a particular point in time, for example, whereby the first source person has a particular body pose (e.g., using a particular hand gesture) and face pose (e.g., tilting the head forward). Additionally, the first source person may have a particular face expression (e.g., mouth open or closed, smiling, etc.), depending, for example, on the word being spoken at the point in time corresponding to frame A 106. As described further herein, the video synthesis system 101 may utilize a neural texture that incorporates characteristics of the neighboring frames (e.g., particular pixel color data and or three-dimensional structural data) of the shot to render an update for frame A that replaces the source actors with the target actors”); and
in a case that the motion trajectory exhibits an opening and closing change, determining that the mouth is in a lip-moving state (see Hadap, Column 24, line 63 – Column 25, line 1: “the second frame 632 may also show, among other things, the same face of the person as in the first frame 630. In this case, the face shows the mouth being closed, with the lips 610 being pursed together. Note that above the lips 610, a micro-expression 636 corresponding to upper-lip skin folds also appear in the second frame 632”).
The motivation for combining the references has been discussed in claim 1 above.
Claim 12 is rejected for the same reasons as discussed in claim 1 above. In addition, the combination teachings of Hadap and Zhang as discussed above also disclose an electronic device, comprising a processor and a memory (see Hadap, Column 31, lines 4-11: “all of the process 800 (or any other processes described herein, or variations, and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware or combinations thereof”); and
wherein the processor reads an executable program code stored in the memory and runs a program corresponding to the executable program code, to enable the processor (see Hadap, Column 31, lines 11-14: “The code may be stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors”).
Claim 13 is rejected for the same reasons as discussed in claim 2 above.
Claim 14 is rejected for the same reasons as discussed in claim 3 above.
Claim 18 is rejected for the same reasons as discussed in claim 7 above.
Claim 19 is rejected for the same reasons as discussed in claim 8 above.
Claim 20 is rejected for the same reasons as discussed in claim 1 above. In addition, the combination teachings of Hadap and Zhang as discussed above also disclose a non-transitory computer readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to cause a computer (see Hadap, Column 31: lines 11-15: “The code may be stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable storage medium may be non-transitory”).
Claims 4-6 and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Hadap and Zhang as applied to claim 1, and further in view of Li (CN 111091811 A, hereinafter referred to as “Li”).
Regarding claim 4, the combination teachings of Hadap and Zhang as discussed above disclose all the claimed limitations with the exceptions of the method of claim 3, wherein in a case that the target audio and video synchronization detection algorithm is the subtitle-audio synchronization detection algorithm, performing the audio and video synchronization detection on the first image frames and the first audio frames based on the target audio and video synchronization detection algorithms, comprises: determining at least one second image frame with the same subtitle from the first image frames; determining at least one second audio frame synchronized with the at least one second image frame from the first audio frames; obtaining an audio recognition result of the at least one second audio frame; and performing the audio and video synchronization detection on the first image frames and the first audio frames according to the same subtitle and the audio recognition result.
Li from the same or similar fields of endeavor discloses the method of claim 3, wherein in a case that the target audio and video synchronization detection algorithm is the subtitle-audio synchronization detection algorithm, performing the audio and video synchronization detection on the first image frames and the first audio frames based on the target audio and video synchronization detection algorithms, comprises:
determining at least one second image frame with the same subtitle from the first image frames (see Li, page 4: “S13, identifying the frame images with the subtitles in the video file to obtain the time periods of the frame images with the same subtitles in the video file”);
determining at least one second audio frame synchronized with the at least one second image frame from the first audio frames (see Li, page 4: “After the audio file synchronized with the video file is extracted, step S13 is executed to identify the frame images with subtitles in the video file, so as to obtain the time periods of the frame images with the same subtitles in the video file”);
obtaining an audio recognition result of the at least one second audio frame (see Li, page 4: “if the same subtitle appears in the continuous 60-frame images, the time segment in which the voice corresponding to the subtitle appears in the video file can be obtained according to the starting time point corresponding to the 1 st frame image and the ending time point corresponding to the 60 th frame image. Then, the synchronous audio file extracted in step S12 is intercepted according to the start time point and the end time point, and the intercepted audio file segment and the corresponding subtitle can be used as the speech training data of the speech recognition model”); and
performing the audio and video synchronization detection on the first image frames and the first audio frames according to the same subtitle and the audio recognition result (see Li, page 4: “the audio files synchronous with the video files are extracted from the video files, the time periods of the frame images with the same subtitles in the video files are obtained through an image recognition technology, the audio files are intercepted according to the time periods to obtain the voice training data, a large number of voice training materials are obtained at extremely low cost, and the technical problem that the voice materials used for training a voice recognition model in the related technology are expensive is solved”).
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings as in Li with the teachings as in Hadap and Zhang. The motivation for doing so would ensure the system to have the ability to use the system and method disclosed in Li to identify the frame images with the subtitles in the video file to obtain the time periods of the frame images with the same subtitles in the video file; to obtain the time periods of the frame images with the same subtitles in the video file after the audio file synchronized with the video file is extracted; to intercept the synchronous audio file extracted according to the start time point and the end time point wherein the intercepted audio file segment and the corresponding subtitle can be used as the speech training data of the speech recognition model; and to obtain the time periods of the frame images with the same subtitles in the video files through an image recognition technology thus determining at least one second image frame with the same subtitle from the first image frames; determining at least one second audio frame synchronized with the at least one second image frame from the first audio frames; obtaining an audio recognition result of the at least one second audio frame; and performing the audio and video synchronization detection on the first image frames and the first audio frames according to the same subtitle and the audio recognition result in order to perform subtitle-audio synchronization so that the problem of miss detection in the process of audio and video synchronization detection can be avoided.
Regarding claim 5, the combination teachings of Hadap, Zhang, and Li as discussed above also disclose the method of claim 4, wherein determining the at least one second image frame with the same subtitle from the first image frames, comprises:
extracting contents of subtitles of the first image frames, and determining the at least one second image frame with the same subtitle (see Li, page 5: “identify a frame image with a subtitle in the video file to obtain a time period of the frame image with the same subtitle in the video file”); and
obtaining an image frame sequence by ranking the at least one second image frame according to a respective timestamp of each second image frame (see Li, page 4: “Identifying each frame of the video by using an image identification technology, and identifying a starting frame image and a last frame image in continuous frame images with the same caption according to a time sequence; and obtaining the time period of the frame images with the same caption in the video file according to the starting time point corresponding to the starting frame image and the ending time point corresponding to the last frame image”).
The motivation for combining the references has been discussed in claim 4 above.
Regarding claim 6, the combination teachings of Hadap, Zhang, and Li as discussed above also disclose the method of claim 4, wherein determining the at least one second audio frame synchronized with the at least one second image frame from the first audio frames, comprises:
determining a respective timestamp of each second image frame and timestamps of the first audio frames (see Li, page 4: “obtaining the time period of the frame images with the same caption in the video file according to the starting time point corresponding to the starting frame image and the ending time point corresponding to the last frame image”);
determining start and end timestamps of the same subtitle according to the respective timestamp of each second image frame (see Li, page 4: “if the same subtitle appears in the continuous 60-frame images, the time segment in which the voice corresponding to the subtitle appears in the video file can be obtained according to the starting time point corresponding to the 1 st frame image and the ending time point corresponding to the 60 th frame image”); and
determining the at least one second audio frame synchronized with the at least one second image frame from the first audio frames according to the start timestamp, the end timestamp and the timestamps of the first audio frames (see Li, page 4: “the synchronous audio file extracted in step S12 is intercepted according to the start time point and the end time point, and the intercepted audio file segment and the corresponding subtitle can be used as the speech training data of the speech recognition model”).
The motivation for combining the references has been discussed in claim 4 above.
Claim 15 is rejected for the same reasons as discussed in claim 4 above.
Claim 16 is rejected for the same reasons as discussed in claim 5 above.
Claim 17 is rejected for the same reasons as discussed in claim 6 above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIENRU YANG whose telephone number is (571)272-4212. The examiner can normally be reached Monday-Friday 10AM-6PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, THAI TRAN can be reached at 571-272-7382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
NIENRU YANG
Examiner
Art Unit 2484
/NIENRU YANG/Examiner, Art Unit 2484
/THAI Q TRAN/Supervisory Patent Examiner, Art Unit 2484
Prosecution Insights