Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claims 1–11 and 13–21 have been submitted for examination.
Claims 1–4 and 13–17 have been examined and rejected.
Claims 5–11 and 18–21 are objected to.
Allowable Subject Matter
Claims 5–11 and 18–21 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1–11 and 13–21 are rejected under 35 U.S.C. 103 as being unpatentable over Shaburov et al. (US 11,089,238) in view of Yi et al. (US 2020/0195882).
Regarding claims 1, 13, and 14, Shaburov discloses:
A video determination method applied to a client, the method comprising:
acquiring, in response to an effect trigger operation, (Shaburov, cols. 12–13, lns. 65–10, “The user interface 700 shows a selfie capturing mode in which a user may take an image of the user face, which is then used as a source face. The user interface 700 shows a live view of a camera of a computing device when the user intends to capture the selfie image. The live view may show the user face. The user interface 700 may show a selfie oval 730 and a camera button 740.”) a target facial image (“an image of a person”) comprising a target object; (“an image of the face of the person”) (Shaburov, col. 8, ln. 35–45, “The face images capturing module 310 can receive an image of a person and generate an image of the face of the person. The image of the face of the person can be used as a source face to replace target face in the videos stored in the video database 320. The image of the person can be captured by the camera 205 of the computing device 105. The image of the person can include an image stored in the memory storage 215 of the computing device 105. Details for the face images capturing module 310 are provided in FIG. 7.”)
determining a target audio, (“generated in the form of an audiovisual media”) and determining a key video frame sequence (“e.g., a video, an animation, or any other type of media”) corresponding to the target audio; (Shaburov, col. 12, ln. 53–64, “the personalized video may be generated in the form of an audiovisual media (e.g., a video, an animation, or any other type of media) that features a face of a user or faces of several users.”)
determining a target effect audio and video (“Synthesis of a face”) (Shaburov, col. 11, ln. 20–40, “Synthesis of a face using the parametric face model 505 and the texture model 510 is not time-consuming; however, the synthesized face may not be photorealistic, especially in the mouth and eyes regions. In some embodiments of the disclosure, the DNN 515 can be trained to generate photorealistic images of the mouth and eye regions of a face. The DNN 515 can be trained using a collection of videos of talking individuals. The mouth and eyes regions of talking individuals can be captured from frames of the videos. The DNN 515 can be trained using a generative adversarial network (GAN) to predict the mouth and eyes regions of the face based on a pre-determined number of previous frames of the mouth and eyes regions and desired facial expression of a current frame.”) based on the target facial feature (“facial expression may include open mouth, smile, anger, astonishment, and so forth”) and the target audio. (“generated in the form of an audiovisual media”) (Shaburov, col. 10, ln. 50–60, “In some embodiments of the disclosure, the parametric face model 505 can be pre-generated based on images of a pre-defined number of individuals of different age, gender, and ethnic background. For each individual, the images may include an image of the individual having a neutral facial expression and one or more images of the individual having different facial expressions. The facial expression may include open mouth, smile, anger, astonishment, and so forth.”)
Shaburov does not explicitly teach “determining, based on the key video frame sequence and the target facial image, a target facial feature in the target facial image that is presented when the target audio is played; and”.
In a similar field of endeavor Yi teaches:
determining, based on the key video frame sequence and the target facial image, (Yi, ¶ [0041], “In accordance with an embodiment, the audio content associated with each of the first object 308 and the second object 310 may be an audio or sound produced by the people (for example an actor) associated with the first object 308 and the second object 310 during the capture of the image frame by the video capturing device.”) a target facial feature (“a facial expression”) in the target facial image (Yi, ¶ [0036], “The captured plurality of images may be further utilized to determine a facial expression of the user 108 based on an effect of the displayed multi-media content and the CC information on the user 108. The facial expression may indicate one or more motions or positions of muscles of a face of the user 108, where the facial expressions may manifest an emotion.”) that is presented when the target audio is played; (Yi, ¶ [0041], “Examples of the audio content associated with the first object 308 and the second object 310 may include, but are not limited to, a dialogue, a sound effect, musical cues, audible tone, or other audio information. In some embodiments, the audio content may not be the sound or audio produced by the people during the capture of the image frame. In contrast, the audio content may be associated with a thought corresponding to the people which are related to the first object 308 and/or the second object 310.”) and
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the system for generating audiovisual content from facial capture as taught by Shaburov with the system for synchronizing sound effects as taught by Yi, the motivation is “The sequence of image frames 312 may be captured by a video capturing device (not shown) to create the multi-media content.” as taught by Yi (¶ [0040]).
Regarding claims 2 and 15, the combination of Shaburov and Yi teaches:
The method according to claim 1, wherein the effect trigger operation comprises at least one of the following: triggering an effect prop; a frame-in picture (“selfie oval 730 ”) comprising the target object; (Shaburov, cols. 12–13, lns. 65–10, “The user interface 700 shows a selfie capturing mode in which a user may take an image of the user face, which is then used as a source face. The user interface 700 shows a live view of a camera of a computing device when the user intends to capture the selfie image. The live view may show the user face. The user interface 700 may show a selfie oval 730 and a camera button 740.”) triggering an effect wake-up word using audio information; and a current body movement being consistent with a preset effect movement.
Regarding claims 3 and 16, the combination of Shaburov and Yi teaches:
The method according to claim 1, wherein the determining a target audio comprises: displaying at least one audio to be selected, and determining the target audio based on a trigger operation on the at least one audio to be selected within first preset duration; or receiving an uploaded audio to be processed as the target audio. (Yi, ¶ [0041], “Examples of the audio content associated with the first object 308 and the second object 310 may include, but are not limited to, a dialogue, a sound effect, musical cues, audible tone, or other audio information. In some embodiments, the audio content may not be the sound or audio produced by the people during the capture of the image frame. In contrast, the audio content may be associated with a thought corresponding to the people which are related to the first object 308 and/or the second object 310.”)
Regarding claims 4 and 17, the combination of Shaburov and Yi teaches:
The method according to claim 1, further comprising: determining the key video frame sequence corresponding to the target audio based on a pre-selected target language type (Yi, ¶ [0063], “the processor 204 may be configured to identify a current geo-location (for example using inbuilt GPS component) of the AV reproduction apparatus 102. The processor 204 may be further configured to identify the language in context with the identified geo-location.”) corresponding to the target audio. (Yi, ¶ [0063], “In some embodiments, the third voice command 508C may be configured to change the language based on a current location of the AV reproduction apparatus 102.”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL B PIERORAZIO whose telephone number is (571)270-3679. The examiner can normally be reached on Monday - Thursday, 8am - 5pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Nasser Goodarzi can be reached on 5712704195. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL B. PIERORAZIO/Primary Examiner, Art Unit 2426