DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Preliminary Amendment
2. The preliminary amendment to claims filed on 4/10/2025 has been accepted and is examined below.
Claim Rejections - 35 USC § 103
3. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
4. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
5. Claims 1, 8, 11-12, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Adams et al. (US Patent No. 10,770,092 B1) in view of Serletic, II et al. (US Patent Application Publication No. 2019/0147838 A1).
6. Regarding Claim 1 (Currently Amended), Adams discloses A method for image processing, (Abstract reciting “Systems and methods for viseme data generation are disclosed. …”) comprising: generating, when playing audio data, a facial patch corresponding to an audio frame, (Abstract reciting “Systems and methods for viseme data generation are disclosed. Uncompressed audio data is generated and/or utilized to determine the beats per minute of the audio data. Visemes are associated with the audio data utilizing a Viterbi algorithm and the beats per minute. A time-stamped list of viseme data is generated that associates the visemes with the portions of the audio data that they correspond to. An animatronic toy and/or an animation is caused to lip sync using the viseme data while audio corresponding to the audio data is output.”;
col. 10, lines 50-65 reciting “In examples, synchronization of audio output and presentation of visemes may be performed. For example, it may be desirable to present the visemes along with output of the audio with the viseme presentation timed to match or substantially match the human sound portions of the audio to which the visemes correspond. Instructions may be sent from the user device 102 to the one or more accessory devices 104 to cause a movable mouth of the one or more accessory devices 104 to operate in coordination with words in the song, as the words are output via the speaker(s) of the user device 102. Additionally, or alternatively, the instructions may cause presentation of mouth-related animations on a display of the accessory device 104 in coordination with the words in the song, as the words are output via the speaker(s) of the user device 102.” The audio output corresponds to audio data that is played back. Each timestamp is an audio frame. Each viseme associated with the time stamp is a facial patch because the viseme is used to animate the mouth.) wherein the facial patch is used to represent a mouth shape, (col. 8, lines 35-45 reciting “The memory 126 of the remote system 114 may also include a viseme database 150. The viseme database 150 may include a list of visemes and/or instructions that, when executed by a processor, cause a device having and/or depicting a mouth to move to a mouth position corresponding to the visemes. A viseme, as described herein, is used to describe the position of a mouth when making a certain sound. For example, when a person is not speaking, a viseme associated with this state may be a closed or at rest mouth position. …” Each viseme is a facial patch and each viseme is used to represent the mouth shape when pronouncing a word.) and the mouth shape corresponds to audio content of the audio frame; (col. 2, lines 50-52 reciting “One or more visemes may then be associated with the portions of the audio data that correspond at least in part to lyrics.” The lyrics correspond to the audio content at each audio frame (timestamp of the visemes) and the mouth shape moves according to the visemes at each timestamp.) While not explicitly disclosed by Adams, Serletic discloses and displaying the facial patch in a first facial area of an image, wherein the first facial area is used to present a change of a mouth shape as the audio data is played. (paragraph [0079] reciting “FIGS. 7A and 7B show two screenshots of an embodiment of an animated character 702. In such an embodiment, the animated character's mouth 704 may move in an animated fashion substantially in time with the lyrics of the multimedia message. In some embodiments, other features of the animated character may move instead of or in addition to the character's mouth; for example, the eyes, ears, eyebrows, or the entire head, etc., may also be animated to move based on the musical and text inputs. In some embodiments, the animated character may not include an obvious representation of a mouth or lips, but such features may be added to the media asset automatically or as selected by a user. For example, a user may select an image of a car as the animated character input. The user may additionally select a location on the car for the system to add an animated representation of a mouth or lips so as to allow the car to talk or sing along with the musical input and lyrics.”;
paragraph [0071] reciting “… The mouth or lips may be tracked and may, in some embodiments, be overlaid with animated lips that may move along with speech, music, or lyrics as described herein.” Therefore, the user can select any image or object that represents a person and overlay the mouth on a first facial area. The mouth shape will change based on the teachings of Adams where the viseme which corresponds to facial patches changes the
positioning of the mouth/lips.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Adams with Serletic so that the moving mouth/lips can be overlaid onto any graphical object such as a graphical face. This is an obviously beneficial modification as it allows the generation of a virtual animation to lip sync to be any graphical image such as a head or other objects.
7. Regarding Claim 8 (Currently Amended), Adams further discloses The method of claim 1, wherein before generating the facial patch corresponding to the audio frame, the method further comprises: (col. 2, lines 6-8 reciting “In instances where the viseme data has not been generated, a process of generating the viseme data may be initiated.”) obtaining beat information of the audio data, (col. 2, lines 15-16 reciting “The audio data may then be analyzed to determine a number of beats per minute (BPM) of the song.”) wherein the beat information represents a time interval between at least two audio segments during playing the audio data, wherein an audio segment comprises a plurality of audio frames, (col. 2, lines 16-17 reciting “ For example, the desired song may have a BPM of approximately 120 BPM.” Therefore, each 2 beats each second so each half second corresponds to an audio segment with a beat in-between the half second audio segment. Each half second has plurality of frames because an audio frame I just a discrete timestamp and each half second can be divided into various microseconds timestamps etc.)
and the audio segment is used to implement a complete utteration of at least one piece of the audio content; (col. 2, lines 26-28 reciting “The remaining portions of the audio data may be analyzed using a Viterbi algorithm to determine which components of the audio data correspond at least in part to lyrics.” The audio segment is used to be a part of the completely lyrics, and the lyrics represent a completely utteration of the audio content.)
and determining the audio frame according to the beat information. (Abstract reciting “Visemes are associated with the audio data utilizing a Viterbi algorithm and the beats per minute.” Each viseme is associated with an an audio frame because each viseme has a timestamp. Therefore, since the viseme is determined by beats per minute, the BPM is used to determine the audio frame.)
8. Regarding Claim 11 (Currently Amended), Adams discloses An electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer executable instructions; the processor executes the computer-executable instructions stored in the memory to implement a method for image processing, comprising: (col. 4, lines 17-19 reciting “The user device 102, the one or more accessory devices 104, and/or a remote system 114 may include one or more processors.”;
col. 4, lines 42-49 reciting “The user device 102, the one or more accessory devices 104, and/or the remote system 114 may also include memory 122, 124, and/or 126. Memory 122, 124, and/or 126 may include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program component, or other data.”)
generating, when playing audio data, a facial patch corresponding to an audio frame, (Abstract reciting “Systems and methods for viseme data generation are disclosed. Uncompressed audio data is generated and/or utilized to determine the beats per minute of the audio data. Visemes are associated with the audio data utilizing a Viterbi algorithm and the beats per minute. A time-stamped list of viseme data is generated that associates the visemes with the portions of the audio data that they correspond to. An animatronic toy and/or an animation is caused to lip sync using the viseme data while audio corresponding to the audio data is output.”;
col. 10, lines 50-65 reciting “In examples, synchronization of audio output and presentation of visemes may be performed. For example, it may be desirable to present the visemes along with output of the audio with the viseme presentation timed to match or substantially match the human sound portions of the audio to which the visemes correspond. Instructions may be sent from the user device 102 to the one or more accessory devices 104 to cause a movable mouth of the one or more accessory devices 104 to operate in coordination with words in the song, as the words are output via the speaker(s) of the user device 102. Additionally, or alternatively, the instructions may cause presentation of mouth-related animations on a display of the accessory device 104 in coordination with the words in the song, as the words are output via the speaker(s) of the user device 102.” The audio output corresponds to audio data that is played back. Each timestamp is an audio frame. Each viseme associated with the time stamp is a facial patch because the viseme is used to animate the mouth.) wherein the facial patch is used to represent a mouth shape, (col. 8, lines 35-45 reciting “The memory 126 of the remote system 114 may also include a viseme database 150. The viseme database 150 may include a list of visemes and/or instructions that, when executed by a processor, cause a device having and/or depicting a mouth to move to a mouth position corresponding to the visemes. A viseme, as described herein, is used to describe the position of a mouth when making a certain sound. For example, when a person is not speaking, a viseme associated with this state may be a closed or at rest mouth position. …” Each viseme is a facial patch and each viseme is used to represent the mouth shape when pronouncing a word.) and the mouth shape corresponds to audio content of the audio frame; (col. 2, lines 50-52 reciting “One or more visemes may then be associated with the portions of the audio data that correspond at least in part to lyrics.” The lyrics correspond to the audio content at each audio frame (timestamp of the visemes) and the mouth shape moves according to the visemes at each timestamp.) While not explicitly disclosed by Adams, Serletic discloses and displaying the facial patch in a first facial area of an image, wherein the first facial area is used to present a change of a mouth shape as the audio data is played. (paragraph [0079] reciting “FIGS. 7A and 7B show two screenshots of an embodiment of an animated character 702. In such an embodiment, the animated character's mouth 704 may move in an animated fashion substantially in time with the lyrics of the multimedia message. In some embodiments, other features of the animated character may move instead of or in addition to the character's mouth; for example, the eyes, ears, eyebrows, or the entire head, etc., may also be animated to move based on the musical and text inputs. In some embodiments, the animated character may not include an obvious representation of a mouth or lips, but such features may be added to the media asset automatically or as selected by a user. For example, a user may select an image of a car as the animated character input. The user may additionally select a location on the car for the system to add an animated representation of a mouth or lips so as to allow the car to talk or sing along with the musical input and lyrics.”;
paragraph [0071] reciting “… The mouth or lips may be tracked and may, in some embodiments, be overlaid with animated lips that may move along with speech, music, or lyrics as described herein.” Therefore, the user can select any image or object that represents a person and overlay the mouth on a first facial area. The mouth shape will change based on the teachings of Adams where the viseme which corresponds to facial patches changes the
positioning of the mouth/lips.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Adams with Serletic so that the moving mouth/lips can be overlaid onto any graphical object such as a graphical face. This is an obviously beneficial modification as it allows the generation of a virtual animation to lip sync to be any graphical image such as a head or other objects.
9. Regarding Claim 12 (Currently Amended), Adams discloses A non-transitory computer-readable storage medium storing computer-executable instructions, wherein when the computer-executable instructions, executed by a processor, implement a method for image processing, comprising: (col. 38, lines 7-16 reciting “
Computer instructions for operating each device 102 and 104 and its various components may be executed by the respective device's controller(s)/processor(s)
1012 and 1104, using the memory 1014 and 1106 as temporary “working” storage at runtime. A device's computer instructions may be stored in a non-transitory manner in non-volatile memory 1014 and 1106, storage 1018 and 1108, or an external device(s). Alternatively, some or all of the executable instructions may be embedded in hardware or firmware on the respective device 102 and 104 in addition to or instead of software.”)
generating, when playing audio data, a facial patch corresponding to an audio frame, (Abstract reciting “Systems and methods for viseme data generation are disclosed. Uncompressed audio data is generated and/or utilized to determine the beats per minute of the audio data. Visemes are associated with the audio data utilizing a Viterbi algorithm and the beats per minute. A time-stamped list of viseme data is generated that associates the visemes with the portions of the audio data that they correspond to. An animatronic toy and/or an animation is caused to lip sync using the viseme data while audio corresponding to the audio data is output.”;
col. 10, lines 50-65 reciting “In examples, synchronization of audio output and presentation of visemes may be performed. For example, it may be desirable to present the visemes along with output of the audio with the viseme presentation timed to match or substantially match the human sound portions of the audio to which the visemes correspond. Instructions may be sent from the user device 102 to the one or more accessory devices 104 to cause a movable mouth of the one or more accessory devices 104 to operate in coordination with words in the song, as the words are output via the speaker(s) of the user device 102. Additionally, or alternatively, the instructions may cause presentation of mouth-related animations on a display of the accessory device 104 in coordination with the words in the song, as the words are output via the speaker(s) of the user device 102.” The audio output corresponds to audio data that is played back. Each timestamp is an audio frame. Each viseme associated with the time stamp is a facial patch because the viseme is used to animate the mouth.) wherein the facial patch is used to represent a mouth shape, (col. 8, lines 35-45 reciting “The memory 126 of the remote system 114 may also include a viseme database 150. The viseme database 150 may include a list of visemes and/or instructions that, when executed by a processor, cause a device having and/or depicting a mouth to move to a mouth position corresponding to the visemes. A viseme, as described herein, is used to describe the position of a mouth when making a certain sound. For example, when a person is not speaking, a viseme associated with this state may be a closed or at rest mouth position. …” Each viseme is a facial patch and each viseme is used to represent the mouth shape when pronouncing a word.) and the mouth shape corresponds to audio content of the audio frame; (col. 2, lines 50-52 reciting “One or more visemes may then be associated with the portions of the audio data that correspond at least in part to lyrics.” The lyrics correspond to the audio content at each audio frame (timestamp of the visemes) and the mouth shape moves according to the visemes at each timestamp.) While not explicitly disclosed by Adams, Serletic discloses and displaying the facial patch in a first facial area of an image, wherein the first facial area is used to present a change of a mouth shape as the audio data is played. (paragraph [0079] reciting “FIGS. 7A and 7B show two screenshots of an embodiment of an animated character 702. In such an embodiment, the animated character's mouth 704 may move in an animated fashion substantially in time with the lyrics of the multimedia message. In some embodiments, other features of the animated character may move instead of or in addition to the character's mouth; for example, the eyes, ears, eyebrows, or the entire head, etc., may also be animated to move based on the musical and text inputs. In some embodiments, the animated character may not include an obvious representation of a mouth or lips, but such features may be added to the media asset automatically or as selected by a user. For example, a user may select an image of a car as the animated character input. The user may additionally select a location on the car for the system to add an animated representation of a mouth or lips so as to allow the car to talk or sing along with the musical input and lyrics.”;
paragraph [0071] reciting “… The mouth or lips may be tracked and may, in some embodiments, be overlaid with animated lips that may move along with speech, music, or lyrics as described herein.” Therefore, the user can select any image or object that represents a person and overlay the mouth on a first facial area. The mouth shape will change based on the teachings of Adams where the viseme which corresponds to facial patches changes the
positioning of the mouth/lips.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Adams with Serletic so that the moving mouth/lips can be overlaid onto any graphical object such as a graphical face. This is an obviously beneficial modification as it allows the generation of a virtual animation to lip sync to be any graphical image such as a head or other objects.
10. Regarding Claim 21 (New), Adams further discloses The electronic device of claim 11, wherein before generating the facial patch corresponding to the audio frame, the method further comprises: (col. 2, lines 6-8 reciting “In instances where the viseme data has not been generated, a process of generating the viseme data may be initiated.”) obtaining beat information of the audio data, (col. 2, lines 15-16 reciting “The audio data may then be analyzed to determine a number of beats per minute (BPM) of the song.”) wherein the beat information represents a time interval between at least two audio segments during playing the audio data, wherein an audio segment comprises a plurality of audio frames, (col. 2, lines 16-17 reciting “ For example, the desired song may have a BPM of approximately 120 BPM.” Therefore, each 2 beats each second so each half second corresponds to an audio segment with a beat in-between the half second audio segment. Each half second has plurality of frames because an audio frame I just a discrete timestamp and each half second can be divided into various microseconds timestamps etc.)
and the audio segment is used to implement a complete utteration of at least one piece of the audio content; (col. 2, lines 26-28 reciting “The remaining portions of the audio data may be analyzed using a Viterbi algorithm to determine which components of the audio data correspond at least in part to lyrics.” The audio segment is used to be a part of the completely lyrics, and the lyrics represent a completely utteration of the audio content.)
and determining the audio frame according to the beat information. (Abstract reciting “Visemes are associated with the audio data utilizing a Viterbi algorithm and the beats per minute.” Each viseme is associated with an an audio frame because each viseme has a timestamp. Therefore, since the viseme is determined by beats per minute, the BPM is used to determine the audio frame.)
Allowable Subject Matter
11. Claims 2-7, 9, 15-20, and 22-23 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including ALL of the limitations of the base claim and any intervening claims.
12. The following is a statement of reasons for the indication of allowable subject matter: Claim 2 recites the limitation wherein the audio frame comprises a first audio frame and a second audio frame, and the first audio frame and the second audio frame are alternately played, displaying the facial patch in the first facial area of the image comprising: if the audio frame is the first audio frame, displaying the image and the facial patch located in the first facial area based on a first magnification coefficient; if the audio frame is the second audio frame, displaying the image and the facial patch located in the first facial area based on a second magnification coefficient; and the first magnification coefficient being different from the second magnification coefficient which is neither disclosed nor suggested by the cited references, either singly or in combination. Claims 15 and 23 are similar to claim 2 and is allowed over cited references as well.
13. The closest art on record is Adams. While Adams discloses generating mouth movement based on visemes based on the lyrics of the audio, nothing in Adams discloses the above limitation of claims 2, 15, and 23.
14. Claim 3 recites the limitation wherein generating the facial patch corresponding to the audio frame comprises: obtaining data for a first mouth shape corresponding to the audio frame, wherein the data for the first mouth shape is used to represent the mouth shape; detecting the image to obtain data for a second mouth shape, wherein the data for the second mouth shape represents a size parameter of the mouth shape; and generating the facial patch based on a first mouth shape parameter and a second mouth shape parameter which is neither disclosed nor suggested by the cited references, either singly or in combination. Claim 16 is similar to claim 3 and is allowed over cited references as well.
15. The closest art on record is Adams. While Adams discloses generating mouth movement based on visemes based on the lyrics of the audio, nothing in Adams discloses the above limitation of claims 3 and 16.
16. Claims 4-7 depend on claim 3. Claims 17-20 are similar to claims 4-7 and claims 17-20 depend on claim 16.
17. Claim 9 recites the limitation wherein the determining the target audio frame according to the beat information comprises: obtaining a number of beats per minute of the audio data according to the beat information, wherein the number of beats per minute represents the number of beats of a melody corresponding to the audio data in each minute; and determining the target audio frame based on a timestamp of an audio frame in the audio data and the number of beats per minute of the audio data which is neither disclosed nor suggested by the cited references, either singly or in combination. Claim 22 is similar to claim 9 and is allowed over cited references as well.
18. The closest art on record is Adams. While Adams discloses generating mouth movement based on visemes based on the lyrics of the audio, nothing in Adams discloses the above limitation of claims 9 and 22.
CONTACT
Any inquiry concerning this communication or earlier communications from the examiner should be directed to FRANK S CHEN whose telephone number is (571)270-7993. The examiner can normally be reached Mon - Fri 8-11:30 and 1:30-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at 5712727794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/FRANK S CHEN/Primary Examiner, Art Unit 2611
