CTNF 18/889,390 CTNF 95839 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-21-aia AIA Claim s 1-6, 8-9, 13-18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao (US 20220345796 A1) in further view of Wang (US 20100082345 A1) . With respect to claims 1 and 13 Zhao teaches (claim 1) A model training method for driving a virtual human to speak, comprising: (claim 13) A computing device comprising: a memory storing executable instructions (0018] FIG. 2 illustrates a block diagram of an exemplary computer system 200 configured to implement various aspects of the synthetic video generation technique disclosed herein. Computer system 200 may be configured as a server, a workstation, or a terminal device such as a mobile device. As shown in FIG. 2, computer system 200 may include a processor 210, a communication interface 220, a memory/storage 230, and a data bus 240. Memory/storage 230 may be configured to store computer-readable instructions that, when executed by processor 210, can cause processor 210 to perform various operations disclosed herein. Memory 230 may include any non-transitory type of storage devices,) ; and a processor configured to execute the executable instructions to (0018] FIG. 2 illustrates a block diagram of an exemplary computer system 200 configured to implement various aspects of the synthetic video generation technique disclosed herein. Computer system 200 may be configured as a server, a workstation, or a terminal device such as a mobile device. As shown in FIG. 2, computer system 200 may include a processor 210, a communication interface 220, a memory/storage 230, and a data bus 240. Memory/storage 230 may be configured to store computer-readable instructions that, when executed by processor 210, can cause processor 210 to perform various operations disclosed herein. Memory 230 may include any non-transitory type of storage devices,) : generating an initial virtual human speaking video based on an audio data set and a person speaking video, wherein duration of the initial virtual human speaking video is greater than duration of the person speaking video (Zhao ¶ [0029] Referring back to FIG. 3, method 300 proceeds to step 340, in which processor 210 may generate a synthetic motion picture of the human face based on the reference video and the audio. The synthetic motion picture of the human face may include a motion of the mouth of the human face presenting the speech of the audio., ¶ [0028] In another example, processor 210 may extend the reference video by playing the reference video forward then backward then forward again, etc., until the total time duration matches that of the audio. Again, the time point at which the forward/backward playing switches can be any time point of the reference video... It is noted that step 330 is performed when there is a mismatch between the time durations of the reference video and the audio. If the reference video has the same length in duration as the audio, step 330 may be skipped.) ; wherein definition of the initial virtual human speaking video is lower than definition of the target virtual human speaking video (Zhao ¶ [0026] In step 310, processor 210 may receive a reference video including a motion picture of a human face. The reference video may be pre-recorded and used as a template for generating a plurality of synthetic videos. For example, the reference video may include a person, such as a realtor, acting naturally while making a normal speech or conversation. The purpose of the reference video is to capture the facial expression, eye blinking, gentle body/face movement, or other similar features that naturally occur during speaking. It is not necessary for the person to speak [lower definition] [lower definition] during the recording of the reference video because the motion of the mouth will be replaced [higher definition] [higher definition] according to specific audio contents during the generation of the synthetic video. ) Zhao does not explicitly disclose however Wang teaches generating a lip synchronization parameter generation model by using the initial virtual human speaking video (Wang ¶ [0023] The Animation Synthesizer then uses the resulting probabilistic model [generating lip synchronization parameter] for selecting , or more specifically, for "predicting," the appropriate animation trajectories for one or more different body parts (e.g., mouth (i.e., lip sync and other mouth motions), nose, eyes, eyebrows, ears, face, head, fingers, hands, arms, legs, feet, torso, spine, skeletal elements of a body, etc.) based on an arbitrary text and/or speech input based on an evaluation of the context, punctuation, and any emotional characteristics associated with that input. ) and using the lip synchronization parameter generation model to obtain a target virtual human speaking video (Wang ¶ [0023] The Animation Synthesizer then uses the resulting probabilistic model [generating lip synchronization parameter] for selecting , or more specifically, for "predicting," the appropriate animation trajectories for one or more different body parts (e.g., mouth (i.e., lip sync and other mouth motions), nose, eyes, eyebrows, ears, face, head, fingers, hands, arms, legs, feet, torso, spine, skeletal elements of a body, etc.) based on an arbitrary text and/or speech input based on an evaluation of the context, punctuation, and any emotional characteristics associated with that input. ) It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify video generation of Zhao to include the parameter generation model of Wang in order to generate videos with arbitrary text/audio inputs ([0002], Wang); With respect to claims 2 and 14 Wang further teaches wherein the step of generating the lip synchronization parameter generation model by using the initial virtual human speaking video comprises: generating the lip synchronization parameter generation model by using the initial virtual human speaking video and a three-dimensional face reconstruction model (Wang ¶ [0072] In general, animation is a sequential changing of 2-D or 3-D model parameters over time, which can be achieved using a variety of techniques, such as, for example, shape/morph target based techniques, bone/cage techniques, skeleton-muscle based systems, motion capture of points on face corresponding to a deformation models associated with a mesh model of the avatar, knowledge based solver deformation based techniques, etc. The Animation Synthesizer is capable of using any such techniques, so long as a mapping or correlation of the avatar mesh model or robot controls is provided to the motions modeled by the HMM or other probabilistic model is provided so that proper deformations of the parameterized 2-D or 3-D model (or robot) will be correctly synchronized with an arbitrary speech input.) It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify video generation of Zhao to include the parameter generation model of Wang in order to generate videos with arbitrary text/audio inputs ([0002], Wang); With respect to claim 3 and 15 Wang further teaches wherein the step of generating the lip synchronization parameter generation model by using the initial virtual human speaking video and a three-dimensional face reconstruction model comprises: extracting a lip synchronization training parameter from the initial virtual human speaking video by using the three-dimensional face reconstruction model (Wang ¶ [0022] Training of the probabilistic models generally involves extracting motions (facial and/or body) from one or more inputs comprising a synchronized speech and motion source, then learning a probabilistic model that best explains those motions in view the corresponding speech. In general, one or more audio/video inputs are used to model different body parts to construct one or more trainable probabilistic models by using conventional motion capture and object recognition techniques to both identify and probabilistically model the motions of specific body parts relative to a corresponding synchronized speech input. Examples of a synchronized speech and motion source include a live audio/video feed of a person talking and moving or making facial expressions, or a pre-recorded audio/video source, such as, for example, a newscast, a movie, or other audio/video recording. Note that the duration of particular sentences, phrases, words, phonemes, etc. can also be modeled in order to account for differences in speaking speeds (e.g., a fast speaker or a slow speaker).) ; and obtaining the lip synchronization parameter generation model through training by using the lip synchronization training parameter as a label and using the audio data set as model input data (Wang ¶ [0098] The information extracted from the synchronized speech and video signal, including the speech prosody information, speech acoustic features, speech duration characteristics, and body part motions as a function of synchronized speech are then used to learn or train 225 the probabilistic animation model with context labels for each learned or probabilistic animation unit. As discussed above, motion training is performed at various levels (e.g., sentences (or multiple sentences), phrases, words, phonemes, sub-phonemes, etc.) for different body parts (e.g., mouth, nose, eyes, eyebrows, ears, face, head, fingers, hands, arms, legs, etc.). Once trained, the Animation Synthesizer stores the probabilistic model or models 230 to a computer-readable medium for later use. However, new models 230 can learned at any time by processing additional synchronized speech and video signals, and existing models can be refined or further trained by either adjusting model parameters or providing additional inputs of synchronized speech and video signals.) It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify video generation of Zhao to include the parameter generation model of Wang in order to generate videos with arbitrary text/audio inputs ([0002], Wang); With respect to claim 4 and 16 Zhao teaches wherein the step of generating an initial virtual human speaking video based on the audio data set and the person speaking video comprises: inputting the audio data set and the person speaking video into a pre-training model to obtain the initial virtual human speaking video in which a person in the person speaking video is driven to speak based on a voice in the audio data set, wherein the duration of the person speaking video is less than duration of the voice in the audio data set (Zhao ¶ [0035] For example, processor 210 may process, for each of the plurality of frames, the image of the human face corresponding to that frame and the audio segment corresponding to that frame using a neural network generator [pretraining model ] to generate the synthetic image [initial human speaking video] corresponding to that frame. The neural network generator can be trained in a generative adversarial network (GAN) including the neural network generator and a discriminator using a plurality of training samples. Each of the training samples may include a training audio segment and a corresponding training human face image containing a mouth shape matching a content of the training audio segment. In some embodiments, the training human face image may include only the mouth region. During training, the neural network generator may generate a human face image based on a training audio segment. The generated human face image can be input into the discriminator to determine whether the input is a generated human face image or a training human face image. The discriminator and the neural network generator can be cross trained iteratively in this manner until a certain condition is met, for example, the discriminator can no longer differentiate the generated human face image from the training human face image. In this way, the neural network generator can be trained to generate a realistic human face image corresponding to an input audio segment, where the mouth shape of the human face image matches the content of the input audio segment.) With respect to claim 5 and 17 Zhao teaches wherein the step of inputting the audio data set and the person speaking video into the pre-training model to obtain the initial virtual human speaking video comprises: using the pre-training model to extract a person speaking feature from the person speaking video, and output the initial virtual human speaking video based on the audio data set and the person speaking feature (Zhao ¶ [0029] Referring back to FIG. 3, method 300 proceeds to step 340, in which processor 210 may generate a synthetic motion picture of the human face based on the reference video and the audio. The synthetic motion picture of the human face may include a motion of the mouth of the human face presenting the speech of the audio. The motion of the mouth may match the content of the speech. For example, the synthetic motion picture may combine the facial expression and other features of the face with a synthetic motion picture of the mouth moving in accordance with the content of the speech to provide a video as if the person is presenting the speech.) With respect to claim 6 and 18 Zhao teaches wherein the duration of the person speaking video is less than or equal to 5 minutes, and the duration of the initial virtual human speaking video is greater than or equal to 10 hours (Zhao ¶0028] In step 330, processor 210 may condition the reference video to match the time duration of the audio. Because the reference video is pre-recorded with a set time duration, the reference video may be longer or shorter in duration than that of the audio. ) In regard to claims 6 Zhao shows the video durations as discussed above. They do not specifically show video duration of 5 minutes and virtual video of duration greater than 10 hours. It would be obvious to one of ordinary skill in the art at the time of the invention to vary the durations as needed because it allows video generation in compatible ways and is a matter of design choice. With respect to claim 8 and 20 Wang further teaches further comprising: using the lip synchronization parameter generation model to generate a lip synchronization parameter that comprises an eye feature parameter and a lip feature parameter (Wang ¶ [0023] The Animation Synthesizer then uses the resulting probabilistic model for selecting, or more specifically, for "predicting," the appropriate animation trajectories for one or more different body parts (e.g., mouth (i.e., lip sync and other mouth motions), nose, eyes, eyebrows, ears, face, head, fingers, hands, arms, legs, feet, torso, spine, skeletal elements of a body, etc.) based on an arbitrary text and/or speech input based on an evaluation of the context, punctuation, and any emotional characteristics associated with that input. These animation trajectories are then used to synthesize a sequence of facial and/or body animations that are synchronized with a speech output corresponding to the text and/or speech input.) It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify video generation of Zhao to include the parameter generation model of Wang in order to generate videos with arbitrary text/audio inputs ([0002], Wang); With respect to claim 9 Zhao teaches wherein the audio data set comprises audio in the person speaking video (Zhao ¶ [0026] In step 310, processor 210 may receive a reference video including a motion picture of a human face. The reference video may be pre-recorded and used as a template for generating a plurality of synthetic videos. For example, the reference video may include a person, such as a realtor, acting naturally while making a normal speech or conversation. The purpose of the reference video is to capture the facial expression, eye blinking, gentle body/face movement, or other similar features that naturally occur during speaking. It is not necessary for the person to speak [lower definition] during the recording of the reference video because the motion of the mouth will be replaced [higher definition] according to specific audio contents during the generation of the synthetic video. With respect to claim 10 Zhao teaches obtaining input audio and a person speaking video with first definition (Zhao ¶ [0026] In step 310, processor 210 may receive a reference video including a motion picture of a human face. The reference video may be pre-recorded and used as a template for generating a plurality of synthetic videos. For example, the reference video may include a person, such as a realtor, acting naturally while making a normal speech or conversation. The purpose of the reference video is to capture the facial expression, eye blinking, gentle body/face movement, or other similar features that naturally occur during speaking. It is not necessary for the person to speak [lower definition] [lower definition] during the recording of the reference video because the motion of the mouth will be replaced [higher definition] [higher definition] according to specific audio contents during the generation of the synthetic video. ) wherein a training set of the lip synchronization parameter generation model is obtained based on a video that comprises the person speaking video with first definition, the first definition is lower than definition of the target virtual human speaking video, and the target virtual human speaking video is obtained based on the person speaking video with first definition (Zhao ¶ [0026] In step 310, processor 210 may receive a reference video including a motion picture of a human face. The reference video may be pre-recorded and used as a template for generating a plurality of synthetic videos. For example, the reference video may include a person, such as a realtor, acting naturally while making a normal speech or conversation. The purpose of the reference video is to capture the facial expression, eye blinking, gentle body/face movement, or other similar features that naturally occur during speaking. It is not necessary for the person to speak [lower definition] [lower definition] during the recording of the reference video because the motion of the mouth will be replaced [higher definition] [higher definition] according to specific audio contents during the generation of the synthetic video. ) Zhao does not explicitly disclose however Want teaches and generating a target virtual human speaking video based on the input audio by using a lip synchronization parameter generation model (Wang ¶ [0023] The Animation Synthesizer then uses the resulting probabilistic model [generating lip synchronization parameter] for selecting , or more specifically, for "predicting," the appropriate animation trajectories for one or more different body parts (e.g., mouth (i.e., lip sync and other mouth motions), nose, eyes, eyebrows, ears, face, head, fingers, hands, arms, legs, feet, torso, spine, skeletal elements of a body, etc.) based on an arbitrary text and/or speech input based on an evaluation of the context, punctuation, and any emotional characteristics associated with that input. ) It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify video generation of Zhao to include the parameter generation model of Wang in order to generate videos with arbitrary text/audio inputs ([0002], Wang); With respect to claim 11, Wang further teaches wherein before the step of generating the target virtual human speaking video based on the input audio by using the lip synchronization parameter generation model, the method further comprises: updating the lip synchronization parameter generation model ([0012] More specifically, given the aforementioned types of inputs for training, the synchronized motion and speech of those inputs is probabilistically modeled at various speech levels, including, for example, sentences (or multiple sentences), phrases, words, phonemes, sub-phonemes, etc., depending upon the available data, and the motion type or body part being modeled. Note that the duration of particular sentences, phrases, words, phonemes, etc. can also be modeled in order to account for differences in speaking speeds (e.g., a fast speaker or a slow speaker). Each body part, e.g., mouth (i.e., lip sync and other mouth motions), nose, eyes, eyebrows, ears, face, head, fingers, hands, arms, legs, feet, torso, spine, skeletal elements of a body, etc., can be modeled using either the same or separate inputs to create one or more probabilistic models. Further, while these speech/motion models can be updated or changed as often as desired, once trained, these models can be stored to a computer-readable medium for later use in synthesizing animations based on new text or speech inputs.) . It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify video generation of Zhao to include the parameter generation model of Wang in order to generate videos with arbitrary text/audio inputs ([0002], Wang); With respect to claim 12 Zhao teaches generating an initial virtual human speaking video based on the input audio and the person speaking video with first definition, wherein duration of the initial virtual human speaking video is greater than duration of the target virtual human speaking video definition (Zhao ¶ [0026] In step 310, processor 210 may receive a reference video including a motion picture of a human face. The reference video may be pre-recorded and used as a template for generating a plurality of synthetic videos. For example, the reference video may include a person, such as a realtor, acting naturally while making a normal speech or conversation. The purpose of the reference video is to capture the facial expression, eye blinking, gentle body/face movement, or other similar features that naturally occur during speaking. It is not necessary for the person to speak [lower definition] [lower definition] during the recording of the reference video because the motion of the mouth will be replaced [higher definition] [higher definition] according to specific audio contents during the generation of the synthetic video. ) Zhao does not explicitly disclose but Wang teaches and updating the lip synchronization parameter generation model by using the initial virtual human speaking video ([0012] More specifically, given the aforementioned types of inputs for training, the synchronized motion and speech of those inputs is probabilistically modeled at various speech levels, including, for example, sentences (or multiple sentences), phrases, words, phonemes, sub-phonemes, etc., depending upon the available data, and the motion type or body part being modeled. Note that the duration of particular sentences, phrases, words, phonemes, etc. can also be modeled in order to account for differences in speaking speeds (e.g., a fast speaker or a slow speaker). Each body part, e.g., mouth (i.e., lip sync and other mouth motions), nose, eyes, eyebrows, ears, face, head, fingers, hands, arms, legs, feet, torso, spine, skeletal elements of a body, etc., can be modeled using either the same or separate inputs to create one or more probabilistic models. Further, while these speech/motion models can be updated or changed as often as desired, once trained, these models can be stored to a computer-readable medium for later use in synthesizing animations based on new text or speech inputs.) . It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify video generation of Zhao to include the parameter generation model of Wang in order to generate videos with arbitrary text/audio inputs ([0002], Wang); 07-21-aia AIA Claim s 7, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao , Wang in further view of Reyes (US 20200007672 A1) . With respect to claims 7 and 19 Zhao and Wang do not explicitly disclose however Reyes teaches wherein the audio data set comprises a plurality of lingual voices, a plurality of tone voices, and a plurality of content voices (Reyes ¶ [0158] least content , tone , format, and language use when sending messages from a user to a given recipient. The at least one processor of each system may analyze the determined content , tone , format, and language use in the context of the relationship between the user and the sender in order to build a dataset of machine knowledge on which to base the generation of messages. Each system on which the present systems and methods are implemented could have access this dataset , both to retrieve data on which to base messages, as well as to provide data to the dataset to improve the dataset ..) It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify video generation of Zhao in view of the parameter generation model of Wang to include datasets of Reyes in order to allow for generalization of video generation with plurality of audio types. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to ATHAR N PASHA whose telephone number is (408)918-7675. The examiner can normally be reached Monday-Thursday Alternate Fridays, 7:30-4:30 PT. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached on (571)272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /ATHAR N PASHA/Primary Examiner, Art Unit 2657 Application/Control Number: 18/889,390 Page 2 Art Unit: 2657 Application/Control Number: 18/889,390 Page 3 Art Unit: 2657 Application/Control Number: 18/889,390 Page 4 Art Unit: 2657 Application/Control Number: 18/889,390 Page 5 Art Unit: 2657 Application/Control Number: 18/889,390 Page 6 Art Unit: 2657 Application/Control Number: 18/889,390 Page 7 Art Unit: 2657 Application/Control Number: 18/889,390 Page 8 Art Unit: 2657 Application/Control Number: 18/889,390 Page 9 Art Unit: 2657 Application/Control Number: 18/889,390 Page 10 Art Unit: 2657 Application/Control Number: 18/889,390 Page 11 Art Unit: 2657 Application/Control Number: 18/889,390 Page 12 Art Unit: 2657 Application/Control Number: 18/889,390 Page 13 Art Unit: 2657 Application/Control Number: 18/889,390 Page 14 Art Unit: 2657
Prosecution Insights