DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Allowable Subject Matter
Claims 3, 4, and 14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement of reasons for allowable subject matter:
Claims 3 and 14 would be allowable over the prior art of record since the cited references taken individually or in combination fails to particularly disclose or suggest a system or method, wherein the one or more processing devices are configured to input the self-attention guidance matrix into the first denoising diffusion model by: computing a product of a self-attention matrix and a first output projection matrix, wherein the self-attention matrix is computed at the corresponding first self-attention layer; and adding the self-attention guidance matrix to the product, as presented in the environment of the remaining limitations of claim 3 (and substantially similar limitations in claim 14). It is noted that the closest prior art, Berlin, shows the limitations of claim 1 (and 12), from which claim 3 (and 14) depends. However, Berlin fails to disclose or suggest computing a product of a self-attention matrix and a first output projection matrix, wherein the self-attention matrix is computed at the corresponding first self-attention layer; and adding the self-attention guidance matrix to the product.
Claim 4 depends from the above claim 3 and would be accordingly allowable.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 5-13, and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Berlin et al. (US Patent 11308657), hereinafter Berlin, in view of Wang et al. (US Pub. 2025/0095250), hereinafter Wang.
Regarding claim 1, Berlin discloses a computing system comprising: one or more processing devices (Fig. 1) configured to: receive a reference image (Column 10, line 58-Column 11, line 5: generic face training images are accessed from one or more data sources. The data sources may be maintained and hosted by the same entity that maintains and operates the learning engine and/or may be maintained and hosted by other entities. As described elsewhere herein, the training images may be option from videos, where the presence of a face may be detected in a given frame or series of frames, the face may be isolated, and a determination may be made as to whether the face satisfies one or more criteria configured to be used to determine if the face images is suitable to be used for training (e.g., includes at least a threshold number of pixels, is not unduly blurred, etc.). Optionally, a given face image may be aligned); receive an input video including a plurality of input video frames (Column 15, lines 34-46: a single source image transfer process may be utilized in conjunction with or instead of the multi-image generic face training process. For example, meta-learning by a generative adversarial network may be performed on a large dataset of images (e.g., images obtained from videos or still images). Then, the generative adversarial network may be capable of few-shot or one-shot learning of neural talking head models of previously unseen people (e.g., as adversarial training problems with generators and discriminators). As described herein in greater detail, the generative adversarial network may include a generator that has a goal producing outputs that a discriminator is unable to distinguish from a given class of training data; Column 26, line 65-Column 27, line 17: some or all of images in the dataset of images may be obtained from one or more videos (e.g., from video frames from thousands, hundreds of thousands, or millions of videos, which may, for example, be posted on open source video sharing sites accessible over a network, such as the Internet). Optionally, in order for a given face from an image to be included in dataset and used for training, multiple images of the given face may need to be identified in a data source); execute a video generation model to compute an output video based at least in part on the reference image and the input video (Fig. 6; Column 4, lines 46-59: conventional techniques for face swapping involve swapping the face of a real person from one image/video to replace the face of another real person in another image/video. By contrast, in order to reduce the needed computer and time resources, and to provide a higher resolution, more realistic output, certain techniques are disclosed that optionally use a computer-generated image (rather than photographs or videos of a real face) to replace the face of a real person in a destination image/video in generating an output image/video. Although certain examples may refer to source, destination, and output videos, it is understood that certain techniques described herein may be also applied to single source, destination, and output images; Column 19, lines 66-Column 20, line 20: FIG. 6 illustrates such an example user interface. The example user interface includes controls organized in function sets, such as a generic face training set of controls, a training set of controls, and a output video creation set of controls. A similar user interface (including a generic face training set of controls, an extract generic face training control, a specify number of processing units for generic face training control, an initiate generic face training control, a terminate generic face training control, an initiate training control, a terminate training control, a select model control, select destination audio control, a select source audio control, select frame size) may be utilized to manage voice processing operations, such as the generic face training, training, and output voice creation. Other controls may include controls for using the trained autoencoders, controls for cleaning and preprocessing the image datasets, and/or the like. For example, noisy images may be identified and removed from the dataset), wherein the video generation model includes: a first denoising diffusion model that includes a plurality of first self-attention layers (Fig, 6; Column 19, line 66-Column 20, line 20: FIG. 6 illustrates such an example user interface. The example user interface includes controls organized in function sets, such as a generic face training set of controls, a training set of controls, and a output video creation set of controls. A similar user interface (including a generic face training set of controls, an extract generic face training control, a specify number of processing units for generic face training control, an initiate generic face training control, a terminate generic face training control, an initiate training control, a terminate training control, a select model control, select destination audio control, a select source audio control, select frame size) may be utilized to manage voice processing operations, such as the generic face training, training, and output voice creation. Other controls may include controls for using the trained autoencoders, controls for cleaning and preprocessing the image datasets, and/or the like. For example, noisy images may be identified and removed from the dataset. Optionally, noisy images may undergo an image de-noising process (using spatial domain filtering, variational denoising, transform techniques, CNN-based denoising methods, and/or using other techniques; Column 29, lines 35-43: a self-attention network module may be utilized that has n inputs, and returns n outputs. The self-attention network enables inputs to interact with each other (“self”) and which inputs should be paid more attention. Corresponding attention scores may be generated for the inputs and/or interactions. The encoder may output aggregates of such interactions and attention scores. Thus, input-dependent dynamic attention weights for aggregating a sequence of hidden states may be generated); and a plurality of dynamics adapters that are respectively associated with the first self-attention layers (Column 8, lines 26-53: such as a learning engine, may be used to identify objects, such as faces in image/video content, and/or to perform a face swapping (or the swapping of other objects) process. For example, the learning engine may include a convolutional neural network (CNN), such as a deep CNN, an example of which is illustrated in FIG. 2A. The CNN may include an input layer 202A, one or more hidden layers 204A, and an output layer 206A. The neural network may be configured as a feed forward network. The neural network may be configured with a shared-weights architecture and with translation invariance characteristics. The hidden layers may be configured as convolutional layers (comprising neurons/nodes), pooling layers, fully connected layers and/or normalization layers. The convolutional deep neural network may be configured with pooling layers that combine outputs of neuron clusters at one layer into a single neuron in the next layer. Max pooling and/or average pooling may be utilized. Max pooling may utilize the maximum value from each of a cluster of neurons at the prior layer. Average pooling may utilize the average value from each of a cluster of neurons at the prior layer. Optionally, attention layers, configured to mask vectors, may be used as hidden layers (e.g., to mask image noise especially for scaled-up images). For example, an attention layer may enable the decoder to look-back at the encoder's hidden states based on its current state. This enables the decoder to extract only relevant information about the input tokens at each decoding, while excluding noise; Column 29, lines 35-43: a self-attention network module may be utilized that has n inputs, and returns n outputs. The self-attention network enables inputs to interact with each other (“self”) and which inputs should be paid more attention. Corresponding attention scores may be generated for the inputs and/or interactions. The encoder may output aggregates of such interactions and attention scores. Thus, input-dependent dynamic attention weights for aggregating a sequence of hidden states may be generated) and are each configured to, for each of the input video frames of the input video: receive the reference image and the input video frame (Column 26, line 65-Column 27, line 15: some or all of images in the dataset of images may be obtained from one or more videos (e.g., from video frames from thousands, hundreds of thousands, or millions of videos, which may, for example, be posted on open source video sharing sites accessible over a network, such as the Internet). Optionally, in order for a given face from an image to be included in dataset and used for training, multiple images of the given face may need to be identified in a data source (e.g., a video). Optionally, a threshold number of images of a given face captured at different angles; Column 28, lines 24-42: the alignment may include alignment of some or all of the following aspects and features relative to a reference face (e.g., a canonical face) or angle: face position, nose position, position of eyes, position of mouth, eye gaze angle, and position/shape of the mouth (e.g., open, closed, talking, smiling, tongue presence/position, etc.). By way of illustration, the geometric structure of the faces in the images (e.g., identifying facial landmarks, such as jawline, face edges, etc.) may be identified and a canonical face alignment may be performed based on normalization, translation, scale, and/or rotation. For example, one or more of the foregoing facial features may be used as an anchor when performing alignment. Optionally, a transform may be applied to a facial image such that the landmarks on the input face match the landmarks of a reference facial image or a supervised learned model); and output the output video for display at a display device (Column 6, lines 51-58: display 112 may be configured to render two dimensional or three dimensional images. The display 112 may include multiple displays which display the output of different applications, GPUs, and/or different content from the data store 110. Thus, for example, a first display may display source content, a second display may display destination content, and a third display may display output content generated by the face swapping application).
Berlin does not explicitly disclose compute a self-attention guidance matrix based at least in part on the reference image and the input video frame; and input the self-attention guidance matrix into the first denoising diffusion model after the first self-attention layer associated with the dynamics adapter.
However, Wang teaches image processing using a reference and input images (Abstract), further comprising compute a self-attention guidance matrix based at least in part on the reference image and the input video frame (Paragraphs [0097]-[0104]: a self-attention feature Ms,t of the first image feature It may be calculated by using the self-attention layer. Specifically, the self-attention layer has three parameters: a query transformation matrix…The first image feature It is linearly transformed separately by using the query transformation matrix…The self-attention feature Ms,t is calculated based on the following formula (2):… the above formula, …is a quantity of rows or a quantity of columns in the key matrix Ks,t, and represents a length of each vector in the key matrix Ks,t. It may be understood that the self-attention feature Ms,t is essentially an attention weight map (attention map), and an element whose coordinates in the weight map are (i, j) indicates a degree of correlation between a feature location i (corresponding to a pixel i in a current image) and a feature location j (corresponding to a pixel j in the current image) in the first image feature…step S232, the self-attention feature Ms,t is multiplied by the value matrix Vs,t calculated based on the first image feature It, to obtain the fourth image feature Is,t, that is, an updated first image feature. That is, in this embodiment, the fourth image feature Is,t is calculated based on the following formula (3):… In step S2321, based on a historical self-attention feature corresponding to the self-attention feature Ms,t, the self-attention feature Ms,t is adjusted, to obtain an adjusted self-attention feature Ms,t′. The historical self-attention feature is an attention feature that is obtained by performing style transfer on the historical image frame of the reference image by using the diffusion model and that is located at a same location as the self-attention feature); and input the self-attention guidance matrix into the first denoising diffusion model after the first self-attention layer associated with the dynamics adapter (Paragraphs [0084]-[0085]: pre-trained diffusion model performs a denoising operation (that is, a reverse diffusion operation) on an initial image for a plurality of times by using the text feature as a condition, to finally obtain the target image. Each denoising operation of the diffusion model corresponds to one time step…pre-trained diffusion model includes a cross-attention layer. An image generation process of the diffusion model is as follows: In each time step t (t=T, T−1, T−2, . . . , 2, 1, where T is an integer greater than 1, such as 50 or 100, and a value of T may be set manually or by a machine), in the cross-attention layer, a currently generated first image feature It and a text feature Text of the description text are used as an input, a first cross-attention feature M*t (that is, a first attention weight map). Wang teaches that this will allow for accurately expressing a visual style feature of the reference image and improving the accuracy of style transfer (Paragraph [0077]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Berlin with the features of above as taught by Wang so as to allow for accurately expressing a visual style feature of the reference image as presented by Wang.
Regarding claim 2, Berlin, in view of Wang teaches the computing system of claim 1, Wang discloses wherein the first denoising diffusion model further includes a plurality of first cross-attention layers and a plurality of first temporal attention layers (Fig. 2; Paragraph [0051]: a first cross-attention feature calculated in an image generation process of a diffusion model is edited by using a second cross-attention feature of an image feature of a reference image and a text feature, so that information in the reference image can be continuously introduced into the image generation process of the diffusion model. Therefore, the information in the reference image can be effectively used to guide image generation of the diffusion model, thereby ensuring that a generated target image can be consistent with the reference image in terms of content and has a specified style; Paragraphs [0104]-[0108]: association between image frames can be established, so that image frames generated after style transfer have good temporal consistency…step S233, a cross-attention feature of the fourth image feature Is,t and the text feature Text is calculated by using the cross-attention layer, and is used as the above first cross-attention feature Mt*. Specifically, a manner of calculating the cross-attention feature of the fourth image feature Is,t and the text feature Text is the same as the above manner of calculating the cross-attention feature of the first image feature).
Regarding claim 5, Berlin, in view of Wang teaches the computing system of claim 1, Berlin discloses wherein: the reference image is an image depicting a first user (Column 4, lines 46-59: conventional techniques for face swapping involve swapping the face of a real person from one image/video to replace the face of another real person in another image/video. By contrast, in order to reduce the needed computer and time resources, and to provide a higher resolution, more realistic output, certain techniques are disclosed that optionally use a computer-generated image (rather than photographs or videos of a real face) to replace the face of a real person in a destination image/video in generating an output image/video. Although certain examples may refer to source, destination, and output videos, it is understood that certain techniques described herein may be also applied to single source, destination, and output images); and the input video is a video depicting a second user (Fig. 3; Column 14, lines 17-35: the images (e.g., facial images) in the destination dataset and the source dataset may be aligned thereby enabling the process to better analyze the differences between the images in destination dataset and the source dataset. For example, the alignment may include alignment of some or all of the following aspects and features: face position, nose position, position of eyes, position of mouth, eye gaze angle, and position/shape of the mouth (e.g., open, closed, talking, smiling, tongue presence/position, etc.). By way of illustration, the process may identify the geometric structure of the faces in the images (e.g., identifying facial landmarks, such as jawline, face edges, etc.) and perform a canonical face alignment based on normalization, translation, scale, and/or rotation. Optionally, a transform may be applied to a facial image such that the landmarks on the input face match the landmarks of a second facial image or a supervised learned model. Optionally, the lighting may be analyzed in respective images to determine if there are lighting gaps that need to be supplemented).
Regarding claim 6, Berlin, in view of Wang teaches the computing system of claim 5, Berlin discloses wherein the video generation model further includes a face control model configured to: receive a first face patch that is included in the reference image and depicts a first user face of the first user (Column 4, lines 46-59: conventional techniques for face swapping involve swapping the face of a real person from one image/video to replace the face of another real person in another image/video. By contrast, in order to reduce the needed computer and time resources, and to provide a higher resolution, more realistic output, certain techniques are disclosed that optionally use a computer-generated image (rather than photographs or videos of a real face) to replace the face of a real person in a destination image/video in generating an output image/video. Although certain examples may refer to source, destination, and output videos, it is understood that certain techniques described herein may be also applied to single source, destination, and output images); and for each of the input video frames: compute a user-swapped face patch that maps the first face patch onto a corresponding second face patch that is included in the input video frame and depicts a second user face of the second user (Fig. 3; Column 12, line 41-Column 13, line 47: block 304B, frames are extracted from the source data set video. The faces in the source frames may be located and aligned (e.g., to remove any rotation relative to the frame base) to facilitate the face swapping operation. Optionally, other preprocessing may be performed. For example automatic time ‘tagging’ may be performed based on which faces are detected in the frames of the video, wherein timing data associated with when in the video a given face was identified may be stored in association with a face identifier…block 306B, the destination data set is accessed. The destination data set may comprise a video including a real, non-CGI character whose face is to be replaced with the CGI face from the source data set. By way of example, the non-CGI character may be engaged in acting a part in a movie, video game, video podcast, music video, or the like. Optionally, the non-CGI character may be speaking or singing in the recording. Advantageously, the destination video may be captured at high resolution and at a high frame rate (e.g., 60-120 frames per second (fps) to provide smoother action), although lower frame rates (e.g., 24 or 30 fps) may be used. At block 308B, frames are extracted from the destination data set video. The faces in the destination frames may be located and aligned (e.g., to remove any rotation relative to the frame base) to facilitate the face swapping operation…. block 310B, at least a portion of the extracted source and destination frames (with the faces aligned) are provided to the learning engine (e.g., a CNN autoencoder). Optionally the same encoder may be trained using both the extracted source frames and the extracted destination frames, but a first decoder may be trained using the latent images generated by the encoder from extracted destination frames, and a second decoder may be trained using the latent images generated by the encoder from extracted source frames. At block 312B, the autoencoder comprising the encoder and the first decoder is used to swap the CGI animated face from the source data set with the face in the destination data set); and input the user-swapped face patch into the first denoising diffusion model (Column 8, lines 26-56: such as a learning engine, may be used to identify objects, such as faces in image/video content, and/or to perform a face swapping (or the swapping of other objects) process. For example, the learning engine may include a convolutional neural network (CNN), such as a deep CNN, an example of which is illustrated in FIG. 2A. The CNN may include an input layer 202A, one or more hidden layers 204A, and an output layer 206A. The neural network may be configured as a feed forward network).
Regarding claim 7, Berlin, in view of Wang teaches the computing system of claim 6, Wang discloses wherein the face control model is a second denoising diffusion model that includes a plurality of second self-attention layers and a plurality of second cross-attention layers (Paragraphs [0084]-[0085]: pre-trained diffusion model performs a denoising operation (that is, a reverse diffusion operation) on an initial image for a plurality of times by using the text feature as a condition, to finally obtain the target image. Each denoising operation of the diffusion model corresponds to one time step…pre-trained diffusion model includes a cross-attention layer. An image generation process of the diffusion model is as follows: In each time step t (t=T, T−1, T−2, . . . , 2, 1, where T is an integer greater than 1, such as 50 or 100, and a value of T may be set manually or by a machine), in the cross-attention layer, a currently generated first image feature It and a text feature Text of the description text are used as an input, a first cross-attention feature M*t (that is, a first attention weight map; Paragraphs [0097]-[0104]: a self-attention feature Ms,t of the first image feature It may be calculated by using the self-attention layer. Specifically, the self-attention layer has three parameters: a query transformation matrix…The first image feature It is linearly transformed separately by using the query transformation matrix…The self-attention feature Ms,t is calculated based on the following formula (2):… the above formula, …is a quantity of rows or a quantity of columns in the key matrix Ks,t, and represents a length of each vector in the key matrix Ks,t. It may be understood that the self-attention feature Ms,t is essentially an attention weight map (attention map), and an element whose coordinates in the weight map are (i, j) indicates a degree of correlation between a feature location i (corresponding to a pixel i in a current image) and a feature location j (corresponding to a pixel j in the current image) in the first image feature…step S232, the self-attention feature Ms,t is multiplied by the value matrix Vs,t calculated based on the first image feature It, to obtain the fourth image feature Is,t, that is, an updated first image feature. That is, in this embodiment, the fourth image feature Is,t is calculated based on the following formula).
Regarding claim 8, Berlin, in view of Wang teaches the computing system of claim 5, Berlin discloses wherein the video generation model further includes a pose control model configured to, for each of the input video frames: receive the input video frame (Column 15, lines 34-46: a single source image transfer process may be utilized in conjunction with or instead of the multi-image generic face training process. For example, meta-learning by a generative adversarial network may be performed on a large dataset of images (e.g., images obtained from videos or still images). Then, the generative adversarial network may be capable of few-shot or one-shot learning of neural talking head models of previously unseen people (e.g., as adversarial training problems with generators and discriminators). As described herein in greater detail, the generative adversarial network may include a generator that has a goal producing outputs that a discriminator is unable to distinguish from a given class of training data); compute a respective body pose of the second user in the input video frame (Column 25, lines 48-60: many-to-one training process will now be described that may be used to train an encoder for use with very large numbers of different faces. A resulting autoencoder will take a source face image and a target face image, and produce a new face image that has the attributes of the target face (pose, lighting, eyeglasses, etc.), but has the identity of the source face. Although the description may refer to a given image containing a source face, the described face swapping operation may be performed for a plurality of video frames containing a source face (e.g., one or more sets of sequential frames), where an autoencoder maps the target face onto the source face, while preserving the expressions, motions, pose, and lighting of the source face); and input the body pose into the first denoising diffusion model (Column 8, lines 26-56: such as a learning engine, may be used to identify objects, such as faces in image/video content, and/or to perform a face swapping (or the swapping of other objects) process. For example, the learning engine may include a convolutional neural network (CNN), such as a deep CNN, an example of which is illustrated in FIG. 2A. The CNN may include an input layer 202A, one or more hidden layers 204A, and an output layer 206A. The neural network may be configured as a feed forward network).
Regarding claim 9, Berlin, in view of Wang teaches the computing system of claim 8, Wang discloses wherein the pose control model is a third denoising diffusion model that includes a plurality of third self-attention layers and a plurality of third cross-attention layers (Paragraphs [0084]-[0085]: pre-trained diffusion model performs a denoising operation (that is, a reverse diffusion operation) on an initial image for a plurality of times by using the text feature as a condition, to finally obtain the target image. Each denoising operation of the diffusion model corresponds to one time step…pre-trained diffusion model includes a cross-attention layer. An image generation process of the diffusion model is as follows: In each time step t (t=T, T−1, T−2, . . . , 2, 1, where T is an integer greater than 1, such as 50 or 100, and a value of T may be set manually or by a machine), in the cross-attention layer, a currently generated first image feature It and a text feature Text of the description text are used as an input, a first cross-attention feature M*t (that is, a first attention weight map; Paragraphs [0097]-[0104]: a self-attention feature Ms,t of the first image feature It may be calculated by using the self-attention layer. Specifically, the self-attention layer has three parameters: a query transformation matrix…The first image feature It is linearly transformed separately by using the query transformation matrix…The self-attention feature Ms,t is calculated based on the following formula (2):… the above formula, …is a quantity of rows or a quantity of columns in the key matrix Ks,t, and represents a length of each vector in the key matrix Ks,t. It may be understood that the self-attention feature Ms,t is essentially an attention weight map (attention map), and an element whose coordinates in the weight map are (i, j) indicates a degree of correlation between a feature location i (corresponding to a pixel i in a current image) and a feature location j (corresponding to a pixel j in the current image) in the first image feature…step S232, the self-attention feature Ms,t is multiplied by the value matrix Vs,t calculated based on the first image feature It, to obtain the fourth image feature Is,t, that is, an updated first image feature. That is, in this embodiment, the fourth image feature Is,t is calculated based on the following formula).
Regarding claim 10, Berlin, in view of Wang teaches the computing system of claim 5, Berlin discloses wherein the dynamics adapters are trained using a training dataset including: a plurality of training reference images (Column 10, line 58-Column 11, line 5: generic face training images are accessed from one or more data sources. The data sources may be maintained and hosted by the same entity that maintains and operates the learning engine and/or may be maintained and hosted by other entities. As described elsewhere herein, the training images may be option from videos, where the presence of a face may be detected in a given frame or series of frames, the face may be isolated, and a determination may be made as to whether the face satisfies one or more criteria configured to be used to determine if the face images is suitable to be used for training (e.g., includes at least a threshold number of pixels, is not unduly blurred, etc.). Optionally, a given face image may be aligned); a plurality of first training input videos that depict humans (Column 16, lines 40-57: generator 1008 is configured to synthesize a real looking face (e.g., starting with random noise as an input). The discriminator 1010 is configured to attempt to identify whether the synthesized face is a real face or a synthesized face. The discriminator 1010 may comprise a neural network that performs classification, where the discriminator 1010 outputs the probability (e.g., in a range of 0 to 1) that image of a face is real. For example, during the training process, the discriminator 1010 may be provided real images of faces from the training source dataset 1004 a portion of the time (e.g., half the time) and fake, synthesized images of faces from the generator 1008 another portion of the time (e.g., half of the time), and attempt to accurately classify the faces as real or synthesized faces. A model error may be generated (minimax loss, Wasserstein loss, etc.) and backpropagated to adjust the generator 1008 neural network weights to improve the performance of the generator 1008 in generating a realistic face); and a plurality of second training input videos that depict dynamic backgrounds and do not depict humans (Column 17, lines 22-36: it is possible or likely that the face generated by the trained generator 1008 will not be perfectly integrated with the rest of the destination head (e.g., neck, ears, hair, long beards, etc.). For example, there may be holes, missing pixels, and/or blurriness in the reconstructed destination image. Hence, an image correction neural network 1012 may be utilized to perform inpainting, hole filing, and/or background replacement (e.g., to add background features from the original destination image that are missing from the generator outputted reconstructed image) on the reconstructed image provided by the generator 1008. For example, the image correction neural network 1012 may reconstruct missing parts of an image so that a view of the image is unable to tell that that regions of the image have undergone restoration).
Regarding claim 11, Berlin, in view of Wang teaches the computing system of claim 1, Wang discloses wherein, during training of the video generation model, at each of the dynamics adapters: respective parameters of a query matrix and the self-attention guidance matrix included in the dynamics adapter are modified (Paragraphs [0097]-[0104]: a self-attention feature Ms,t of the first image feature It may be calculated by using the self-attention layer. Specifically, the self-attention layer has three parameters: a query transformation matrix…The first image feature It is linearly transformed separately by using the query transformation matrix…The self-attention feature Ms,t is calculated based on the following formula (2):… the above formula, …is a quantity of rows or a quantity of columns in the key matrix Ks,t, and represents a length of each vector in the key matrix Ks,t. It may be understood that the self-attention feature Ms,t is essentially an attention weight map (attention map), and an element whose coordinates in the weight map are (i, j) indicates a degree of correlation between a feature location i (corresponding to a pixel i in a current image) and a feature location j (corresponding to a pixel j in the current image) in the first image feature…step S232, the self-attention feature Ms,t is multiplied by the value matrix Vs,t calculated based on the first image feature It, to obtain the fourth image feature Is,t, that is, an updated first image feature. That is, in this embodiment, the fourth image feature Is,t is calculated based on the following formula (3):… In step S2321, based on a historical self-attention feature corresponding to the self-attention feature Ms,t, the self-attention feature Ms,t is adjusted, to obtain an adjusted self-attention feature Ms,t′. The historical self-attention feature is an attention feature that is obtained by performing style transfer on the historical image frame of the reference image by using the diffusion model and that is located at a same location as the self-attention feature); and respective parameters of a key matrix and a value matrix included in the dynamics adapter are held constant (Paragraphs [0091]-[0100]: cross-attention layer uses the first image feature I.sub.t and the text feature Text of the description text as an input. The first image feature I.sub.t is linearly transformed by using the query transformation matrix WQ to obtain a query matrix Qt=IrWQ. The text feature Text is linearly transformed by using the key transformation matrix WK and the value transformation matrix WV, to obtain a key matrix …and a value matrix …first cross-attention feature M*.sub.t is calculated based on the following formula… is a quantity of rows or a quantity of columns in the key matrix Kt, and represents a length of each vector in the key matrix Kt. It may be understood that the first cross-attention feature M*t is essentially an attention weight map (attention map), and an element whose coordinates in the weight map are (i, j) indicates a degree of correlation between a feature location i (corresponding to a pixel i in a current image) in the first image feature It and a feature location j (corresponding to a jth token in the description text) in the text feature Text).
Regarding claim 12, the limitations of this claim substantially correspond to the limitations of claim 1; thus they are rejected on similar grounds.
Regarding claim 13, the limitations of this claim substantially correspond to the limitations of claim 2; thus they are rejected on similar grounds.
Regarding claim 15, the limitations of this claim substantially correspond to the limitations of claim 5; thus they are rejected on similar grounds.
Regarding claim 16, the limitations of this claim substantially correspond to the limitations of claim 6; thus they are rejected on similar grounds.
Regarding claim 17, the limitations of this claim substantially correspond to the limitations of claim 8; thus they are rejected on similar grounds.
Regarding claim 18, the limitations of this claim substantially correspond to the limitations of claim 10; thus they are rejected on similar grounds.
Regarding claim 19, the limitations of this claim substantially correspond to the limitations of claim 11; thus they are rejected on similar grounds.
Regarding claim 20, Berlin discloses a computing system comprising: one or more processing devices (Fig. 1) configured to: receive a reference image, wherein the reference image is an image depicting a first user (Column 4, lines 46-59: conventional techniques for face swapping involve swapping the face of a real person from one image/video to replace the face of another real person in another image/video. By contrast, in order to reduce the needed computer and time resources, and to provide a higher resolution, more realistic output, certain techniques are disclosed that optionally use a computer-generated image (rather than photographs or videos of a real face) to replace the face of a real person in a destination image/video in generating an output image/video. Although certain examples may refer to source, destination, and output videos, it is understood that certain techniques described herein may be also applied to single source, destination, and output images; Column 10, line 58-Column 11, line 5: generic face training images are accessed from one or more data sources. The data sources may be maintained and hosted by the same entity that maintains and operates the learning engine and/or may be maintained and hosted by other entities. As described elsewhere herein, the training images may be option from videos, where the presence of a face may be detected in a given frame or series of frames, the face may be isolated, and a determination may be made as to whether the face satisfies one or more criteria configured to be used to determine if the face images is suitable to be used for training (e.g., includes at least a threshold number of pixels, is not unduly blurred, etc.). Optionally, a given face image may be aligned); receive an input video including a plurality of input video frames, wherein the input video is a video depicting a second user (Fig. 3; Column 14, lines 17-35: the images (e.g., facial images) in the destination dataset and the source dataset may be aligned thereby enabling the process to better analyze the differences between the images in destination dataset and the source dataset. For example, the alignment may include alignment of some or all of the following aspects and features: face position, nose position, position of eyes, position of mouth, eye gaze angle, and position/shape of the mouth (e.g., open, closed, talking, smiling, tongue presence/position, etc.). By way of illustration, the process may identify the geometric structure of the faces in the images (e.g., identifying facial landmarks, such as jawline, face edges, etc.) and perform a canonical face alignment based on normalization, translation, scale, and/or rotation. Optionally, a transform may be applied to a facial image such that the landmarks on the input face match the landmarks of a second facial image or a supervised learned model. Optionally, the lighting may be analyzed in respective images to determine if there are lighting gaps that need to be supplemented; Column 15, lines 34-46: a single source image transfer process may be utilized in conjunction with or instead of the multi-image generic face training process. For example, meta-learning by a generative adversarial network may be performed on a large dataset of images (e.g., images obtained from videos or still images). Then, the generative adversarial network may be capable of few-shot or one-shot learning of neural talking head models of previously unseen people (e.g., as adversarial training problems with generators and discriminators). As described herein in greater detail, the generative adversarial network may include a generator that has a goal producing outputs that a discriminator is unable to distinguish from a given class of training data; Column 26, line 65-Column 27, line 17: some or all of images in the dataset of images may be obtained from one or more videos (e.g., from video frames from thousands, hundreds of thousands, or millions of videos, which may, for example, be posted on open source video sharing sites accessible over a network, such as the Internet). Optionally, in order for a given face from an image to be included in dataset and used for training, multiple images of the given face may need to be identified in a data source); execute a video generation model to compute an output video based at least in part on the reference image and the input video (Fig. 6; Column 4, lines 46-59: conventional techniques for face swapping involve swapping the face of a real person from one image/video to replace the face of another real person in another image/video. By contrast, in order to reduce the needed computer and time resources, and to provide a higher resolution, more realistic output, certain techniques are disclosed that optionally use a computer-generated image (rather than photographs or videos of a real face) to replace the face of a real person in a destination image/video in generating an output image/video. Although certain examples may refer to source, destination, and output videos, it is understood that certain techniques described herein may be also applied to single source, destination, and output images; Column 19, lines 66-Column 20, line 20: FIG. 6 illustrates such an example user interface. The example user interface includes controls organized in function sets, such as a generic face training set of controls, a training set of controls, and a output video creation set of controls. A similar user interface (including a generic face training set of controls, an extract generic face training control, a specify number of processing units for generic face training control, an initiate generic face training control, a terminate generic face training control, an initiate training control, a terminate training control, a select model control, select destination audio control, a select source audio control, select frame size) may be utilized to manage voice processing operations, such as the generic face training, training, and output voice creation. Other controls may include controls for using the trained autoencoders, controls for cleaning and preprocessing the image datasets, and/or the like. For example, noisy images may be identified and removed from the dataset), wherein the video generation model includes: a first denoising diffusion model that includes a plurality of first self-attention layers (Fig, 6; Column 19, line 66-Column 20, line 20: FIG. 6 illustrates such an example user interface. The example user interface includes controls organized in function sets, such as a generic face training set of controls, a training set of controls, and a output video creation set of controls. A similar user interface (including a generic face training set of controls, an extract generic face training control, a specify number of processing units for generic face training control, an initiate generic face training control, a terminate generic face training control, an initiate training control, a terminate training control, a select model control, select destination audio control, a select source audio control, select frame size) may be utilized to manage voice processing operations, such as the generic face training, training, and output voice creation. Other controls may include controls for using the trained autoencoders, controls for cleaning and preprocessing the image datasets, and/or the like. For example, noisy images may be identified and removed from the dataset. Optionally, noisy images may undergo an image de-noising process (using spatial domain filtering, variational denoising, transform techniques, CNN-based denoising methods, and/or using other techniques; Column 29, lines 35-43: a self-attention network module may be utilized that has n inputs, and returns n outputs. The self-attention network enables inputs to interact with each other (“self”) and which inputs should be paid more attention. Corresponding attention scores may be generated for the inputs and/or interactions. The encoder may output aggregates of such interactions and attention scores. Thus, input-dependent dynamic attention weights for aggregating a sequence of hidden states may be generated); and a plurality of dynamics adapters (Column 8, lines 26-53: such as a learning engine, may be used to identify objects, such as faces in image/video content, and/or to perform a face swapping (or the swapping of other objects) process. For example, the learning engine may include a convolutional neural network (CNN), such as a deep CNN, an example of which is illustrated in FIG. 2A. The CNN may include an input layer 202A, one or more hidden layers 204A, and an output layer 206A. The neural network may be configured as a feed forward network. The neural network may be configured with a shared-weights architecture and with translation invariance characteristics. The hidden layers may be configured as convolutional layers (comprising neurons/nodes), pooling layers, fully connected layers and/or normalization layers. The convolutional deep neural network may be configured with pooling layers that combine outputs of neuron clusters at one layer into a single neuron in the next layer. Max pooling and/or average pooling may be utilized. Max pooling may utilize the maximum value from each of a cluster of neurons at the prior layer. Average pooling may utilize the average value from each of a cluster of neurons at the prior layer. Optionally, attention layers, configured to mask vectors, may be used as hidden layers (e.g., to mask image noise especially for scaled-up images). For example, an attention layer may enable the decoder to look-back at the encoder's hidden states based on its current state. This enables the decoder to extract only relevant information about the input tokens at each decoding, while excluding noise; Column 29, lines 35-43: a self-attention network module may be utilized that has n inputs, and returns n outputs. The self-attention network enables inputs to interact with each other (“self”) and which inputs should be paid more attention. Corresponding attention scores may be generated for the inputs and/or interactions. The encoder may output aggregates of such interactions and attention scores. Thus, input-dependent dynamic attention weights for aggregating a sequence of hidden states may be generated) that are each configured to, for each of the input video frames of the input video: receive the reference image and the input video frame (Column 26, line 65-Column 27, line 15: some or all of images in the dataset of images may be obtained from one or more videos (e.g., from video frames from thousands, hundreds of thousands, or millions of videos, which may, for example, be posted on open source video sharing sites accessible over a network, such as the Internet). Optionally, in order for a given face from an image to be included in dataset and used for training, multiple images of the given face may need to be identified in a data source (e.g., a video). Optionally, a threshold number of images of a given face captured at different angles; Column 28, lines 24-42: the alignment may include alignment of some or all of the following aspects and features relative to a reference face (e.g., a canonical face) or angle: face position, nose position, position of eyes, position of mouth, eye gaze angle, and position/shape of the mouth (e.g., open, closed, talking, smiling, tongue presence/position, etc.). By way of illustration, the geometric structure of the faces in the images (e.g., identifying facial landmarks, such as jawline, face edges, etc.) may be identified and a canonical face alignment may be performed based on normalization, translation, scale, and/or rotation. For example, one or more of the foregoing facial features may be used as an anchor when performing alignment. Optionally, a transform may be applied to a facial image such that the landmarks on the input face match the landmarks of a reference facial image or a supervised learned model); a face control model configured to: receive a first face patch that is included in the reference image and depicts a first user face of the first user (Column 4, lines 46-59: conventional techniques for face swapping involve swapping the face of a real person from one image/video to replace the face of another real person in another image/video. By contrast, in order to reduce the needed computer and time resources, and to provide a higher resolution, more realistic output, certain techniques are disclosed that optionally use a computer-generated image (rather than photographs or videos of a real face) to replace the face of a real person in a destination image/video in generating an output image/video. Although certain examples may refer to source, destination, and output videos, it is understood that certain techniques described herein may be also applied to single source, destination, and output images); and for each of the input video frames: compute a user-swapped face patch that maps the face patch onto a corresponding second face patch that is included in the input video frame and depicts a second user face of the second user (Fig. 3; Column 12, line 41-Column 13, line 47: block 304B, frames are extracted from the source data set video. The faces in the source frames may be located and aligned (e.g., to remove any rotation relative to the frame base) to facilitate the face swapping operation. Optionally, other preprocessing may be performed. For example automatic time ‘tagging’ may be performed based on which faces are detected in the frames of the video, wherein timing data associated with when in the video a given face was identified may be stored in association with a face identifier…block 306B, the destination data set is accessed. The destination data set may comprise a video including a real, non-CGI character whose face is to be replaced with the CGI face from the source data set. By way of example, the non-CGI character may be engaged in acting a part in a movie, video game, video podcast, music video, or the like. Optionally, the non-CGI character may be speaking or singing in the recording. Advantageously, the destination video may be captured at high resolution and at a high frame rate (e.g., 60-120 frames per second (fps) to provide smoother action), although lower frame rates (e.g., 24 or 30 fps) may be used. At block 308B, frames are extracted from the destination data set video. The faces in the destination frames may be located and aligned (e.g., to remove any rotation relative to the frame base) to facilitate the face swapping operation…. block 310B, at least a portion of the extracted source and destination frames (with the faces aligned) are provided to the learning engine (e.g., a CNN autoencoder). Optionally the same encoder may be trained using both the extracted source frames and the extracted destination frames, but a first decoder may be trained using the latent images generated by the encoder from extracted destination frames, and a second decoder may be trained using the latent images generated by the encoder from extracted source frames. At block 312B, the autoencoder comprising the encoder and the first decoder is used to swap the CGI animated face from the source data set with the face in the destination data set); and input the user-swapped face patch into the first denoising diffusion model (Column 8, lines 26-56: such as a learning engine, may be used to identify objects, such as faces in image/video content, and/or to perform a face swapping (or the swapping of other objects) process. For example, the learning engine may include a convolutional neural network (CNN), such as a deep CNN, an example of which is illustrated in FIG. 2A. The CNN may include an input layer 202A, one or more hidden layers 204A, and an output layer 206A. The neural network may be configured as a feed forward network); and a pose control model configured to, for each of the input video frames: receive the input video frame (Column 15, lines 34-46: a single source image transfer process may be utilized in conjunction with or instead of the multi-image generic face training process. For example, meta-learning by a generative adversarial network may be performed on a large dataset of images (e.g., images obtained from videos or still images). Then, the generative adversarial network may be capable of few-shot or one-shot learning of neural talking head models of previously unseen people (e.g., as adversarial training problems with generators and discriminators). As described herein in greater detail, the generative adversarial network may include a generator that has a goal producing outputs that a discriminator is unable to distinguish from a given class of training data); compute a respective body pose of the second user in the input video frame (Column 25, lines 48-60: many-to-one training process will now be described that may be used to train an encoder for use with very large numbers of different faces. A resulting autoencoder will take a source face image and a target face image, and produce a new face image that has the attributes of the target face (pose, lighting, eyeglasses, etc.), but has the identity of the source face. Although the description may refer to a given image containing a source face, the described face swapping operation may be performed for a plurality of video frames containing a source face (e.g., one or more sets of sequential frames), where an autoencoder maps the target face onto the source face, while preserving the expressions, motions, pose, and lighting of the source face); and input the body pose into the first denoising diffusion model (Column 8, lines 26-56: such as a learning engine, may be used to identify objects, such as faces in image/video content, and/or to perform a face swapping (or the swapping of other objects) process. For example, the learning engine may include a convolutional neural network (CNN), such as a deep CNN, an example of which is illustrated in FIG. 2A. The CNN may include an input layer 202A, one or more hidden layers 204A, and an output layer 206A. The neural network may be configured as a feed forward network); and output the output video for display at a display device (Column 6, lines 51-58: display 112 may be configured to render two dimensional or three dimensional images. The display 112 may include multiple displays which display the output of different applications, GPUs, and/or different content from the data store 110. Thus, for example, a first display may display source content, a second display may display destination content, and a third display may display output content generated by the face swapping application).
Berlin does not explicitly disclose compute a self-attention guidance matrix based at least in part on the reference image and the input video frame; and input the self-attention guidance matrix into the first denoising diffusion model.
However, Wang teaches image processing using a reference and input images (Abstract), further comprising compute a self-attention guidance matrix based at least in part on the reference image and the input video frame (Paragraphs [0097]-[0104]: a self-attention feature Ms,t of the first image feature It may be calculated by using the self-attention layer. Specifically, the self-attention layer has three parameters: a query transformation matrix…The first image feature It is linearly transformed separately by using the query transformation matrix…The self-attention feature Ms,t is calculated based on the following formula (2):… the above formula, …is a quantity of rows or a quantity of columns in the key matrix Ks,t, and represents a length of each vector in the key matrix Ks,t. It may be understood that the self-attention feature Ms,t is essentially an attention weight map (attention map), and an element whose coordinates in the weight map are (i, j) indicates a degree of correlation between a feature location i (corresponding to a pixel i in a current image) and a feature location j (corresponding to a pixel j in the current image) in the first image feature…step S232, the self-attention feature Ms,t is multiplied by the value matrix Vs,t calculated based on the first image feature It, to obtain the fourth image feature Is,t, that is, an updated first image feature. That is, in this embodiment, the fourth image feature Is,t is calculated based on the following formula (3):… In step S2321, based on a historical self-attention feature corresponding to the self-attention feature Ms,t, the self-attention feature Ms,t is adjusted, to obtain an adjusted self-attention feature Ms,t′. The historical self-attention feature is an attention feature that is obtained by performing style transfer on the historical image frame of the reference image by using the diffusion model and that is located at a same location as the self-attention feature); and input the self-attention guidance matrix into the first denoising diffusion model (Paragraphs [0084]-[0085]: pre-trained diffusion model performs a denoising operation (that is, a reverse diffusion operation) on an initial image for a plurality of times by using the text feature as a condition, to finally obtain the target image. Each denoising operation of the diffusion model corresponds to one time step…pre-trained diffusion model includes a cross-attention layer. An image generation process of the diffusion model is as follows: In each time step t (t=T, T−1, T−2, . . . , 2, 1, where T is an integer greater than 1, such as 50 or 100, and a value of T may be set manually or by a machine), in the cross-attention layer, a currently generated first image feature It and a text feature Text of the description text are used as an input, a first cross-attention feature M*t (that is, a first attention weight map). Wang teaches that this will allow for accurately expressing a visual style feature of the reference image and improving the accuracy of style transfer (Paragraph [0077]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Berlin with the features of above as taught by Wang so as to allow for accurately expressing a visual style feature of the reference image as presented by Wang.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Klingler et al. (US Pub. 2025/0182365) teaches use of self-attention and/or cross-attention in video generation.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW D SALVUCCI whose telephone number is (571)270-5748. The examiner can normally be reached M-F: 7:30-4:00PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XIAO WU can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW SALVUCCI/Primary Examiner, Art Unit 2613
Prosecution Insights