DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are pending under this Office action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 14 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over He, etc. (US 20260099978 A1) in view of Daha. etc. (US 20240386661 A1) further in view of Ravi, etc. (US 20250117972 A1) and Choi, etc. (US 20250356171 A1).
Regarding claim 1, He teaches that a system (See He: Figs. 1-2, and [0062], “FIG. 2 illustrates a block diagram of an exemplary hardware/software architecture of a communication device such as, for example, user equipment (UE) 30, in accordance with various aspects of the present disclosure. In some exemplary respects, the UE 30 may be any of communication devices 105, 110, 115, 120. In some exemplary aspects, the UE 30 may be a computer system such as, for example, a desktop computer, notebook or laptop computer, netbook, a tablet computer (e.g., a smart tablet), e-book reader, GPS device, camera, personal digital assistant, handheld electronic device, cellular telephone, smartphone, smart glasses, augmented/virtual reality device, smart watch, charging case, or any other suitable electronic device. As shown in FIG. 2, the UE 30 (also referred to herein as node 30) may include a processor 32, non-removable memory 44, removable memory 46, a speaker/microphone 38, a display, touchpad, and/or user interface(s) 42, a power source 48, a GPS chipset 50, and other peripherals 52. In some exemplary aspects, the display, touchpad, and/or user interface(s) 42 may be referred to herein as display/touchpad/user interface(s) 42. The display/touchpad/user interface(s) 42 may include a user interface capable of presenting one or more content items and/or capturing input of one or more user interactions/actions associated with the user interface. The power source 48 may be capable of receiving electric power for supplying electric power to the UE 30. For example, the power source 48 may include an alternating current to direct current (AC-to-DC) converter allowing the power source 48 to be connected/plugged to an AC electrical receptacle and/or Universal Serial Bus (USB) port for receiving electric power. The UE 30 may also include a camera 54. In an exemplary embodiment, the camera 54 may be a smart camera configured to sense images/video appearing within one or more bounding boxes. The UE 30 may also include communication circuitry, such as a transceiver 34 and a transmit/receive element 36. It will be appreciated the UE 30 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment”) comprising:
at least one processor (See He: Figs. 2-6, and [0155], “One or more of the components shown in the drawings, such as in FIG. 2, FIG. 4, FIG. 5, or FIG. 6, may be implemented using hardware (e.g., a processor of a machine) or a combination of hardware and software. For example, a component described herein may configure a processor to perform the operations described herein for that component. Moreover, two or more of these components may be combined into a single component, and the functions described herein for a single component may be subdivided among multiple components. Furthermore, according to various examples, components described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or device”);
at least one memory component storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations (See He: Fig. 13, and [0254], “The head-wearable apparatus 1302 includes a memory 1322 which stores instructions to perform at least a subset of functions of the head-wearable apparatus 1302. The memory 1322 can also include a storage device. As further shown in FIG. 13, the high-speed circuitry 1318 includes a high-speed processor 1320, the memory 1322, and high-speed wireless circuitry 1324. In FIG. 13, the image display driver 1308 is coupled to the high-speed circuitry 1318 and operated by the high-speed processor 1320 in order to drive the left and right image displays of the image displays 1304. The high-speed processor 1320 may be any processor capable of managing high-speed communications and operation of any general computing system needed for the head-wearable apparatus 1302. The high-speed processor 1320 includes processing resources needed for managing high-speed data transfers over the communication link 1336 to a wireless local area network (WLAN) using high-speed wireless circuitry 1324. In certain examples, the high-speed processor 1320 executes an operating system such as a LINUX operating system or other such operating system of the head-wearable apparatus 1302 and the operating system is stored in memory 1322 for execution. In addition to any other responsibilities, the high-speed processor 1320 executing a software architecture for the head-wearable apparatus 1302 is used to manage data transfers with high-speed wireless circuitry 1324. In certain examples, high-speed wireless circuitry 1324 is configured to implement Institute of Electrical and Electronic Engineers (IEEE) 1302.11 communication standards, also referred to herein as Wi-Fi™. In other examples, other high-speed communications standards may be implemented by high-speed wireless circuitry 1324”) comprising:
receiving a text prompt describing a video to be generated, a plurality of reference images each depicting a corresponding subject (See He: Figs. 1-2, and [0112], “The user interface component 402 enables interactions between a user of a user system 102 and the image generation system 230. The user interface component 402 is configured to receive user inputs, such as image uploads, text prompts, and selections of desired image attributes. The user interface component 402 facilitates navigation and operation of image generation system 230, providing tools and options, for example via the interaction client 104, that are useful for running or customizing the image generation process”; and [0127], “In some examples in the present disclosure, including those described with reference to FIG. 5, FIG. 6, and FIG. 11, a diffusion model is employed by the image generation system 230 to generate images. A diffusion model is a type of generative machine learning model that can be used to generate images conditioned on one or more inputs. It is based on the concept of “diffusing” noise throughout an image to transform it gradually into a new image. A diffusion model may use a sequence of invertible transformations to transform a random noise image into a final image. During training, a diffusion model may learn sequences of transformations that can best transform random noise images into desired output images. A diffusion model can be fed with input data (e.g., text describing the desired images) and the corresponding output images, and the parameters of the model are adjusted iteratively to improve its ability to generate accurate or good quality images”), and binding information that associates each reference image with a corresponding entity word in the text prompt;
extracting features from each reference image and fusing the extracted features with text embeddings of the corresponding entity word (See He: Figs. 1-6, and [0153], “By processing the combined identity representation 604 and the text prompt representation 606 via separate layers of the diffusion model 602 (instead of, for example, concatenating them and embedding them in the same layers), the accuracy and fidelity of a subject's portrayal (as captured in the combined identity representation 604) can be better maintained. Accordingly, the cross-attention mechanism depicted in FIG. 6 can be classified as “decoupled” since it separates cross-attention layers for text-based features (e.g., text prompts) and image-based features (e.g., image prompts as merged into a combined representation)”; [0157], “The method 700 commences at opening loop operation 702, and proceeds to operation 704 in which the image generation system 230 accesses a plurality of input images (e.g., via the input image processing component 404). The input images are provided by a user via the interaction client 104 (as an example of an interaction application), and all depict a subject (e.g., the user or another person). For example, the user uses their user system 102 (e.g., the mobile device 114 or some other user device) to upload three input images via an image upload user interface provided by the interaction client 104. The image upload user interface can be provided as part of an onboarding process related to a personalized image generation feature”; and [0176], “In some examples, the image generation system 230 processes each input image submitted by the user using computer vision techniques to determine (e.g., via the input image processing component 404) whether the input image is suitable for use in generating a combined identity representation. For example, the image generation system 230 checks whether the input image depicts the relevant body part (e.g., the face), whether the lighting is suitable for feature extraction, or, where relevant, whether the image depicts the subject in a requested pose, with a requested facial expression, or from a requested angle. In some examples, the image generation system 230 automatically rejects an unsuitable or sub-optimal image and causes the user to receive a message, via the interaction client 104, indicating that they should upload a new image”) based on the binding information to create personalization embeddings;
generating an initial noisy video (See He: Figs. 1-6, and [0035], “Techniques described herein can be used to generate individual images or video content. In some examples, the personalized output image is one of a plurality of frames of a personalized video. For example, a personalized video is generated for the user, via the interaction application, based on the combined identity representation and an additional image generation control, such as a text prompt. The personalized video comprises multiple frames that depict a person resembling the subject, based on the combined identity representation”; and [0088], “The artificial intelligence and machine learning system 232 may also provide chatbot functionality to interactions 120 between user systems 102 and between a user system 102 and the interaction server system 110. The artificial intelligence and machine learning system 232 may work with the audio communication system 216 to provide speech recognition and natural language processing capabilities, allowing users to interact with the interaction system 100 using voice commands. The artificial intelligence and machine learning system 232 may also provide or facilitate generative AI functionality, e.g., allowing a user to generate text, image, or video content based on prompts and/or other instructions. In some examples, the artificial intelligence and machine learning system 232 provide a generative AI assistant that can answer questions provided by the user or otherwise help the user to learn about topics or obtain useful information”); and
iteratively denoising the noisy video using a diffusion model conditioned on both the text prompt and the personalization embeddings (See He: Figs. 1-6, and [0151], “One approach to enabling the diffusion model 602 to handle both image-based features (e.g., the combined identity representation 604) and text-based features (e.g., the text prompt representation 606) is to concatenate these features into the same cross-attention layers of the diffusion model 602. However, this may prevent the diffusion model 602 from capturing fine-grained features from input images. Instead, in some examples, the diffusion model 602 is adapted from a pre-trained model by embedding the image-based features via separate cross-attention layers that are different than the cross-attention layers handling the text-based features”; [0152], “In some examples, the diffusion model 602 is obtained by taking a pre-trained diffusion model with cross-attention layers that handle the text-based features, and adding components in the form of new cross-attention layers for the image-based features. In some examples, a new cross-attention layer is added for each cross-attention layer in the original denoising network 608 (e.g., U-Net model component)”; and [0127], “In some examples in the present disclosure, including those described with reference to FIG. 5, FIG. 6, and FIG. 11, a diffusion model is employed by the image generation system 230 to generate images. A diffusion model is a type of generative machine learning model that can be used to generate images conditioned on one or more inputs. It is based on the concept of “diffusing” noise throughout an image to transform it gradually into a new image. A diffusion model may use a sequence of invertible transformations to transform a random noise image into a final image. During training, a diffusion model may learn sequences of transformations that can best transform random noise images into desired output images. A diffusion model can be fed with input data (e.g., text describing the desired images) and the corresponding output images, and the parameters of the model are adjusted iteratively to improve its ability to generate accurate or good quality images”) to generate a personalized video that preserves identity of the corresponding subjects in the reference images while allowing for natural motion and pose variations (See He: Figs. 1-3, and [0001], “Cross-paired data may include both real and synthetic examples. Real cross-paired data may be sourced from videos containing multiple views of the same person. Synthetic cross-paired data may be generated using a personalization image generation model to create novel reference images from a base frame, varying properties such as facial expression, head pose, or lighting. Generated synthetic reference images may be filtered based on similarity thresholds to ensure identity consistency”).
However, He fails to explicitly disclose that binding information that associates each reference image with a corresponding entity word in the text prompt; based on the binding information to create personalization embeddings; and to generate a personalized video that preserves identity of the corresponding subjects in the reference images while allowing for natural motion.
However, Daha teaches that binding information that associates each reference image with a corresponding entity word in the text prompt (See Daha: Figs. 2A-B, and [0046], “The client application 214 can generate the arbitrary text string or identifier [V] that will be associated with the object of the object image 104 or the background of the background image 105. As described below, the client application 214 can also submit the identifier [V] with the respective image(s) of the object or background to the 3D image generating AI model 107 to fine-tune the model 107, so that the output 3D image 108 is true in appearance to the specific object and background desired by the user 212. Specifically, the data for the image of the object and/or the background is implanted in the output domain of the image generating AI model 107 and associated with a unique identifier. Otherwise, the object and background would be generated according to the entire class of such objects/backgrounds in the training set of the AI model 107 and would not be specific to the inputs of the user 212”; and [005], “As noted above, the disclosed technique can fine-tune an AI model with input images and text cues that contain a unique identifier followed by the category name of the object (e.g., “A [V] hat”), where [V] is the identifier or token associated with image data representing a enables the model to use its prior knowledge of the object's category, while instances of the particular category, i.e., the specific hat that the user has imaged, are bound to the unique identifier. Thus, the model can produce the desired 3D image of the object with the specified background, the 3D image being true to the appearance of the specified object and background”; and [0055], “In the case of either or both of the object and background, a special identifier is generated, represented herein as [V]. As noted above, this identifier or unique identifier can be an arbitrary text string that would not typically occur in any natural language communication. This “rare text” identifier will then represent the corresponding object or background in the lexicon of the image generating AI model. For example, [V] is a special text identifier in the system, such as the string ‘sks.’ If the object is a hat, a text instruction invoking the specific hat associated with this identifier could be “a photo of sks hat.” The AI model will then output an image, not just of any hat, but an image of the specific hat associated with the identifier “sks.” Again, the identifier could be any text string, arbitrarily, such as a very rare string or word that would not be used in a common sentence”. Note that the image bound to the identifier (the rare text) is mapped to binding information that associates each reference image with a corresponding entity word in the text prompt).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify He to have a predictor for predicting the prediction target as taught by Daha in order to allow customers to better visualize the size, shape and overall design of a product interested in purchasing, which helps customer to make a more informed decision (See Daha: Fig. 1, and [0024], “Having a 3D image or model that allows customers to better visualize the size, shape, and overall design of a product they are interested in purchasing can help them make a more informed decision. Additionally, 3D images can provide a more immersive and interactive experience, allowing users to explore and interact with the product in a way that 2D images simply cannot. Another important benefit of viewing 3D images of products before buying them is the ability to see them in different environments and against diverse backgrounds. This can be helpful in many ways. For example, it can help a user better understand how a product will look in their own home or office, or in a specific room or space they have in mind. Additionally, these types of images and models can help users better understand how a product will look when used in a particular context”). He teaches a method and system that may receive text prompt and reference image, embed identity feature vis encoder, use a generation model to generate video that preserves identity with natural motion variation; while Daha teaches a system and method that may associate reference image’s visual data with a corresponding identifier/entity word that is used in the prompt and for personalization. Therefore, it is obvious for one of ordinary skill in the art to modify He by Daha to bind the reference image with the text prompt and using the embedding to generate personalized image (video). The motivation to modify He by Daha is “Use of known technique to improve similar devices (methods, or products) in the same way”.
However, He, modified by Daha, fails to explicitly disclose that based on the binding information to create personalization embeddings; and to generate a personalized video that preserves identity of the corresponding subjects in the reference images while allowing for natural motion.
However, Ravi teaches that based on the binding information to create personalization embeddings (See Ravi: Figs. 1-3, and [0024, “The present disclosure describes systems and methods that improve on conventional image generation models by providing more accurate depiction of image-related attributes in synthesized images. For example, generated images can include attributes and elements depicted in a reference image. That is, users can achieve more precise control over information represented in an image embedding of the reference image compared to conventional generative models. Embodiments achieve this improved accuracy and control by having separate learnable cross-attention layers for text embedding and image embedding and the model processes the text embedding and the image embedding in parallel (i.e., different from concatenating different types of embeddings into a single long sequence of embeddings). In some cases, the text attention output and the image attention output are generated in separate pathways”: [0028], “In some examples, image generation apparatus 110 encodes the text prompt to obtain a text embedding and encodes the image prompt to obtain an image embedding. Image generation apparatus 110 performs, using a text attention layer of an image generation model, cross-attention on the text embedding to obtain a text attention output. Image generation apparatus 110 performs, using an image attention layer of the image generation model, cross-attention on the image embedding to obtain an image attention output. Image generation apparatus 110 generates, using a generator network of the image generation model, a synthesized image based on the text attention output and the image attention output. For example, the synthesized image includes one or more elements from the text prompt. The synthesized image includes a dog and a cat playing outside a building. Image generation apparatus 110 returns the synthesized image to user 100 via cloud 115 and user device 105.”; and [0098], “Text-to-image generation models are trained with text embedding though image embeddings are generally more informative. Image generation model 900 uses text embedding 940 and image embedding 935. To improve multi-modal conditioning ability of the model and to improve controllability and capacity of the model, embodiments of the present disclosure provide a separate learnable way to encode text and image that the model attends to equally and independently and then compose the output information”; Note that the combined identity representations is mapped to the binding information to create personalization embeddings).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify He to have based on the binding information to create personalization embeddings as taught by Ravi in order to improve accuracy and control by having separate learnable cross-attention layers for text embedding and image embedding and the model processes the text embedding and the image embedding in parallel (i.e., different from concatenating different types of embeddings into a single long sequence of embeddings) (See Ravi: Fig. 1, and [0024], “The present disclosure describes systems and methods that improve on conventional image generation models by providing more accurate depiction of image-related attributes in synthesized images. For example, generated images can include attributes and elements depicted in a reference image. That is, users can achieve more precise control over information represented in an image embedding of the reference image compared to conventional generative models. Embodiments achieve this improved accuracy and control by having separate learnable cross-attention layers for text embedding and image embedding and the model processes the text embedding and the image embedding in parallel (i.e., different from concatenating different types of embeddings into a single long sequence of embeddings). In some cases, the text attention output and the image attention output are generated in separate pathways”). He teaches a method and system that may receive text prompt and reference image, embed identity feature vis encoder, use a generation model to generate video that preserves identity with natural motion variation; while Ravi teaches a system and method that may effectively inject both text embeddings and image/reference embeddings into a diffusion model so that the diffusion model can accurately follow the text prompt while faithfully incorporate visual attributes from the reference image to generate personalized images. Therefore, it is obvious for one of ordinary skill in the art to modify He by Ravi to use the cross-attention based subject maps and feature blending mechanisms for personalized image generation. The motivation to modify He by Ravi is “Use of known technique to improve similar devices (methods, or products) in the same way”.
However, He, modified by Daha and Ravi, fails to explicitly disclose that to generate a personalized video that preserves identity of the corresponding subjects in the reference images while allowing for natural motion.
However, Choi teaches that to generate a personalized video that preserves identity of the corresponding subjects in the reference images while allowing for natural motion (See Choi: Fig. 5, and [0068], “In some aspects, the spatial portion of the generative artificial intelligence model is configured to customize an appearance of the subject of the video output independently of motion performed by the subject of the video output”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify He to have to generate a personalized video that preserves identity of the corresponding subjects in the reference images while allowing for natural motion as taught by Choi in order to improve or enable video generation based on desired appearances and movement, uses the motion customization by personalizing text-to-video generation models using reference videos to enhance user control over video content to allow the specification of desired motions through video inputs (See Choi: Fig. 1, and [0005], “To allow for generative artificial intelligence models to be fine-tuned or modified, smaller model adapters may be trained for large models. For example, adapters may be trained to improve or enable video generation based on desired appearances, movement, and the like. More generally, an adapter may allow for a machine learning model to be trained to perform tasks for which the model was not originally trained without retraining the model itself”). He teaches a method and system that may receive text prompt and reference image, embed identity feature vis encoder, use a generation model to generate video that preserves identity with natural motion variation; while Choi teaches a system and method that may combine multiple identity representation into a single combined identity representation associated with the subject and feed this combined representation to the diffusion model to generate the personalized images (video). Therefore, it is obvious for one of ordinary skill in the art to modify He by Choi to feed multiple identity representation into the image generation model to generate personalized images that allow for natural motion. The motivation to modify He by Choi is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 2, He, Daha, Ravi and Choi teach all the features with respect to claim 1 as outlined above. Further, Ravi teaches that the system of claim 1, wherein the diffusion model comprises multiple cross-attention layers and a self-attention layer (See Ravi: Fig. 9, and [0101], “In some embodiments, image encoder 920 separately encodes image prompt 910 to obtain preliminary image encoding 925. For example, preliminary image encoding 925 has a dimension of 1×1024. Preliminary image encoding 925 is projected to 8 tokens using a learnable image projector 930. The output from image projector 930 has a dimension of 8×1024. Separate learnable cross-attention layers for text and image are applied onto the embeddings separately in parallel. Image generation model 900 generates cross-attention output on text and image. The cross-attention outputs for text and image are then combined together. Detail regarding the process of using cross-attention for text and image is described in FIG. 10”. Note that in multi cross-attention layer diffusion model, the self-attention layer is standard inherent component of the diffusion backbones in the U-net or transformer).
Regarding claim 14, He, Daha, Ravi and Choi teach all the features with respect to claim 1 as outlined above. Further, Ravi teaches that the system of claim 1, the operations comprising:
processing the personalization embeddings through a first cross-attention layer from the text embeddings that are processed through a second cross-attention layer (See Ravi: Figs. 7-8, and [0057], At operation 705, the system encodes a text prompt to obtain a text embedding. In some cases, the operations of this step refer to, or may be performed by, a text encoder as described with reference to FIGS. 8 and 9”; [0061], “At operation 710, the system encodes an image prompt to obtain an image embedding. In some cases, the operations of this step refer to, or may be performed by, an image encoder as described with reference to FIGS. 8 and 9. In some examples, an image prompt refers to an image provided to a machine learning model, typically in a digital image format, to guide the generation of a synthetic image that matches or resembles certain attributes, styles, or elements from the image prompt. Image prompts are used in text-to-image generation models”; [0064], “At operation 715, the system performs, using a text attention layer of an image generation model, cross-attention on the text embedding to obtain a text attention output. In some cases, the operations of this step refer to, or may be performed by, an image generation model as described with reference to FIGS. 8-10”; and [0066], “At operation 720, the system performs, using an image attention layer of the image generation model, cross-attention on the image embedding to obtain an image attention output. In some cases, the operations of this step refer to, or may be performed by, an image generation model as described with reference to FIGS. 8-10”); and
combining outputs of the first and second cross-attention layers to condition the diffusion model (See Ravi: Figs. 10, and [0104], FIG. 10 shows an example of an image generation model 1000 according to aspects of the present disclosure. The example shown includes image generation model 1000, noise input 1005, text embedding 1010, image embedding 1015, noise encoder 1020, text attention layer 1025, image attention layer 1030, text attention output 1035, image attention output 1040, diffusion model 1045, output latent 1050, latent-to-pixel decoder 1055, and synthesized image 1060. Image generation model 1000 is an example of, or includes aspects of, the corresponding element described with reference to FIGS. 8 and 9. In some cases, text embedding 1010 and image embedding 1015 correspond to text embedding 940 and image embedding 935, in FIG. 9, respectively”).
Regarding claim 19, He, Daha, Ravi and Choi teach all the features with respect to claim 1 as outlined above. Further, He, Daha, Ravi and Choi teach that a computer-implemented method (See He: Figs. 1-2, and [0062], “FIG. 2 illustrates a block diagram of an exemplary hardware/software architecture of a communication device such as, for example, user equipment (UE) 30, in accordance with various aspects of the present disclosure. In some exemplary respects, the UE 30 may be any of communication devices 105, 110, 115, 120. In some exemplary aspects, the UE 30 may be a computer system such as, for example, a desktop computer, notebook or laptop computer, netbook, a tablet computer (e.g., a smart tablet), e-book reader, GPS device, camera, personal digital assistant, handheld electronic device, cellular telephone, smartphone, smart glasses, augmented/virtual reality device, smart watch, charging case, or any other suitable electronic device. As shown in FIG. 2, the UE 30 (also referred to herein as node 30) may include a processor 32, non-removable memory 44, removable memory 46, a speaker/microphone 38, a display, touchpad, and/or user interface(s) 42, a power source 48, a GPS chipset 50, and other peripherals 52. In some exemplary aspects, the display, touchpad, and/or user interface(s) 42 may be referred to herein as display/touchpad/user interface(s) 42. The display/touchpad/user interface(s) 42 may include a user interface capable of presenting one or more content items and/or capturing input of one or more user interactions/actions associated with the user interface. The power source 48 may be capable of receiving electric power for supplying electric power to the UE 30. For example, the power source 48 may include an alternating current to direct current (AC-to-DC) converter allowing the power source 48 to be connected/plugged to an AC electrical receptacle and/or Universal Serial Bus (USB) port for receiving electric power. The UE 30 may also include a camera 54. In an exemplary embodiment, the camera 54 may be a smart camera configured to sense images/video appearing within one or more bounding boxes. The UE 30 may also include communication circuitry, such as a transceiver 34 and a transmit/receive element 36. It will be appreciated the UE 30 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment”) comprising:
receiving a text prompt describing a video to be generated, a plurality of reference images each depicting a corresponding subject (See He: Figs. 1-2, and [0112], “The user interface component 402 enables interactions between a user of a user system 102 and the image generation system 230. The user interface component 402 is configured to receive user inputs, such as image uploads, text prompts, and selections of desired image attributes. The user interface component 402 facilitates navigation and operation of image generation system 230, providing tools and options, for example via the interaction client 104, that are useful for running or customizing the image generation process”; and [0127], “In some examples in the present disclosure, including those described with reference to FIG. 5, FIG. 6, and FIG. 11, a diffusion model is employed by the image generation system 230 to generate images. A diffusion model is a type of generative machine learning model that can be used to generate images conditioned on one or more inputs. It is based on the concept of “diffusing” noise throughout an image to transform it gradually into a new image. A diffusion model may use a sequence of invertible transformations to transform a random noise image into a final image. During training, a diffusion model may learn sequences of transformations that can best transform random noise images into desired output images. A diffusion model can be fed with input data (e.g., text describing the desired images) and the corresponding output images, and the parameters of the model are adjusted iteratively to improve its ability to generate accurate or good quality images”), and binding information that associates each reference image with a corresponding entity word in the text prompt(See Daha: Figs. 2A-B, and [0046], “The client application 214 can generate the arbitrary text string or identifier [V] that will be associated with the object of the object image 104 or the background of the background image 105. As described below, the client application 214 can also submit the identifier [V] with the respective image(s) of the object or background to the 3D image generating AI model 107 to fine-tune the model 107, so that the output 3D image 108 is true in appearance to the specific object and background desired by the user 212. Specifically, the data for the image of the object and/or the background is implanted in the output domain of the image generating AI model 107 and associated with a unique identifier. Otherwise, the object and background would be generated according to the entire class of such objects/backgrounds in the training set of the AI model 107 and would not be specific to the inputs of the user 212”; and [005], “As noted above, the disclosed technique can fine-tune an AI model with input images and text cues that contain a unique identifier followed by the category name of the object (e.g., “A [V] hat”), where [V] is the identifier or token associated with image data representing a enables the model to use its prior knowledge of the object's category, while instances of the particular category, i.e., the specific hat that the user has imaged, are bound to the unique identifier. Thus, the model can produce the desired 3D image of the object with the specified background, the 3D image being true to the appearance of the specified object and background”; and [0055], “In the case of either or both of the object and background, a special identifier is generated, represented herein as [V]. As noted above, this identifier or unique identifier can be an arbitrary text string that would not typically occur in any natural language communication. This “rare text” identifier will then represent the corresponding object or background in the lexicon of the image generating AI model. For example, [V] is a special text identifier in the system, such as the string ‘sks.’ If the object is a hat, a text instruction invoking the specific hat associated with this identifier could be “a photo of sks hat.” The AI model will then output an image, not just of any hat, but an image of the specific hat associated with the identifier “sks.” Again, the identifier could be any text string, arbitrarily, such as a very rare string or word that would not be used in a common sentence”. Note that the image bound to the identifier (the rare text) is mapped to binding information that associates each reference image with a corresponding entity word in the text prompt);
extracting features from each reference image and fusing the extracted features with text embeddings of the corresponding entity word (See He: Figs. 1-6, and [0153], “By processing the combined identity representation 604 and the text prompt representation 606 via separate layers of the diffusion model 602 (instead of, for example, concatenating them and embedding them in the same layers), the accuracy and fidelity of a subject's portrayal (as captured in the combined identity representation 604) can be better maintained. Accordingly, the cross-attention mechanism depicted in FIG. 6 can be classified as “decoupled” since it separates cross-attention layers for text-based features (e.g., text prompts) and image-based features (e.g., image prompts as merged into a combined representation)”; [0157], “The method 700 commences at opening loop operation 702, and proceeds to operation 704 in which the image generation system 230 accesses a plurality of input images (e.g., via the input image processing component 404). The input images are provided by a user via the interaction client 104 (as an example of an interaction application), and all depict a subject (e.g., the user or another person). For example, the user uses their user system 102 (e.g., the mobile device 114 or some other user device) to upload three input images via an image upload user interface provided by the interaction client 104. The image upload user interface can be provided as part of an onboarding process related to a personalized image generation feature”; and [0176], “In some examples, the image generation system 230 processes each input image submitted by the user using computer vision techniques to determine (e.g., via the input image processing component 404) whether the input image is suitable for use in generating a combined identity representation. For example, the image generation system 230 checks whether the input image depicts the relevant body part (e.g., the face), whether the lighting is suitable for feature extraction, or, where relevant, whether the image depicts the subject in a requested pose, with a requested facial expression, or from a requested angle. In some examples, the image generation system 230 automatically rejects an unsuitable or sub-optimal image and causes the user to receive a message, via the interaction client 104, indicating that they should upload a new image”) based on the binding information to create personalization embeddings (See Ravi: Figs. 1-3, and [0024, “The present disclosure describes systems and methods that improve on conventional image generation models by providing more accurate depiction of image-related attributes in synthesized images. For example, generated images can include attributes and elements depicted in a reference image. That is, users can achieve more precise control over information represented in an image embedding of the reference image compared to conventional generative models. Embodiments achieve this improved accuracy and control by having separate learnable cross-attention layers for text embedding and image embedding and the model processes the text embedding and the image embedding in parallel (i.e., different from concatenating different types of embeddings into a single long sequence of embeddings). In some cases, the text attention output and the image attention output are generated in separate pathways”: [0028], “In some examples, image generation apparatus 110 encodes the text prompt to obtain a text embedding and encodes the image prompt to obtain an image embedding. Image generation apparatus 110 performs, using a text attention layer of an image generation model, cross-attention on the text embedding to obtain a text attention output. Image generation apparatus 110 performs, using an image attention layer of the image generation model, cross-attention on the image embedding to obtain an image attention output. Image generation apparatus 110 generates, using a generator network of the image generation model, a synthesized image based on the text attention output and the image attention output. For example, the synthesized image includes one or more elements from the text prompt. The synthesized image includes a dog and a cat playing outside a building. Image generation apparatus 110 returns the synthesized image to user 100 via cloud 115 and user device 105.”; and [0098], “Text-to-image generation models are trained with text embedding though image embeddings are generally more informative. Image generation model 900 uses text embedding 940 and image embedding 935. To improve multi-modal conditioning ability of the model and to improve controllability and capacity of the model, embodiments of the present disclosure provide a separate learnable way to encode text and image that the model attends to equally and independently and then compose the output information”; Note that the combined identity representations is mapped to the binding information to create personalization embeddings);
generating an initial noisy video (See He: Figs. 1-6, and [0035], “Techniques described herein can be used to generate individual images or video content. In some examples, the personalized output image is one of a plurality of frames of a personalized video. For example, a personalized video is generated for the user, via the interaction application, based on the combined identity representation and an additional image generation control, such as a text prompt. The personalized video comprises multiple frames that depict a person resembling the subject, based on the combined identity representation”; and [0088], “The artificial intelligence and machine learning system 232 may also provide chatbot functionality to interactions 120 between user systems 102 and between a user system 102 and the interaction server system 110. The artificial intelligence and machine learning system 232 may work with the audio communication system 216 to provide speech recognition and natural language processing capabilities, allowing users to interact with the interaction system 100 using voice commands. The artificial intelligence and machine learning system 232 may also provide or facilitate generative AI functionality, e.g., allowing a user to generate text, image, or video content based on prompts and/or other instructions. In some examples, the artificial intelligence and machine learning system 232 provide a generative AI assistant that can answer questions provided by the user or otherwise help the user to learn about topics or obtain useful information”); and
iteratively denoising the noisy video using a diffusion model conditioned on both the text prompt and the personalization embeddings (See He: Figs. 1-6, and [0151], “One approach to enabling the diffusion model 602 to handle both image-based features (e.g., the combined identity representation 604) and text-based features (e.g., the text prompt representation 606) is to concatenate these features into the same cross-attention layers of the diffusion model 602. However, this may prevent the diffusion model 602 from capturing fine-grained features from input images. Instead, in some examples, the diffusion model 602 is adapted from a pre-trained model by embedding the image-based features via separate cross-attention layers that are different than the cross-attention layers handling the text-based features”; [0152], “In some examples, the diffusion model 602 is obtained by taking a pre-trained diffusion model with cross-attention layers that handle the text-based features, and adding components in the form of new cross-attention layers for the image-based features. In some examples, a new cross-attention layer is added for each cross-attention layer in the original denoising network 608 (e.g., U-Net model component)”; and [0127], “In some examples in the present disclosure, including those described with reference to FIG. 5, FIG. 6, and FIG. 11, a diffusion model is employed by the image generation system 230 to generate images. A diffusion model is a type of generative machine learning model that can be used to generate images conditioned on one or more inputs. It is based on the concept of “diffusing” noise throughout an image to transform it gradually into a new image. A diffusion model may use a sequence of invertible transformations to transform a random noise image into a final image. During training, a diffusion model may learn sequences of transformations that can best transform random noise images into desired output images. A diffusion model can be fed with input data (e.g., text describing the desired images) and the corresponding output images, and the parameters of the model are adjusted iteratively to improve its ability to generate accurate or good quality images”) to generate a personalized video that preserves identity of the corresponding subjects in the reference images while allowing for natural motion (See Choi: Fig. 5, and [0068], “In some aspects, the spatial portion of the generative artificial intelligence model is configured to customize an appearance of the subject of the video output independently of motion performed by the subject of the video output”) and pose variations (See He: Figs. 1-3, and [0001], “Cross-paired data may include both real and synthetic examples. Real cross-paired data may be sourced from videos containing multiple views of the same person. Synthetic cross-paired data may be generated using a personalization image generation model to create novel reference images from a base frame, varying properties such as facial expression, head pose, or lighting. Generated synthetic reference images may be filtered based on similarity thresholds to ensure identity consistency”).
Regarding claim 20, He, Daha, Ravi and Choi teach all the features with respect to claim 1 as outlined above. Further, He, Daha, Ravi and Choi teach that a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations (See He: Figs. 1-2, and [0062], “FIG. 2 illustrates a block diagram of an exemplary hardware/software architecture of a communication device such as, for example, user equipment (UE) 30, in accordance with various aspects of the present disclosure. In some exemplary respects, the UE 30 may be any of communication devices 105, 110, 115, 120. In some exemplary aspects, the UE 30 may be a computer system such as, for example, a desktop computer, notebook or laptop computer, netbook, a tablet computer (e.g., a smart tablet), e-book reader, GPS device, camera, personal digital assistant, handheld electronic device, cellular telephone, smartphone, smart glasses, augmented/virtual reality device, smart watch, charging case, or any other suitable electronic device. As shown in FIG. 2, the UE 30 (also referred to herein as node 30) may include a processor 32, non-removable memory 44, removable memory 46, a speaker/microphone 38, a display, touchpad, and/or user interface(s) 42, a power source 48, a GPS chipset 50, and other peripherals 52. In some exemplary aspects, the display, touchpad, and/or user interface(s) 42 may be referred to herein as display/touchpad/user interface(s) 42. The display/touchpad/user interface(s) 42 may include a user interface capable of presenting one or more content items and/or capturing input of one or more user interactions/actions associated with the user interface. The power source 48 may be capable of receiving electric power for supplying electric power to the UE 30. For example, the power source 48 may include an alternating current to direct current (AC-to-DC) converter allowing the power source 48 to be connected/plugged to an AC electrical receptacle and/or Universal Serial Bus (USB) port for receiving electric power. The UE 30 may also include a camera 54. In an exemplary embodiment, the camera 54 may be a smart camera configured to sense images/video appearing within one or more bounding boxes. The UE 30 may also include communication circuitry, such as a transceiver 34 and a transmit/receive element 36. It will be appreciated the UE 30 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment”) comprising:
receiving a text prompt describing a video to be generated, a plurality of reference images each depicting a corresponding subject (See He: Figs. 1-2, and [0112], “The user interface component 402 enables interactions between a user of a user system 102 and the image generation system 230. The user interface component 402 is configured to receive user inputs, such as image uploads, text prompts, and selections of desired image attributes. The user interface component 402 facilitates navigation and operation of image generation system 230, providing tools and options, for example via the interaction client 104, that are useful for running or customizing the image generation process”; and [0127], “In some examples in the present disclosure, including those described with reference to FIG. 5, FIG. 6, and FIG. 11, a diffusion model is employed by the image generation system 230 to generate images. A diffusion model is a type of generative machine learning model that can be used to generate images conditioned on one or more inputs. It is based on the concept of “diffusing” noise throughout an image to transform it gradually into a new image. A diffusion model may use a sequence of invertible transformations to transform a random noise image into a final image. During training, a diffusion model may learn sequences of transformations that can best transform random noise images into desired output images. A diffusion model can be fed with input data (e.g., text describing the desired images) and the corresponding output images, and the parameters of the model are adjusted iteratively to improve its ability to generate accurate or good quality images”), and binding information that associates each reference image with a corresponding entity word in the text prompt (See Daha: Figs. 2A-B, and [0046], “The client application 214 can generate the arbitrary text string or identifier [V] that will be associated with the object of the object image 104 or the background of the background image 105. As described below, the client application 214 can also submit the identifier [V] with the respective image(s) of the object or background to the 3D image generating AI model 107 to fine-tune the model 107, so that the output 3D image 108 is true in appearance to the specific object and background desired by the user 212. Specifically, the data for the image of the object and/or the background is implanted in the output domain of the image generating AI model 107 and associated with a unique identifier. Otherwise, the object and background would be generated according to the entire class of such objects/backgrounds in the training set of the AI model 107 and would not be specific to the inputs of the user 212”; and [005], “As noted above, the disclosed technique can fine-tune an AI model with input images and text cues that contain a unique identifier followed by the category name of the object (e.g., “A [V] hat”), where [V] is the identifier or token associated with image data representing a enables the model to use its prior knowledge of the object's category, while instances of the particular category, i.e., the specific hat that the user has imaged, are bound to the unique identifier. Thus, the model can produce the desired 3D image of the object with the specified background, the 3D image being true to the appearance of the specified object and background”; and [0055], “In the case of either or both of the object and background, a special identifier is generated, represented herein as [V]. As noted above, this identifier or unique identifier can be an arbitrary text string that would not typically occur in any natural language communication. This “rare text” identifier will then represent the corresponding object or background in the lexicon of the image generating AI model. For example, [V] is a special text identifier in the system, such as the string ‘sks.’ If the object is a hat, a text instruction invoking the specific hat associated with this identifier could be “a photo of sks hat.” The AI model will then output an image, not just of any hat, but an image of the specific hat associated with the identifier “sks.” Again, the identifier could be any text string, arbitrarily, such as a very rare string or word that would not be used in a common sentence”. Note that the image bound to the identifier (the rare text) is mapped to binding information that associates each reference image with a corresponding entity word in the text prompt);
extracting features from each reference image and fusing the extracted features with text embeddings of the corresponding entity word (See He: Figs. 1-6, and [0153], “By processing the combined identity representation 604 and the text prompt representation 606 via separate layers of the diffusion model 602 (instead of, for example, concatenating them and embedding them in the same layers), the accuracy and fidelity of a subject's portrayal (as captured in the combined identity representation 604) can be better maintained. Accordingly, the cross-attention mechanism depicted in FIG. 6 can be classified as “decoupled” since it separates cross-attention layers for text-based features (e.g., text prompts) and image-based features (e.g., image prompts as merged into a combined representation)”; [0157], “The method 700 commences at opening loop operation 702, and proceeds to operation 704 in which the image generation system 230 accesses a plurality of input images (e.g., via the input image processing component 404). The input images are provided by a user via the interaction client 104 (as an example of an interaction application), and all depict a subject (e.g., the user or another person). For example, the user uses their user system 102 (e.g., the mobile device 114 or some other user device) to upload three input images via an image upload user interface provided by the interaction client 104. The image upload user interface can be provided as part of an onboarding process related to a personalized image generation feature”; and [0176], “In some examples, the image generation system 230 processes each input image submitted by the user using computer vision techniques to determine (e.g., via the input image processing component 404) whether the input image is suitable for use in generating a combined identity representation. For example, the image generation system 230 checks whether the input image depicts the relevant body part (e.g., the face), whether the lighting is suitable for feature extraction, or, where relevant, whether the image depicts the subject in a requested pose, with a requested facial expression, or from a requested angle. In some examples, the image generation system 230 automatically rejects an unsuitable or sub-optimal image and causes the user to receive a message, via the interaction client 104, indicating that they should upload a new image”) based on the binding information to create personalization embeddings (See Ravi: Figs. 1-3, and [0024, “The present disclosure describes systems and methods that improve on conventional image generation models by providing more accurate depiction of image-related attributes in synthesized images. For example, generated images can include attributes and elements depicted in a reference image. That is, users can achieve more precise control over information represented in an image embedding of the reference image compared to conventional generative models. Embodiments achieve this improved accuracy and control by having separate learnable cross-attention layers for text embedding and image embedding and the model processes the text embedding and the image embedding in parallel (i.e., different from concatenating different types of embeddings into a single long sequence of embeddings). In some cases, the text attention output and the image attention output are generated in separate pathways”: [0028], “In some examples, image generation apparatus 110 encodes the text prompt to obtain a text embedding and encodes the image prompt to obtain an image embedding. Image generation apparatus 110 performs, using a text attention layer of an image generation model, cross-attention on the text embedding to obtain a text attention output. Image generation apparatus 110 performs, using an image attention layer of the image generation model, cross-attention on the image embedding to obtain an image attention output. Image generation apparatus 110 generates, using a generator network of the image generation model, a synthesized image based on the text attention output and the image attention output. For example, the synthesized image includes one or more elements from the text prompt. The synthesized image includes a dog and a cat playing outside a building. Image generation apparatus 110 returns the synthesized image to user 100 via cloud 115 and user device 105.”; and [0098], “Text-to-image generation models are trained with text embedding though image embeddings are generally more informative. Image generation model 900 uses text embedding 940 and image embedding 935. To improve multi-modal conditioning ability of the model and to improve controllability and capacity of the model, embodiments of the present disclosure provide a separate learnable way to encode text and image that the model attends to equally and independently and then compose the output information”; Note that the combined identity representations is mapped to the binding information to create personalization embeddings);
generating an initial noisy video (See He: Figs. 1-6, and [0035], “Techniques described herein can be used to generate individual images or video content. In some examples, the personalized output image is one of a plurality of frames of a personalized video. For example, a personalized video is generated for the user, via the interaction application, based on the combined identity representation and an additional image generation control, such as a text prompt. The personalized video comprises multiple frames that depict a person resembling the subject, based on the combined identity representation”; and [0088], “The artificial intelligence and machine learning system 232 may also provide chatbot functionality to interactions 120 between user systems 102 and between a user system 102 and the interaction server system 110. The artificial intelligence and machine learning system 232 may work with the audio communication system 216 to provide speech recognition and natural language processing capabilities, allowing users to interact with the interaction system 100 using voice commands. The artificial intelligence and machine learning system 232 may also provide or facilitate generative AI functionality, e.g., allowing a user to generate text, image, or video content based on prompts and/or other instructions. In some examples, the artificial intelligence and machine learning system 232 provide a generative AI assistant that can answer questions provided by the user or otherwise help the user to learn about topics or obtain useful information”); and
iteratively denoising the noisy video using a diffusion model conditioned on both the text prompt and the personalization embeddings(See He: Figs. 1-6, and [0151], “One approach to enabling the diffusion model 602 to handle both image-based features (e.g., the combined identity representation 604) and text-based features (e.g., the text prompt representation 606) is to concatenate these features into the same cross-attention layers of the diffusion model 602. However, this may prevent the diffusion model 602 from capturing fine-grained features from input images. Instead, in some examples, the diffusion model 602 is adapted from a pre-trained model by embedding the image-based features via separate cross-attention layers that are different than the cross-attention layers handling the text-based features”; [0152], “In some examples, the diffusion model 602 is obtained by taking a pre-trained diffusion model with cross-attention layers that handle the text-based features, and adding components in the form of new cross-attention layers for the image-based features. In some examples, a new cross-attention layer is added for each cross-attention layer in the original denoising network 608 (e.g., U-Net model component)”; and [0127], “In some examples in the present disclosure, including those described with reference to FIG. 5, FIG. 6, and FIG. 11, a diffusion model is employed by the image generation system 230 to generate images. A diffusion model is a type of generative machine learning model that can be used to generate images conditioned on one or more inputs. It is based on the concept of “diffusing” noise throughout an image to transform it gradually into a new image. A diffusion model may use a sequence of invertible transformations to transform a random noise image into a final image. During training, a diffusion model may learn sequences of transformations that can best transform random noise images into desired output images. A diffusion model can be fed with input data (e.g., text describing the desired images) and the corresponding output images, and the parameters of the model are adjusted iteratively to improve its ability to generate accurate or good quality images”) to generate a personalized video that preserves identity of the corresponding subjects in the reference images while allowing for natural motion (See Choi: Fig. 5, and [0068], “In some aspects, the spatial portion of the generative artificial intelligence model is configured to customize an appearance of the subject of the video output independently of motion performed by the subject of the video output”) and pose variations (See He: Figs. 1-3, and [0001], “Cross-paired data may include both real and synthetic examples. Real cross-paired data may be sourced from videos containing multiple views of the same person. Synthetic cross-paired data may be generated using a personalization image generation model to create novel reference images from a base frame, varying properties such as facial expression, head pose, or lighting. Generated synthetic reference images may be filtered based on similarity thresholds to ensure identity consistency”).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over He, etc. (US 20260099978 A1) in view of Daha. etc. (US 20240386661 A1) further in view of Ravi, etc. (US 20250117972 A1), Choi, etc. (US 20250356171 A1) and Karras, etc. (US 20210383241 A1).
Regarding claim 5, He, Daha, Ravi and Choi teach all the features with respect to claim 1 as outlined above. However, He, modified by Daha, Ravi and Choi, fails to explicitly disclose that the system of claim 1, the operations comprising: applying data augmentation to the reference images before extracting the features to prevent the diffusion model from overfitting to unintended properties of the reference images.
However, Karras teaches that the system of claim 1, the operations comprising: applying data augmentation to the reference images before extracting the features to prevent the diffusion model from overfitting to unintended properties of the reference images (See Karras: Figs. 2A-E, and [0050], “At step 260, at least one augmentation is applied to the input data to produce augmented input data associated with a second distribution, where an augmentation operator corresponding to a transformation from the first distribution to the second distribution is invertible and specifies the at least one augmentation. In an embodiment, the input data is output data produced by the generator 100. In an embodiment, the first distribution is the only such distribution for which applying the at least one augmentation to the input data associated with the first distribution results in an augmented distribution that matches the second distribution”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify He to have the system of claim 1, the operations comprising: applying data augmentation to the reference images before extracting the features to prevent the diffusion model from overfitting to unintended properties of the reference images as taught by Karras in order to prevent overfitting of the discriminator while ensuring that the augmentations do not leak to generated data (See Karras: Fig. 1, and [0005], “To allow for generative artificial intelligence models to be fine-tuned or modified, smaller model adapters may be trained for large models. For example, adapters may be trained to improve or enable video generation based on desired appearances, movement, and the like. More generally, an adapter may allow for a machine learning model to be trained to perform tasks for which the model was not originally trained without retraining the model itself”). He teaches a method and system that may receive text prompt and reference image, embed identity feature vis encoder, use a generation model to generate video that preserves identity with natural motion variation; while Karras teaches a system and method that may augment the input data before feature extraction to prevent overfitting. Therefore, it is obvious for one of ordinary skill in the art to modify He by Karras to augment the reference image before feature extraction to prevent overfitting. The motivation to modify He by Karras is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Claims 6 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over He, etc. (US 20260099978 A1) in view of Daha. etc. (US 20240386661 A1) further in view of Ravi, etc. (US 20250117972 A1), Choi, etc. (US 20250356171 A1). Karras, etc. (US 20210383241 A1) and Yu, etc. (US 20250111552 A1).
Regarding claim 6, He, Daha, Ravi and Choi teach all the features with respect to claim 1 as outlined above. However, He, modified by Daha, Ravi, Choi and Karras, fails to explicitly disclose that the system of claim 1, the operations comprising: encoding the initial noisy video into a set of video tokens using a video encoder; and decoding a de-noised set of video tokens back into a video using a video decoder.
However, Yu teaches that the system of claim 1, the operations comprising: encoding the initial noisy video into a set of video tokens using a video encoder (See Yu: Figs. 1A-B, and [0030], “In other words, the CMD 10 includes an autoencoder 20 and two latent diffusion models 50 and 60. First, during the first phase 110, the autoencoder 20 is trained that encodes a video x.sup.1:L as a single content frame x and low-dimensional motion latent representation z. After that, learning the target distribution p.sub.data(x.sup.1:L|c) may be defined as learning the following distribution: p(x|z|c)=p(z|x, c)p(x|c). Each distribution (e.g., p(z|x, c) and p(x|c)) may be modeled using two DMs (e.g., DMs 50 and 60), where a pretrained image diffusion model for learning the content frame distribution p(x|c) is utilized”); and
decoding a de-noised set of video tokens back into a video using a video decoder (See Yu: Figs. 2A-B, and [0038], “FIG. 2B illustrates a block diagram of an exemplary video decoder 40 of the CDM 10, in accordance with one or more embodiments of the present disclosure. For instance, FIG. 2B shows the operation 250 of the video encoder 40. In operation, the content frame 114 and the motion latent representation 116 (e.g., the first and second motion latent representations z.sub.x 252 and z.sub.y 254) are provided to linear projection blocks 256-260 to generate linear projections 262-266. As mentioned above, the linear projection blocks 256-260 may be MLPs that linearly project the inputs (e.g., the content frame 114 as well as the first and second motion latent representations z.sub.x 252 and z.sub.y 254). The repeat blocks 268-272 may be used to generate an additional dimension for each of the linear projections 262-266 such that all three projections have the same dimension to be summed together. For example, the content frame 114 includes the height and width dimensions, the first motion latent representation 252 includes the time dimension and the height dimension, and the second motion latent representation 254 includes the time dimension and the width dimension. To enable aggregation, the repeat blocks 268-272 may be used to transform the linear projections 262-266 back into four dimensions (including time, height and width and embedding length), and then the aggregation block 274 sums the outputs from the repeat blocks 268-272 (e.g., the three tensors from the repeat blocks 268-272) into a single tensor v. The single tensor v is then passed to the video network 276, which may be similar to the video network 202 of the encoder 30, to generate the output video 118”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify He to have the system of claim 1, the operations comprising: encoding the initial noisy video into a set of video tokens using a video encoder; and decoding a de-noised set of video tokens back into a video using a video decoder as taught by Yu in order to improve text-guided generation of high-resolution videos (See Yu: Figs. 1A-B, and [0006], “Therefore, embodiments of the present disclosure describe a system and method to improve text-guided generation of high-resolution videos by training DMs (e.g., a content frame DM and a motion DM) based on decomposing a video into a singular content frame and motion latent representations, and using the trained DMs to generate high-resolution videos during the inference phase. For instance, to address the aforementioned shortcomings described above, a content frame-motion latent diffusion model (CDM) may be and/or include memory and computation efficient latent video DMs that leverage visual knowledge present in pretrained image DMs”). He teaches a method and system that may receive text prompt and reference image, embed identity feature vis encoder, use a generation model to generate video that preserves identity with natural motion variation; while Yu teaches a system and method that may provide ordinary diffusion model implementation with video encoder to generate token/latent representation of the images, and video decoder architecture to reconstruct the image from the latent/token. Therefore, it is obvious for one of ordinary skill in the art to modify He by Yu to video encoder and decoder implementation of the diffusion model. The motivation to modify He by Yu is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 13, He, Daha, Ravi and Choi teach all the features with respect to claim 1 as outlined above. Further, Choi and Yu teaches that the system of claim 1, the operations comprising:
tokenizing the video into a sequence of 1-D video tokens (See Choi: Fig. 4, and [0056], “The adapted output of the temporal self-attention block 420 may be processed by the temporal feedforward network 424 to generate a frame in the video output. Generally, the temporal feedforward network 424 may project the adapted output of the temporal self-attention block 420 into a set of tokens or other data representing pixels or other portions of a frame. In some aspects, the projected set of tokens may be modified by the output of a temporal adapter 426 (e.g., a temporal LoRA adapter) and the cross-attention map 405 generated based on an output of the previous inferencing round to adjust the content rendered in the generated frame. The output of the temporal adapter 426 and the cross-attention map 405 may be combined (e.g., multiplicatively) to allow for the addition or other combination of adapter values to the output pixels/tokens/other data generated by the temporal feedforward network 424”); and
adding Gaussian noise to the video tokens to obtain the initial noisy video (See Yu: Fig. 1, and [0044], “After training the autoencoder 20, the second phase 130 is performed to train the content frame diffusion model 50. Initially, the content frame diffusion model 50 may be an already trained (e.g., pre-trained) image diffusion model such as a stable diffusion model. In the second phase 130, the content frame diffusion model 50 may be fine-tuned (e.g., additionally trained) to account for the content frame generation. For instance, given the condition-video pair dataset D that includes the video clip and condition c, the video clip may be provided to the already trained video encoder 30 (e.g., after completing the first phase 110) to generate a content frame 114. Further, the condition input 134 (e.g., the condition c from the same sample) and Gaussian noise 132 may be provided to the content frame diffusion model 50 to generate the content frame output 136. By comparing the generated content frame output 136 with the content frame 114 that is output from the video encoder 30, which was trained during the first phase 110, a standard diffusion loss (e.g., denoising objective) may be computed. The computed loss may be used to train/update/fine-tune the content frame diffusion model 50”).
Allowable Subject Matter
Claims 3-4 are objected to as being dependent upon a rejected base claim 2, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The best arts searched, over He, etc. (US 20260099978 A1), Daha. etc. (US 20240386661 A1), Ravi, etc. (US 20250349040 A1) and Choi, etc. (US 20250356171 A1), do not teach the cited limitations of “the system of claim 2, the operations comprising: encoding each of the plurality of reference images into image tokens; projecting fused embeddings of each reference image and its corresponding entity word; applying a residual connection with the image tokens for each reference image; adding a learnable image index embedding to separate tokens from different reference images, wherein tokens from the same reference image share the same image index embedding; and concatenating the personalization embeddings from all reference images to create a combined personalization embedding for conditioning the diffusion model.”
Claims 7-12 are objected to as being dependent upon a rejected base claim 1, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The best arts searched, over He, etc. (US 20260099978 A1), Daha. etc. (US 20240386661 A1), Ravi, etc. (US 20250349040 A1); Choi, etc. (US 20250356171 A1). Karras, etc. (US 20210383241 A1) and Yu, etc. (US 20250111552 A1)., do not teach the cited limitations of “the system of claim 1, wherein the diffusion model is trained to perform a set of training operations comprising: accessing a plurality of training data, the plurality of training data comprising a plurality of videos depicting multiple subjects and corresponding captions associated with the plurality of videos; selecting an individual video from the plurality of training data along with an individual caption of the corresponding captions associated with the individual video; retrieving entity words from the individual caption using a large language model (LLM), wherein the entity words include subject entities and background entities; selecting multiple frames from the individual video; extracting individual images for individual subjects and a background from the selected multiple frames; binding each extracted individual image with its corresponding entity word in the individual caption to create a training personalization embedding; generating a training video using the diffusion model conditioned on both the individual caption and the training personalization embedding; computing a deviation between the training video and the selected individual video; and updating parameters of self-attention layers and cross-attention layers of the diffusion model based on the computed deviation.”
Claims 15-18 are objected to as being dependent upon a rejected base claim 1, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The best arts searched, over He, etc. (US 20260099978 A1), Daha. etc. (US 20240386661 A1), Ravi, etc. (US 20250349040 A1); Choi, etc. (US 20250356171 A1). Karras, etc. (US 20210383241 A1) and Yu, etc. (US 20250111552 A1)., do not teach the cited limitations of “the system of claim 1, the operations comprising: evaluating the generated personalized video using a subject similarity metric that measures similarity between reference images and segmented subjects in the generated personalized video; and evaluating the generated personalized video using a dynamic degree metric that measures optical flow magnitude between consecutive generated video frames.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights