Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“an AI-assisted storyboarding module configured to”; “a 3D digitization module configured to”; “a virtual camera controller configured to”; “an AI animation module configured to”; “an AI-assisted compositing module configured to” in claim 1.
“the 3D digitization module is configured” in claim 2.
“a visual cue extraction module configured” in claim 3.
“the AI animation module is configured” in claim 4.
“the visual cue extraction module is configured to “ in claim 5.
“the AI-assisted compositing module is configured” in claim 6.
“the AI-assisted compositing module is configured” in claim 7.
“a post-production module configured to:”; “the AI-assisted compositing module” in claim 8.
“the post-production module is configured to”; “the AI-assisted compositing module” in claim 9.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
1. Claims 1-3, 7-12, 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Rivers, U.S Patent Application Publication No.20250218464 (“Rivers”) in view of Zhou et al. U.S Patent Application Publication No.20250392795 (“Zhou”)
Regarding independent claim 1, Rivers teaches a computer-implemented system for artificial intelligence (AI) assisted filmmaking (Advantages of the AI-Generative Video Generation System [0017] The system can also be configured for customization and scalability. In some embodiments, users are enabled to tailor the visual style, animations, and media format according to their preferences. This flexibility allows for a diverse range of applications, whether it involves adapting a novel into an animated film or creating educational videos based on textbooks”, [0024] FIG. 1 is an example operating environment of a system for transforming written text into animated scenes and videos, according to some embodiments.), the system comprising:
an AI-assisted storyboarding module configured to generate one or more images of a storyboard ([0050] Thereby, in some embodiments, rather than treating the entire text as a single entity, text processing module 105 dissects it into discrete scenes or segments. These scenes are defined by their content and context within the narrative. For example, in a novel, each scene could correspond to a specific chapter, a significant event, or a change in the story's location or perspective. In some embodiments, the system 101 can perform one or more automated processes for scene division and image prompt generation. Once the text is processed, text processing module 105 segments the narration into discrete scenes. Each scene's distinct narrative segment then undergoes analysis by the AI image generation endpoint 108, which generates contextually relevant image prompts. This process involves one or more AI algorithms that interpret narrative elements like mood, setting, and character actions to create visually coherent prompts. System 101 can be configured based on user or cohort customization or other considerations described herein to create accurate and diverse prompts for a wide array of narrative styles. Thus, system 101 is configured to automatically tailor each scene's visual representation to its narrative content. Embodiments described herein can thereby enhance the storytelling aspect of the videos while contributing to a more engaging and immersive viewer experience. System 101 can accurately transform textual nuances into visual cues.”; [0080] At operation 302, Text-to-Scene Transformation Module 203 processes the incoming text. This module identifies and categorizes elements such as setting, character interactions, and events. In an example, if the text describes a rainy evening in a city, the module identifies rain, the evening setting, city landscape, vehicles, and pedestrians.), one or more prompts associated with the storyboard ([0052] Image generation module 107 complements the text processing module by translating textual descriptions into visual representations by connecting operably to AI image generation endpoint 108. AI image generation endpoint 108 can include a generative AI tool to generate photo-realistic images, or other types of images, by inputting text, employing computer vision algorithms to create images that align with the narrative. In some embodiments, AI image generation endpoint 108 can be exemplified by Stability.AI, or similar AI image generation applications. These images can visually convey the content, providing a foundation for and enriching the video's overall presentation. In some embodiments, AI image generation endpoint 108 can generate unique images based on user-defined or AI-suggested text prompts, distinguishing each scene with original visual content and providing an alternative from reliance on stock imagery.”), and a guide based on a script ([0064] Users may interact directly with User Interface Module 202. For example, a user can upload text, choose from several templates, adjust settings according to their preferences, etc. User Interface Module 202 can be configured to accept diverse textual inputs, from book chapters to short scripts, and to integrate functionalities including integrated text editing, and direct linking to external content sources. In some embodiments, User Interface Module 202 can be operably connected to a Content Validation component, which can be a component of Security Module 210, or an element (not shown) external to Security Module 210. For example, a Content Validation component of Security Module 210 can be configured to ascertain the legality of the uploaded textual content, a crucial aspect of content creation and distribution. Content Validation component can be configured to cross-reference uploaded material against predefined legal parameters and standards. The Content Validation component ensures that the textual content uploaded by users adheres to certain terms and conditions of the system, and/or with broader legal requirements. It can be configured to perform validation to verify whether users have the necessary rights or permissions to use the content for video creation.”; [0111] Parameter Setting Module 502 interprets the text, and can be configured to employ one or more algorithms, for example NLP algorithm(s), to discern contextual information, emotions, undertones, and themes. PSM 502 receives segmented text from Input Module 501 and translates it into parameters that guide the AI-driven visual representation process. The module can be configured to incorporate a feedback loop, such as feedback and refinement module 507, for iterative refinement, providing that the translation from text to visual parameters aligns with the content's intent. PSM 502 enables users and/or the system to set specific parameters for the conversion process. These parameters determine how the AI will interpret the text content, taking into account context, emotions, and specific scenes described within the text.”);
a 3D digitization module configured to generate a 3D model for a scene based on the script ([0081] At operation 303, Scene-to-Image Module 204 creates visual representations of the identified scenes. It generates images that align accurately and contextually with the text. For example, text describing a beach will produce an image featuring the sea, sand, and beachgoers”);
an AI animation module configured to generate a first video based on the one or more images of the storyboard and the one or more prompts associated with the storyboard ([0082] At operation 304, Image-to-Animation Module 205 adds motion to the created images. It bases the animation on the narrative flow and context, ensuring logical progression from one scene to another in sync with the story”; [0123] The system also includes an integration module 506, which receives the visuals from the AI processing module 503, sequences them, and synchronizes them with the original audiobook or text-based content. This module may also add additional elements, such as audio, including sound effects, and/or background music, for example, to enhance the final video. [0124] The output module 507 provides the final video or animation. This video combines the visuals generated from the text with the original audio, offering a multimedia experience that visually represents the content of the original text.”);
and an AI-assisted compositing module configured to generate a composited video based on the first video generated by the AI animation module and a model(0123] The system also includes an integration module 506, which receives the visuals from the AI processing module 503, sequences them, and synchronizes them with the original audiobook or text-based content. This module may also add additional elements, such as audio, including sound effects, and/or background music, for example, to enhance the final video. [0124] The output module 507 provides the final video or animation. This video combines the visuals generated from the text with the original audio, offering a multimedia experience that visually represents the content of the original text.”) Rivers is understood to be silent on the remaining limitation of claim 1.
In the same field of endeavor, LI teaches AI-assisted module configured to generate a guide based on a script ([0080] The prompt formatting unit 302 receives a user subject request (including subject image(s)) and a style request from the request processing unit 122, then generates a subject meta prompt and a style meta prompt respectively for the LMM 126a. The LMM 126a generates a subject textual description of the subject image(s) using the subject meta prompt and a style textual description of the style request using the style meta prompt respectively, and then rewrites the textual descriptions into a textual prompt for the LVM 126b to generate an avatar with the desired style (e.g., the avatar with flat design style 154a in FIG. 1B)”.
a 3D digitization module configured to generate a 3D model for a scene based on the script and the guide ([0076] The second generative model (e.g., the LVM 126b) can be any text-to-image model trained to generate visual content (e.g., image, video, and the like) applying style(s) on subject(s) seamlessly in response to textual prompts. For instance, the second generative model may be CLIP, Vision Transformer (ViT), Megatron-Turing NLG, DALL-E, Imagen, GauGAN2, VQGAN+CLIP, or the like. In some implementations, the system may select a text-to-image model based on factors such as open source, photorealistic, creative control, computational requirements, case of use, licensing, and the like. The less sophisticated a text-to-image model, the more meta prompting and/or additional tools/models are required to provide the same quality avatars. In one embodiment, the first and second generative models are embodied in one generative model. [0083] The textual prompt (e.g., the meta prompt in Table 3) can include instructions that guide the LVM 126b to generate an avatar with desired style(s), and an optional background with object(s) if the user adda an object request. The system can instruct the generative model(s) 126 to generate a single-shot prompt (i.e., including a single example or instruction to guide the LMM's response) or a multi-shot prompt (i.e., including multiple examples or instructions to give the LMM more context and improve its understanding of the task) for generating the avatar.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of convert user-supplied textual content into animated videos of Rivers with generate model based on textual description of LI because this modification would generate an avatar with desired style(s), and an optional background with object(s) ([0083] of LI). Both Rivers and Li are understood to be silent on the remaining limitations of claim 1.
In the same field of endeavor, Zhou teaches a 3D digitization module configured to generate a 3D model for a scene based on the script ([0066] The inputs also include one or more reference images 516 and another reference caption 518. The reference images 516 depict animals dressed in different adult clothes. The accompanying reference caption 518 states: “Dressed animals.” A conventional image customization machine-learning model customized on the inputs 516, 518, and a portion of target text prompt 514 (e.g., goat wearing a suit) is to generate a customized image of the target object. The base diffusion model 202 is then customized with the trained motion characteristics module 206 and the customized target object to support both appearance and motion customization, generating target digital video 520. The target digital video 520 shows a goat in a suit on its hind legs as it dances and twirls on a farm.”)
virtual camera controller configured to generate a prompt associated with the 3D model for the scene for controlling virtual camera settings ([0061] FIG. 5 depicts multiple example implementations 500 to generate target digital videos using the video generation system as employing techniques described herein. The diffusion model 202 of the video generation system (e.g., the video generation service 116 of FIG. 1) receives as inputs the reference digital video 122 with the reference object 124 and the reference caption 126 as depicted in FIG. 2. As described above, the reference digital video 122 depicts a lady in a blue dress (e.g., the reference object 124) dancing and twirling (e.g., the reference movement). The reference caption 126 describes the reference movement of reference object 124 in the reference digital video 122 as: “Lady in a blue dress is dancing and twirling.” In the example implementations 500, the target text prompt 128 is replaced with different target text prompts to generate different target videos mimicking the dancing and twirling in the reference digital video 122, but with different objects, camera views, and/or backgrounds. [0063] In a second example implementation, multiple movements are customized into a target digital video. In addition to inputs 122, 124, and 126, a target text prompt 506 identifies a different environment, reference object, and camera view. The target text prompt 506 states: “Santa dancing, twirling in front of people with aerial camera over.” A first instance of the diffusion model 202 customized on the inputs 122, 126, and a first portion of target text prompt 506 (e.g., Santa dancing, twirling in front of people) generates a digital video of Santa dancing, twirling in front of people. [0064] The inputs also include another reference video 508 and another reference caption 510. The reference video 508 depicts an aerial camera flight over a herd of sheep in a meadow. The accompanying reference caption 510 states: “Aerial camera over a herd.” A second instance of the diffusion model 202 customized on the inputs 508, 510, and a second portion of target text prompt 506 (e.g., aerial camera over) to generate a digital video of a camera flying over a target object (e.g., Santa). The base diffusion model 202 is then customized with the two trained motion characteristics modules 206 from the above instances to integrate various movements (e.g., dancing, twirling, and aerial camera) into one target digital video 512.”);
an AI animation module configured to generate a first video based on the one or more images of the storyboard and the one or more prompts associated with the storyboard, as well as the prompt associated with the 3D model for the scene for controlling the virtual camera settings ([0061] FIG. 5 depicts multiple example implementations 500 to generate target digital videos using the video generation system as employing techniques described herein. The diffusion model 202 of the video generation system (e.g., the video generation service 116 of FIG. 1) receives as inputs the reference digital video 122 with the reference object 124 and the reference caption 126 as depicted in FIG. 2. As described above, the reference digital video 122 depicts a lady in a blue dress (e.g., the reference object 124) dancing and twirling (e.g., the reference movement). The reference caption 126 describes the reference movement of reference object 124 in the reference digital video 122 as: “Lady in a blue dress is dancing and twirling.” In the example implementations 500, the target text prompt 128 is replaced with different target text prompts to generate different target videos mimicking the dancing and twirling in the reference digital video 122, but with different objects, camera views, and/or backgrounds. [0063] In a second example implementation, multiple movements are customized into a target digital video. In addition to inputs 122, 124, and 126, a target text prompt 506 identifies a different environment, reference object, and camera view. The target text prompt 506 states: “Santa dancing, twirling in front of people with aerial camera over.” A first instance of the diffusion model 202 customized on the inputs 122, 126, and a first portion of target text prompt 506 (e.g., Santa dancing, twirling in front of people) generates a digital video of Santa dancing, twirling in front of people. [0064] The inputs also include another reference video 508 and another reference caption 510. The reference video 508 depicts an aerial camera flight over a herd of sheep in a meadow. The accompanying reference caption 510 states: “Aerial camera over a herd.” A second instance of the diffusion model 202 customized on the inputs 508, 510, and a second portion of target text prompt 506 (e.g., aerial camera over) to generate a digital video of a camera flying over a target object (e.g., Santa). The base diffusion model 202 is then customized with the two trained motion characteristics modules 206 from the above instances to integrate various movements (e.g., dancing, twirling, and aerial camera) into one target digital video 512.” )
and an AI-assisted compositing module configured to generate a composited video based on the first video generated by the AI animation module and a low-rank adaptation (LoRA) model ([0016] In particular, the described video generation service leverages low-rank adaptation (LoRA), which is an image customization technique, applied to a pre-trained T2V diffusion model to capture the motion signature in the reference digital video. In one aspect, LoRA is applied on temporal attentional layers to train the T2V diffusion model to the temporal motion dynamics of reference movements in reference digital videos. To disentangle the spatial and temporal information during the training pipeline, appearance absorbers detach the original appearance of the reference object from the respective reference digital video before motion learning. In other words, the appearance information is absorbed, leaving only the motion information for the LoRA. In a subsequent inference stage, the trained machine-learning model generates digital videos of desired target objects with the reference movement. The described staged pipeline also enables video generation adaptable to different subjects and scenes with both spatial and temporal varieties. [0021] Using the trained second aspect loaded therein, the machine-learning model generates the target digital video depicting the target object with the reference movement based on the target text prompt. For example, the dancing and twirling motion dynamics are used to generate a target digital video showing Ironman dancing and twirling in a similar manner. In particular, the appearance absorber is removed, and the trained temporal LoRA is loaded into the T2V diffusion model. Given the target text prompt describing a different subject (or scene), the trained T2V diffusion model generates the target digital video. The trained diffusion model accurately transfers the learned motion information to different objects (e.g., Ironman) and produces diverse motions regarding their intensities, positions, and camera views.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of convert user-supplied textual content into animated videos of Rivers and LI with using low-rank adaption as seen in Zhou because this modification would enable video generation adaptable to different subjects and scenes with both spatial and temporal varieties ([0016] of Zhou).
Thus, the combination of Rivers, LI and Zhou teaches a computer-implemented system for artificial intelligence (AI) assisted filmmaking, the system comprising: an AI-assisted storyboarding module configured to generate one or more images of a storyboard, one or more prompts associated with the storyboard, and a guide based on a script; a 3D digitization module configured to generate a 3D model for a scene based on the script and the guide; a virtual camera controller configured to generate a prompt associated with the 3D model for the scene for controlling virtual camera settings; an AI animation module configured to generate a first video based on the one or more images of the storyboard and the one or more prompts associated with the storyboard, as well as the prompt associated with the 3D model for the scene for controlling the virtual camera settings; and an AI-assisted compositing module configured to generate a composited video based on the first video generated by the AI animation module and a low-rank adaptation (LoRA) model.
Regarding claim 2, Rivers, LI and Zhou teach the system of claim 1, wherein the 3D digitization module is further configured to generate the LoRA model based on the script and the guide, wherein the 3D model for the scene represents a digitized 3D environment and the LoRA model represents a digitized character (see at least of [0081] Rivers “ At operation 303, Scene-to-Image Module 204 creates visual representations of the identified scenes. It generates images that align accurately and contextually with the text. For example, text describing a beach will produce an image featuring the sea, sand, and beachgoers.”; [0021] of Zhou “Using the trained second aspect loaded therein, the machine-learning model generates the target digital video depicting the target object with the reference movement based on the target text prompt. For example, the dancing and twirling motion dynamics are used to generate a target digital video showing Ironman dancing and twirling in a similar manner. In particular, the appearance absorber is removed, and the trained temporal LoRA is loaded into the T2V diffusion model. Given the target text prompt describing a different subject (or scene), the trained T2V diffusion model generates the target digital video. The trained diffusion model accurately transfers the learned motion information to different objects (e.g., Ironman) and produces diverse motions regarding their intensities, positions, and camera views.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 3, Rivers, LI and Zhou teach the system of claim 1, further comprising: a 2D camera capturing module configured to generate a second video based on the script and the guide and a visual cue extraction module configured to generate a prompt associated with the second video (see at least Rivers [0050] Thereby, in some embodiments, rather than treating the entire text as a single entity, text processing module 105 dissects it into discrete scenes or segments. These scenes are defined by their content and context within the narrative. For example, in a novel, each scene could correspond to a specific chapter, a significant event, or a change in the story's location or perspective. In some embodiments, the system 101 can perform one or more automated processes for scene division and image prompt generation. Once the text is processed, text processing module 105 segments the narration into discrete scenes. Each scene's distinct narrative segment then undergoes analysis by the AI image generation endpoint 108, which generates contextually relevant image prompts. This process involves one or more AI algorithms that interpret narrative elements like mood, setting, and character actions to create visually coherent prompts. System 101 can be configured based on user or cohort customization or other considerations described herein to create accurate and diverse prompts for a wide array of narrative styles. Thus, system 101 is configured to automatically tailor each scene's visual representation to its narrative content. Embodiments described herein can thereby enhance the storytelling aspect of the videos while contributing to a more engaging and immersive viewer experience. System 101 can accurately transform textual nuances into visual cues”; [0052] Image generation module 107 complements the text processing module by translating textual descriptions into visual representations by connecting operably to AI image generation endpoint 108. AI image generation endpoint 108 can include a generative AI tool to generate photo-realistic images, or other types of images, by inputting text, employing computer vision algorithms to create images that align with the narrative. In some embodiments, AI image generation endpoint 108 can be exemplified by Stability.AI, or similar AI image generation applications. These images can visually convey the content, providing a foundation for and enriching the video's overall presentation. In some embodiments, AI image generation endpoint 108 can generate unique images based on user-defined or AI-suggested text prompts, distinguishing each scene with original visual content and providing an alternative from reliance on stock imagery.; [0075] of LI “ The first generative model (e.g., the LMM 126a) can be any generative model trained to generate textual content describing visual prompts with details/nuances and accuracy. For instance, the first generative model may be GPT-4V, Imagen, Contrastive Language-Image Pretraining (CLIP), Flamingo, Perceiver, Multitask Unified Model (MUM), or the like. [0076] The second generative model (e.g., the LVM 126b) can be any text-to-image model trained to generate visual content (e.g., image, video, and the like) applying style(s) on subject(s) seamlessly in response to textual prompts. For instance, the second generative model may be CLIP, Vision Transformer (ViT), Megatron-Turing NLG, DALL-E, Imagen, GauGAN2, VQGAN+CLIP, or the like. In some implementations, the system may select a text-to-image model based on factors such as open source, photorealistic, creative control, computational requirements, case of use, licensing, and the like. The less sophisticated a text-to-image model, the more meta prompting and/or additional tools/models are required to provide the same quality avatars. In one embodiment, the first and second generative models are embodied in one generative model. [0061] of Zhou “ FIG. 5 depicts multiple example implementations 500 to generate target digital videos using the video generation system as employing techniques described herein. The diffusion model 202 of the video generation system (e.g., the video generation service 116 of FIG. 1) receives as inputs the reference digital video 122 with the reference object 124 and the reference caption 126 as depicted in FIG. 2. As described above, the reference digital video 122 depicts a lady in a blue dress (e.g., the reference object 124) dancing and twirling (e.g., the reference movement). The reference caption 126 describes the reference movement of reference object 124 in the reference digital video 122 as: “Lady in a blue dress is dancing and twirling.” In the example implementations 500, the target text prompt 128 is replaced with different target text prompts to generate different target videos mimicking the dancing and twirling in the reference digital video 122, but with different objects, camera views, and/or backgrounds. [0062] In a first example implementation, a target text prompt 502 identifies a different environment, reference object, and quantity of reference objects. In the illustrated example, the target text prompt 502 states: “Two astronauts are dancing, twirling on the moon.” Using the motion customization techniques described above, the diffusion model 202 generates a target digital video 504 models the reference movement (e.g., dancing, twirling) of the reference object 124 (e.g., lady in a blue dress) onto multiple objects without additional editing controls. [0063] In a second example implementation, multiple movements are customized into a target digital video. In addition to inputs 122, 124, and 126, a target text prompt 506 identifies a different environment, reference object, and camera view. The target text prompt 506 states: “Santa dancing, twirling in front of people with aerial camera over.” A first instance of the diffusion model 202 customized on the inputs 122, 126, and a first portion of target text prompt 506 (e.g., Santa dancing, twirling in front of people) generates a digital video of Santa dancing, twirling in front of people. [0064] The inputs also include another reference video 508 and another reference caption 510. The reference video 508 depicts an aerial camera flight over a herd of sheep in a meadow. The accompanying reference caption 510 states: “Aerial camera over a herd.” A second instance of the diffusion model 202 customized on the inputs 508, 510, and a second portion of target text prompt 506 (e.g., aerial camera over) to generate a digital video of a camera flying over a target object (e.g., Santa). The base diffusion model 202 is then customized with the two trained motion characteristics modules 206 from the above instances to integrate various movements (e.g., dancing, twirling, and aerial camera) into one target digital video 512. [0065] In a third example implementation, motion customization and image customization are combined into a target digital video. In addition to inputs 122, 124, and 126, a target text prompt 514 is received, identifying a different environment and a different reference object. The target text prompt 514 states: “A goat wearing a suit is dancing, twirling on a farm.” The diffusion model 202 is customized on inputs 122 and 126 and the target text prompt 514, and then generates a digital video of a different object dancing and twirling on a farm. [0066] The inputs also include one or more reference images 516 and another reference caption 518. The reference images 516 depict animals dressed in different adult clothes. The accompanying reference caption 518 states: “Dressed animals.” A conventional image customization machine-learning model customized on the inputs 516, 518, and a portion of target text prompt 514 (e.g., goat wearing a suit) is to generate a customized image of the target object. The base diffusion model 202 is then customized with the trained motion characteristics module 206 and the customized target object to support both appearance and motion customization, generating target digital video 520. The target digital video 520 shows a goat in a suit on its hind legs as it dances and twirls on a farm.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 7, Rivers, LI and Zhou teach the system of claim 1, wherein the AI-assisted compositing module is further configured to add visual effects, sound effects, or a combination thereof in the composited video (see at least [0001] of Rivers “ Conventionally, the process of adapting textual content into visual media has relied heavily on human intervention, often involving time-consuming and labor-intensive efforts. This manual approach typically involves scriptwriting, storyboarding, animation, manual editing, voice over, sound effects, video production, etc., which demand considerable expertise and resources”; [0098] s “ For example, elements in column 5, e.g., ranging from cell 401.1.5 to 401.M.5, may specifically pertain to negative prompts for stable diffusion processes across each scene, from 1 to M. Negative prompts can serve as explicit instructions to the system's AI, directing it to avoid specific actions or attributes in the animated output. These could range from excluding certain color schemes to preventing the addition of particular background noises. In a non-limiting example, an element at grid coordinate 401.3.5 could contain a negative prompt that instructs the system not to use any rapid scene transitions in the third scene, thus providing a more effective depiction of visual elements.”; [0105] Settings and Preferences Panel 404 allows customization. Users can adjust animation speeds, voice tonalities, scene durations, and transition effects. Sliders, dropdown menus, and toggle switches enable these adjustments.; [0123] The system also includes an integration module 506, which receives the visuals from the AI processing module 503, sequences them, and synchronizes them with the original audiobook or text-based content. This module may also add additional elements, such as audio, including sound effects, and/or background music, for example, to enhance the final video.”; [0021] of Zhou “Using the trained second aspect loaded therein, the machine-learning model generates the target digital video depicting the target object with the reference movement based on the target text prompt. For example, the dancing and twirling motion dynamics are used to generate a target digital video showing Ironman dancing and twirling in a similar manner. In particular, the appearance absorber is removed, and the trained temporal LoRA is loaded into the T2V diffusion model. Given the target text prompt describing a different subject (or scene), the trained T2V diffusion model generates the target digital video. The trained diffusion model accurately transfers the learned motion information to different objects (e.g., Ironman) and produces diverse motions regarding their intensities, positions, and camera views.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 8, Rivers, LI and Zhou teach the system of claim 1, further comprising: a post-production module configured to generate and output a complete film based on the composited video generated by the AI-assisted compositing module (see at least [0123] of Rivers “ The system also includes an integration module 506, which receives the visuals from the AI processing module 503, sequences them, and synchronizes them with the original audiobook or text-based content. This module may also add additional elements, such as audio, including sound effects, and/or background music, for example, to enhance the final video. [0124] The output module 507 provides the final video or animation. This video combines the visuals generated from the text with the original audio, offering a multimedia experience that visually represents the content of the original text.” [0021] of Zhou “Using the trained second aspect loaded therein, the machine-learning model generates the target digital video depicting the target object with the reference movement based on the target text prompt. For example, the dancing and twirling motion dynamics are used to generate a target digital video showing Ironman dancing and twirling in a similar manner. In particular, the appearance absorber is removed, and the trained temporal LoRA is loaded into the T2V diffusion model. Given the target text prompt describing a different subject (or scene), the trained T2V diffusion model generates the target digital video. The trained diffusion model accurately transfers the learned motion information to different objects (e.g., Ironman) and produces diverse motions regarding their intensities, positions, and camera views.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 9, Rivers, LI and Zhou teach the system of claim 8, wherein the post-production module is configured to perform one or more post-processing operations with respect to the composited video generated by the AI-assisted compositing module to generate the complete film (see at least [0123] of Rivers “ The system also includes an integration module 506, which receives the visuals from the AI processing module 503, sequences them, and synchronizes them with the original audiobook or text-based content. This module may also add additional elements, such as audio, including sound effects, and/or background music, for example, to enhance the final video. [0124] The output module 507 provides the final video or animation. This video combines the visuals generated from the text with the original audio, offering a multimedia experience that visually represents the content of the original text.”; [0021] of Zhou “Using the trained second aspect loaded therein, the machine-learning model generates the target digital video depicting the target object with the reference movement based on the target text prompt. For example, the dancing and twirling motion dynamics are used to generate a target digital video showing Ironman dancing and twirling in a similar manner. In particular, the appearance absorber is removed, and the trained temporal LoRA is loaded into the T2V diffusion model. Given the target text prompt describing a different subject (or scene), the trained T2V diffusion model generates the target digital video. The trained diffusion model accurately transfers the learned motion information to different objects (e.g., Ironman) and produces diverse motions regarding their intensities, positions, and camera views.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding independent claim 10, Rivers teaches a computer-implemented method for artificial intelligence (AI) assisted filmmaking(see at least Advantages of the AI-Generative Video Generation System [0017] The system can also be configured for customization and scalability. In some embodiments, users are enabled to tailor the visual style, animations, and media format according to their preferences. This flexibility allows for a diverse range of applications, whether it involves adapting a novel into an animated film or creating educational videos based on textbooks”, [0024] FIG. 1 is an example operating environment of a system for transforming written text into animated scenes and videos, according to some embodiments.”), the method comprising: Remaining limitations of claim 10 is similar scope to claim 1 and therefore rejected under the same rationale.
Regarding claim 11, Rivers, LI and Zhou teach the method of claim 10, further comprising: Remaining limitations of claim 11 is similar scope to claim 2 and therefore rejected under the same rationale.
Regarding claim 12, Rivers, LI and Zhou teach the method of claim 10, further comprising: Remaining limitations of claim 12 is similar scope to claim 2 and therefore rejected under the same rationale.
Regarding claim 16, Rivers, LI and Zhou teach the method of claim 10, further comprising: Remaining limitations of claim 16 is similar scope to claim 7 and therefore rejected under the same rationale.
Regarding claim 17, Rivers, LI and Zhou teach the method of claim 10, further comprising: Remaining limitations of claim 17 is similar scope to claim 8 and therefore rejected under the same rationale.
Regarding claim 18, Rivers, LI and Zhou teach the method of claim 17, further comprising: Remaining limitations of claim 18 is similar scope to claim 9 and therefore rejected under the same rationale.
Allowable Subject Matter
Claims 4-6, 13-15 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SARAH LE whose telephone number is (571)270-7842. The examiner can normally be reached Monday: 8AM-4:30PM EST, Tuesday: 8 AM-3:30PM EST, Wednesday: 8AM-2:30PM EST, Thursday and Friday off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SARAH LE/Primary Examiner, Art Unit 2614
