DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6-8, 15-16 and 20-21 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 6 recites the limitation "to compute the cross-consistency data" in lines 6-7. There is insufficient antecedent basis for this limitation in the claim. The claim previously recites “cross-image consistency data” thus making it unclear as to whether cross-consistency data is meant to refer to a broader type or different type of consistency data or whether it is mean to refer to the cross-image consistency data calculated as in claim 1. In the interest of compact prosecution the Examiner will interpret the claim as if it recites “to computer the cross-image consistency data” such that the cross-image consistency data of the original claim is required to be computed using the second prompt and second intermediate data as recited, which would cure the antecedent basis issue. Claims 7 and 8 carry through the deficiency of their parent claim 6 and are rejected for the same reasons.
Claims 15 and 20 are rejected for the same reasons as claim 6 above and are interpreted in the same manner as claim 6 above in order to render them definite for other Examination purposes. Claims 16 and 21 are rejected for carrying through the deficiencies of their parent claims and are rejected as such.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-2, 5, 9, 12, 14, and 17-19 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Cao et al1 (“Cao”).
Regarding claim 1, Cao teaches a computer-implemented method (note the method is addressed by rejection of the method steps below; see Cao, sections 5-5.4, teaching “We apply the proposed method” where figures 6-12 are all examples of the computer implementing the method to generate the output images and example results as such generation of these images requires use of a computer, and further note page 1 provides a “github” address for the software which provides the computer implemented method as understood by one having ordinary skill in the art), comprising:
obtaining a subject definition comprising a text description of a subject (note that a “subject definition” is some data defining a subject of any kind and a text description corresponds to some string of text input where for example receiving a text description in any format (text or some description of text) of a subject is a manner of obtaining a subject definition; see Cao, sections 4-4.2 teaching “Given a source image Is and the corresponding text prompt Ps, our goal is to synthesize the desired image I that complies with the target edited text prompt P (directly modified from Ps). Note that the edited target image I is spatially edited from Is and should preserve the object contents (e.g., textures and identity) in Is. For instance, consider a photo (corresponding to Ps) where a dog is sitting, and we want the dog to be in the running pose with the edited text prompt P (by adding the ‘running’ into the source prompt Ps)” and the “core idea is to combine the semantic layout synthesized with the target prompt P and the contents in the source image Is to synthesize the desired image I” and as in figure 3, “input source image Is is either a real image or a generated one from the SD model with text prompt Ps 2. During each denoising step t of synthesizing target image I, we assemble the inputs of the self-attention by 1) keeping the current Query features Q unchanged, and 2) obtaining the Key and Value features Ks, Vs from the self-attention layer in the process of syn the sizing source image Is. We dub this strategy mutual self attention” such that here a subject definition is provided comprising a text description of a subject such as the dog subject and other terms describing the dog and see section 5.4 teaching in the context of generating shots comprising multiple images depicting the subject “we first generate a source image with the prompt P and guidance (shown in the first column of Fig. 11). In the following frame synthesis process, we perform MasaCtrl on each frame to generate the target frame that is content consistent with the source frame. Fig. 11 shows the video synthesis results with coherent dense guidance. The proposed method successfully synthesizes consistent frames with highly similar content (please visit the project page for more video results)” such that here as in figure 11 for example an image and text description of a subject such as “source image with prompt P” to then generate shots of a “bear dancing on the street…” and “car is moving on the road” for example);
receiving a first prompt describing a first scene for generation of a first shot comprising two or more images depicting the subject (see Cao, sections 4-4.2 and section 5.4 as explained above where for example as in section 5.4, “first generate a source image with the prompt P and guidance” and “perform MasaCtrl on each frame to generate the target frame that is content consistent with the source frame. Fig. 11 shows the video synthesis results…: and the “proposed method successfully synthesizes consistent frames with highly similar content” where the prompt for video synthesis describing a first scene for generation of a first shot comprising two or more images depicting the subject corresponds to “bear dancing on the street…” and “car is moving on the road” for example);
generating intermediate data associated with the first shot by processing the first prompt by at least one layer of a neural network according to pre-trained weights (note that intermediate data is broadly any internal or hidden data data representation such as latent codes or features maps, or may be any data produced in the intermediate between reception of the prompt data and ultimate output data, so long as it is generated associated with the first shot by processing the first prompt by a layer of a neural network having pre-trained weights; thus see Cao, section 3.1-3.2 describing use of at least one neural network in generating such intermediate data where “method is based on the recent state-of-the-art text conditioned model Stable Diffusion (SD)… which further performs the diffusion-denoising process in the latent space rather than image space” where Stable Diffusion utilizes multiple layers of neural networks such as for example “denoising U-Net θ in the SD model, consists of a series of basic blocks, and each basic block contains a residual block [9], a self-attention module, and a cross attention [35] module. At denoising step t, the features from the previous (l−1)-th basic block first pass through the residual block to generate intermediate features flt; then they are reorganized by a self-attention layer, and receive textual information from the given text prompt P by the following cross-attention layer” and as in sections 4-4.2 and figure 3 above teaching “to combine the semantic layout synthesized with the target prompt P and the contents in the source image Is to synthesize the desired image I. To achieve so, we propose MasaCtrl, which adapts the self-attention mechanism in the SD model into a cross one to query semantically similar contents from the source image. Therefore, we can synthesize the target image I by querying the contents from Is with the modified self-attention mechanism during the denoising process” and “propose a mask-aware mutual self-attention strategy guided by the masks obtained from the cross-attention mechanism. The object mask is automatically generated from the cross-attention maps of the text token associated with the foreground object” such that the intermediate data corresponds to the various attention layers and outputs in the pipeline as in figure 3 including for example “cross-attention maps of the text token associated with foreground object” where as explained below the subject masks are derived from such intermediate data processed by the layers; finally note that as in section 5 it is explained that the neural network used utilizes pre-trained weights as in the teaching to “apply the proposed method to the state-of-the art text-to-image Stable Diffusion…model with publicly available checkpoints v1.4” and further note that the method is explicitly “tuning-free” as titled and described in the Abstract for example);
computing cross-image consistency data by (note that the following limitations are computation steps such that if the steps below are performed then they are considered to compute cross-image consistency data necessarily; furthermore, any data that relates to the manner in which across two images any aspect related to the subject is considered consistent or not consistent, or data which functions to provide or ensure consistency across images specific to a subject, would be cross-image consistency data but must be computed according to the steps below)
obtaining subject masks localizing the subject in the intermediate data using the intermediate data and the subject definition (note that the limitation requires masks which localize the subject in the intermediate data of the previous step and thus the mask must localize the subject in such latent space after processing through some layer, and must in some way be using the intermediate data to obtain such a mask as well as the subject definition, which comprises a text description in some manner as defined previously, though the manner in which they are used to obtain the localized mask is not limited; thus see Cao, section 4-4.2 and figure 3 as explained above teaching “MasaCtrl, which adapts the self-attention mechanism in the SD model into a cross one to query semantically similar contents from the source image. Therefore, we can synthesize the target image I by querying the contents from Is with the modified self-attention mechanism during the denoising process” and “propose a mask-aware mutual self-attention strategy guided by the masks obtained from the cross-attention mechanism. The object mask is automatically generated from the cross-attention maps of the text token associated with the foreground object” where as can be seen in figure 3 for example “dog” is a subject definition and is used to extract a mask that localizes the subject in the intermediate data of the “cross-attention” intermediate data and “we utilize the semantic cross-attention maps to create a mask to distinguish the foreground and background in both source and target images Is and I” and “at step t, we first perform a forward pass with the fixed U-Net backbone with prompt Ps and edited prompt P, respectively, to generate intermediate cross attention maps” and “then obtain the averaged cross-attention map for the token correlated to the foreground object. We denote Ms and M as masks extracted for the foreground objects in Is and I, respectively. With these masks, we can restrict the object in I to query contents information only from the object region in Is” such that this specifically is how the subject masks are localized using the intermediate data and the subject definition and as in section 5.4 when generating the first shot as in figure 11 then as Masactrl is performed on each frame then the intermediate data and masks would be used in generating the shots of multiple consistent images) and
computing a self-attention map by combining the subject masks with query tensors and keys (see Cao, section 3.2 as explained above and describing the “attention mechanism” where “Q is the query features projected from the spatial features, and K,V are the key and value features projected from the spatial features (in self-attention layers) or the textual embedding (in cross-attention layers) with corresponding projection matrices” and “we adapt the self-attention mechanism in T2I model to query contents from source images with delicate designs” as further explained in sections 4-4.2 and figure 3 teaching “During each denoising step t of synthesizing target image I, we assemble the inputs of the self-attention by 1) keeping the current Query features Q unchanged, and 2) obtaining the Key and Value features Ks, Vs from the self-attention layer in the process of syn the sizing source image Is. We dub this strategy mutual self attention” and “propose a mask-aware mutual self-attention strategy guided by the masks obtained from the cross-attention mechanism. The object mask is automatically generated from the cross-attention maps of the text token associated with the foreground object” such that as in equations 6, 7 and 8, “With these masks, we can restrict the object in I to query contents information only from the object region in Is” such that the “final attention output” is such a self-attention map which is used to compute cross-image consistency data and does so by combining the subject masks with query tensors and keys as in equations 6, 7 and 8 and as in figure 3 so that the “object region and the background region query the content information from corresponding restricted areas rather than all features”); and
processing the cross-image consistency data by at least one remaining layer of the neural network according to the pre-trained weights to generate a video comprising at least the first shot (see Cao, section 4-4.2, “algorithm 1” and figure 3 where the mutual self-attention maps and cross-image consistency data are processed by remaining layers of the neural network such as where “the composition and the shape of the object can be roughly generated complying with the target prompt P. Then the content information from the source image Is is queried by the mutual self-attention mechanism to fill the generated layout of I. After iterative denoising, we can obtain the synthesized image with similar contents in the source image and structure of I∗ that conforms to the input prompt” such that this iterative denoising is processing of the cross-image consistency data explained above to synthesize or generate a video comprising the shot when extended to video synthesis as in section 5.4 and this is according to the pretrained weights of the SD model in order to generate a video comprising at least a first shot such as in section 5.4 and figure 11 teaching “first generate a source image with the prompt P and guidance” and “perform MasaCtrl on each frame to generate the target frame that is content consistent with the source frame. Fig. 11 shows the video synthesis results…: and the “proposed method successfully synthesizes consistent frames with highly similar content”).
Regarding claim 2, Cao teaches all that is required as applied to claim 1 above and further teaches wherein the subject masks corresponding to the two or more images are combined to compute the cross-image consistency data (see Cao, section 4.2 and figure 3 teaching “we utilize the semantic cross-attention maps to create a mask to distinguish the foreground and background in both source and target images Is and I” and “obtain the averaged cross-attention map for the token correlated to the foreground object. We denote Ms and M as masks extracted for the foreground objects in Is and I, respectively. With these masks, we can restrict the object in I to query contents information only from the object region in Is” where to get the cross-image consistency data such as the “final attention output” the “masks” are combined through equations 6-8 to compute the cross-image consistency data).
Regarding claim 5, Cao teaches all that is required as applied to claim 2 above and further teaches aligning common features within the cross-image consistency data before processing the cross-image consistency data by the at least one remaining layer (see Cao, sections 4-4.2 and figure 3 teaching aligning features of source and target images and prompts through the initial layers of the Unet which results in a semantic similarity of query features at corresponding layers so that mask extraction and cross and mutual self-attention can be performed before processing the cross-image consistency data at the at least one remaining layer that produces the output image).
Regarding claim 9, Cao teaches all that is required as applied to claim 1 above and further teaches wherein the pre-trained weights are determined by training the neural network to generate independent images in response to processing training text descriptions (see Cao, sections 3-5 all teaching use of “Stable Diffusion (SD))”, where as in section 5 “We apply the proposed method to the state-of-the-art text-to-image Stable Diffusion model with publicly available checkpoints v1.4” where this of course refers to training text descriptions processed along with images in pairs which trains the neural network to generate independent images such that the pre-trained weights of the SD model are determined by training the network to generate independent images (hence text-to-image model) in response to processing training text descriptions, prior to Cao utilizing the pre-trained weights of checkpoint v1.4).
Regarding claim 12, Cao teaches all that is required as applied to claim 1 above and further teaches wherein at least one of the steps of receiving, generating, computing, or processing is performed for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle (note that the claim essentially recites an intended use as it merely specifies that some above step is “for training, testing, or certifying a neural network employed” without specifying how exactly such performing of the steps for such purposes is actually accomplished, nor how such steps “for” such purposes would necessarily impact the functioning of such steps, nor what “training, testing, or certifying” such a network would encompass; thus see Cao, sections 4 and 5 and figures 6-12 where the receiving, generating, computing and processing steps above are for testing a neural network such as that utilized as described above and in a machine such as used to generate the outputs as in these figures where this design of the neural network can be seen as training, testing, and certifying a neural network in a machine that produced the data as in the produced shots).
Regarding claim 14, the instant claim recites a “system” comprising two functionally defined devices in “a memory that stores pre-trained weights for a neural network” and “a processor that is connected to the memory, wherein the processor is configured to” perform the same functions as those recited, addressed and rejected with respect to the computer-implemented method of claim 1. Thus the only difference is the pre-trained weights are associated with a “memory” and the steps performed by the computer are by a processor that is connected to the memory. Cao already teaches such pre-trained weights and a computer implementation using such pre-trained weights in order to perform the recited functions as explained in the rejection of claim 1. Cao also teaches such pre-trained weights are stored in a memory given that these weights are digital data structures comprising millions of numerical values, such that to be available for use they must necessarily be stored in some memory, in order for the processor necessary to calculate these millions of numerical value, to retrieve such weights and output the computer-generated results as seen in the figures for example (see Cao, page 1 “github.com/tencentarc/masactrl” for example such that one of ordinary skill in the art recognizes that the computer-implementation of Cao necessarily teaches such a memory and processor as broadly recited where such data is available at such a github page). In light of this, the limitations of claim 14 correspond to the limitations of claim 1; thus it is rejected on the same grounds as claim 1.
Regarding claim 17, in view of the analysis above, it can be seen that the limitations of claim 17 correspond to the limitations of claim 2; thus it is rejected on the same grounds as claim 2.
Regarding claim 18, the instant claim is directed toward an apparatus in the form of a “non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps” which are the same as the same steps as recited in claim 1. Thus similarly to the analysis of claim 14 above, the only difference is the pre-trained weights are associated with a memory in the form of a “non-transitory computer-readable media” and the steps performed by the computer are by a processor that uses the stored computer instructions. Cao already teaches such pre-trained weights and a computer implementation using such pre-trained weights in order to perform the recited functions as explained in the rejection of claim 1. Cao also teaches such pre-trained weights are stored in a memory given that these weights are digital data structures comprising millions of numerical values, such that to be available for use they must necessarily be stored in some memory, in order for the processor necessary to calculate these millions of numerical value, to retrieve such weights and output the computer-generated results as seen in the figures for example (see Cao, page 1 “github.com/tencentarc/masactrl” are available at a website providing access to memory with such code and pre-trained weights such that this is also the code implemented to obtain the results as in figures showing the image generation results for example such that one of ordinary skill in the art recognizes that the computer-implementation of Cao necessarily teaches such a memory and processor as broadly recited). In light of this, the limitations of claim 18 correspond to the limitations of claim 1; thus it is rejected on the same grounds as claim 1.
Regarding claim 21, in view of the analysis above, it can be seen that the limitations of claim 21 corresponds to the limitations of claims 2; thus it is rejected on the same grounds as claim 2.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 10, 11 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Cao in view of Ceylon Aksit et al2 (“Aksit”).
Regarding claim 10, Cao teaches all that is required as applied to claim 1 above but fails to specifically teach wherein at least one of the steps of receiving, generating, computing, or processing is performed on a server or in a data center to generate the first shot, and the first shot is streamed to a user device. Rather while Cao’s method is and must be computer-implemented, such an arrangement with a server and user device is not explicitly detailed. However one having ordinary skill in the art would recognize in the context of the invention, any practical implementation of the technique requires massive computing capabilities which one of ordinary skill in the art would also recognize is in any real world case supplied by compute resources external to the end user device. Thus while Cao does not specifically teach such steps to generate the first shot as performed on a server or in a data center where the results are then streamed or communicated to the user device which would show the shots as depicted in figure 11 for example, one of ordinary skill in the art would understand that such an arrangement is likely the only way possible for Cao to have implemented the technique. Regardless, the following prior art supports all of such analysis and provides this known technique applicable to the base method of Cao.
In the same field of endeavor relating to utilizing image diffusion models and their neural networks for generating video images relating to text prompts guiding generation of video (see Aksit, paragraphs 0026-0035 teaching “the video editing system of the present disclosure enables editing of a video using a pre-trained image diffusion model and a prompt (e.g., text-based editing instruction) without any additional training. In some embodiments, the input video clip is inverted and edited based on a textual prompt received from a user or other entity” and “image generation model 110 may be implemented as a neural network. A neural network may include a machine-learning model that can be tuned (e.g., trained) based on training input to approximate unknown functions. In particular, a neural network can include a model of interconnected digital neurons that communicate and learn to approximate complex functions and generate outputs based on a plurality of inputs provided to the model. For instance, the neural network includes one or more machine learning algorithms. In other words, a neural network is an algorithm that implements deep learning techniques, i.e., machine learning that utilizes a set of algorithms to attempt to model high-level abstractions in data. In various embodiments, the image generation model 110 may execute in a neural network manager 112. The neural network manager 112 may include an execution environment for the image generation model, providing any needed libraries, hardware resources, software resources, etc. for the image generation model” and “Edited video 126 has had its appearance edited based on the input prompt 104 without requiring any per-scene training by the image generation model and also maintains appearance and temporal consistency”), Aksit teaches that receiving, generating, computing, or processing related to the generation of some first generated video shot is performed on a server or in a data center (see Aksit, paragraphs 0026-0035 and figure 1, teaching “a video editing system 100 can enable video editing using an image generation model, such as Stable Diffusion, or other image diffusion-based model(s). The video editing system 100 may be implemented as a standalone system, such as an application executing on a client computing device, server computing device, or other computing device. In some embodiments, the video editing system may be implemented as a tool incorporated into another system, service, application, etc. to provide image diffusion-based video editing. The video editing system 100 may be implemented in a user device, in a service provider device as part of a cloud computing model, or other device which may receive input videos and return output videos” and see also paragraphs 0067-0081 and figures 9-11 teaching “the components 902-910 of the video editing system 900 may, for example, be implemented as one or more stand-alone applications, as one or more modules of an application, as one or more plug-ins, as one or more library functions or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components 902-910 of the video editing system 900 may be implemented as a stand-alone application, such as a desktop or mobile application. Furthermore, the components 902-910 of the video editing system 900 may be implemented as one or more web-based applications hosted on a remote server. Alternatively, or additionally, the components of the video editing system 900 may be implemented in a suite of mobile device applications or “apps.”” And “the video editing system 900 can be implemented as a single system. In other embodiments, the video editing system 900 can be implemented in whole, or in part, across multiple systems. For example, one or more functions of the video editing system 900 can be performed by one or more servers, and one or more functions of the video editing system 900 can be performed by one or more client devices. The one or more servers and/or one or more client devices may generate, store, receive, and transmit any type of data used by the video editing system 900, as described herein” and finally “the one or more client devices can include or implement at least a portion of the video editing system 900. In other implementations, the one or more servers can include or implement at least a portion of the video editing system 900. For instance, the video editing system 900 can include an application running on the one or more servers or a portion of the video editing system 900 can be downloaded from the one or more servers. Additionally or alternatively, the video editing system 900 can include a web hosting application that allows the client device(s) to interact with content hosted at the one or more server(s)” where the computing devices utilized for the processing and computing may be “GPUs”, such that here a user device can be streamed the output through being sent it through one of the remote server or cloud based or data center based implementations where any of the steps may be performed by any combination of the device architecture ), and the first shot is streamed to a user device (see Aksit, paragraphs 0067-0089 and figure 9-11 and paragraphs 0098-0106 as explained above teaching for example “the components 902-910 of the video editing system 900 may, for example, be implemented as one or more stand-alone applications, as one or more modules of an application, as one or more plug-ins, as one or more library functions or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components 902-910 of the video editing system 900 may be implemented as a stand-alone application, such as a desktop or mobile application. Furthermore, the components 902-910 of the video editing system 900 may be implemented as one or more web-based applications hosted on a remote server. Alternatively, or additionally, the components of the video editing system 900 may be implemented in a suite of mobile device applications or “apps” and “cloud computing environments. In this description, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be employed in the marketplace to offer ubiquitous and convenient on-demand access to the shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization and released with low management effort or service provider interaction, and then scaled accordingly” such that here a user device can be streamed the output through being sent it through one of the remote server or cloud based or data center based implementations where any of the steps may be performed by any combination of the device architecture). Thus Aksit teaches known techniques applicable to the base system of Gui which is ready for improvement through solutions to known ways to implement video diffusion networks based on input prompts and pre-trained weights.
Therefore it would have been obvious for one of ordinary skill the art before the effective filing date of the invention to modify Cao by applying the known techniques of Aksit as doing so would be no more than application of a known technique to a base device ready for improvement which would yield predictable results and result in an improved system. The predictable result of the combination would be that the video generation method of Cao is utilized and implemented as in Aksit, where at least one of the above claimed steps would be implemented on a server or data center where such result would then be streamed to a user device. This would result in an improved system as it would allow a user of any end user device to avoid the intensive video generation steps requiring specialized hardware while allowing to leverage the massive computing power of servers running specialized hardware not available to user devices as would be apparent to one having ordinary skill in the art as well as is suggested by Aksit (see Aksit, paragraph 0099 teaching “cloud computing environments. In this description, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be employed in the marketplace to offer ubiquitous and convenient on-demand access to the shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization and released with low management effort or service provider interaction, and then scaled accordingly”).
Regarding claim 11, Cao teaches all that is required as applied to claim 1 above but fails to specifically teach wherein at least one of the steps of receiving, generating, computing, or processing is performed within a cloud computing environment. Rather while Cao’s method is and must be computer-implemented, such an arrangement or mention of a cloud computing environment is not detailed. Thus Cao stands as a base device upon which the claimed invention can be seen as an improvement through utilization of cloud computing to perform one of the recited steps of the method.
In the same field of endeavor relating to utilizing image diffusion models and their neural networks for generating video images relating to text prompts guiding generation of video (see Aksit, paragraphs 0026-0035 teaching “the video editing system of the present disclosure enables editing of a video using a pre-trained image diffusion model and a prompt (e.g., text-based editing instruction) without any additional training. In some embodiments, the input video clip is inverted and edited based on a textual prompt received from a user or other entity” and “image generation model 110 may be implemented as a neural network. A neural network may include a machine-learning model that can be tuned (e.g., trained) based on training input to approximate unknown functions. In particular, a neural network can include a model of interconnected digital neurons that communicate and learn to approximate complex functions and generate outputs based on a plurality of inputs provided to the model. For instance, the neural network includes one or more machine learning algorithms. In other words, a neural network is an algorithm that implements deep learning techniques, i.e., machine learning that utilizes a set of algorithms to attempt to model high-level abstractions in data. In various embodiments, the image generation model 110 may execute in a neural network manager 112. The neural network manager 112 may include an execution environment for the image generation model, providing any needed libraries, hardware resources, software resources, etc. for the image generation model” and “Edited video 126 has had its appearance edited based on the input prompt 104 without requiring any per-scene training by the image generation model and also maintains appearance and temporal consistency”), Aksit teaches that at least one of the steps of receiving, generating, computing, or processing related to the generation of some first generated video shot is performed within a cloud computing environment (see Aksit, paragraphs 0026-0035 and figure 1, teaching “a video editing system 100 can enable video editing using an image generation model, such as Stable Diffusion, or other image diffusion-based model(s). The video editing system 100 may be implemented as a standalone system, such as an application executing on a client computing device, server computing device, or other computing device. In some embodiments, the video editing system may be implemented as a tool incorporated into another system, service, application, etc. to provide image diffusion-based video editing. The video editing system 100 may be implemented in a user device, in a service provider device as part of a cloud computing model, or other device which may receive input videos and return output videos” and see also paragraphs 0067-0081 and figures 9-11 teaching “the components 902-910 of the video editing system 900 may, for example, be implemented as one or more stand-alone applications, as one or more modules of an application, as one or more plug-ins, as one or more library functions or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components 902-910 of the video editing system 900 may be implemented as a stand-alone application, such as a desktop or mobile application. Furthermore, the components 902-910 of the video editing system 900 may be implemented as one or more web-based applications hosted on a remote server. Alternatively, or additionally, the components of the video editing system 900 may be implemented in a suite of mobile device applications or “apps.”” And “the video editing system 900 can be implemented as a single system. In other embodiments, the video editing system 900 can be implemented in whole, or in part, across multiple systems. For example, one or more functions of the video editing system 900 can be performed by one or more servers, and one or more functions of the video editing system 900 can be performed by one or more client devices. The one or more servers and/or one or more client devices may generate, store, receive, and transmit any type of data used by the video editing system 900, as described herein” and finally “the one or more client devices can include or implement at least a portion of the video editing system 900. In other implementations, the one or more servers can include or implement at least a portion of the video editing system 900. For instance, the video editing system 900 can include an application running on the one or more servers or a portion of the video editing system 900 can be downloaded from the one or more servers. Additionally or alternatively, the video editing system 900 can include a web hosting application that allows the client device(s) to interact with content hosted at the one or more server(s)” where the computing devices utilized for the processing and computing may be “GPUs”, such that here a user device can be streamed the output through being sent it through one of the remote server or cloud based or data center based implementations where any of the steps may be performed by any combination of the device architecture ). Thus Aksit teaches known techniques applicable to the base system of Gui which is ready for improvement through solutions to known ways to implement video diffusion networks based on input prompts and pre-trained weights.
Therefore it would have been obvious for one of ordinary skill the art before the effective filing date of the invention to modify Cao by applying the known techniques of Aksit as doing so would be no more than application of a known technique to a base device ready for improvement which would yield predictable results and result in an improved system. The predictable result of the combination would be that the video generation method of Cao is implemented using the architectures as in Aksit where at least one of the above claimed steps would be implemented in a cloud computing environment with results provided to the user through the cloud computing architecture chosen by the system designer. This would result in an improved system as it would allow a user of any end user device to avoid the intensive video generation steps requiring specialized hardware while allowing to leverage the massive computing power of servers running specialized hardware not available to user devices as would be apparent to one having ordinary skill in the art as well as is suggested by Aksit (see Aksit, paragraph 0099 teaching “cloud computing environments. In this description, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be employed in the marketplace to offer ubiquitous and convenient on-demand access to the shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization and released with low management effort or service provider interaction, and then scaled accordingly”).
Regarding claim 13, Cao teaches all that is required as applied to claim 1 above but fails to specifically teach wherein at least one of the steps of receiving the first prompt, receiving the second prompt, generating, computing, or processing is performed on a virtual machine comprising a portion of a graphics processing unit. Rather while Cao’s method is and must be computer-implemented, such an arrangement or mention of a virtual machine comprising a portion of a graphics processing unit performing at least one of the steps is not disclosed. Thus Cao stands as a base device upon which the claimed invention can be seen as an improvement through utilization of a virtual machine comprising a portion of a graphics processing unit as recited which would give the benefits of virtualization as understood by one having ordinary skill in the art.
In the same field of endeavor relating to utilizing image diffusion models and their neural networks for generating video images relating to text prompts guiding generation of video (see Aksit, paragraphs 0026-0035 teaching “the video editing system of the present disclosure enables editing of a video using a pre-trained image diffusion model and a prompt (e.g., text-based editing instruction) without any additional training. In some embodiments, the input video clip is inverted and edited based on a textual prompt received from a user or other entity” and “image generation model 110 may be implemented as a neural network. A neural network may include a machine-learning model that can be tuned (e.g., trained) based on training input to approximate unknown functions. In particular, a neural network can include a model of interconnected digital neurons that communicate and learn to approximate complex functions and generate outputs based on a plurality of inputs provided to the model. For instance, the neural network includes one or more machine learning algorithms. In other words, a neural network is an algorithm that implements deep learning techniques, i.e., machine learning that utilizes a set of algorithms to attempt to model high-level abstractions in data. In various embodiments, the image generation model 110 may execute in a neural network manager 112. The neural network manager 112 may include an execution environment for the image generation model, providing any needed libraries, hardware resources, software resources, etc. for the image generation model” and “Edited video 126 has had its appearance edited based on the input prompt 104 without requiring any per-scene training by the image generation model and also maintains appearance and temporal consistency”), Aksit teaches that at least one of the steps of receiving the first prompt, receiving the second prompt, generating, computing, or processing is performed on a virtual machine comprising a portion of a graphics processing unit (see Aksit, paragraphs 0026-0035 and figure 1, teaching “a video editing system 100 can enable video editing using an image generation model, such as Stable Diffusion, or other image diffusion-based model(s). The video editing system 100 may be implemented as a standalone system, such as an application executing on a client computing device, server computing device, or other computing device. In some embodiments, the video editing system may be implemented as a tool incorporated into another system, service, application, etc. to provide image diffusion-based video editing. The video editing system 100 may be implemented in a user device, in a service provider device as part of a cloud computing model, or other device which may receive input videos and return output videos” and see also paragraphs 0067-0081 and figures 9-11 teaching “the components 902-910 of the video editing system 900 may, for example, be implemented as one or more stand-alone applications, as one or more modules of an application, as one or more plug-ins, as one or more library functions or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components 902-910 of the video editing system 900 may be implemented as a stand-alone application, such as a desktop or mobile application. Furthermore, the components 902-910 of the video editing system 900 may be implemented as one or more web-based applications hosted on a remote server. Alternatively, or additionally, the components of the video editing system 900 may be implemented in a suite of mobile device applications or “apps.”” And “the video editing system 900 can be implemented as a single system. In other embodiments, the video editing system 900 can be implemented in whole, or in part, across multiple systems. For example, one or more functions of the video editing system 900 can be performed by one or more servers, and one or more functions of the video editing system 900 can be performed by one or more client devices. The one or more servers and/or one or more client devices may generate, store, receive, and transmit any type of data used by the video editing system 900, as described herein” and finally “the one or more client devices can include or implement at least a portion of the video editing system 900. In other implementations, the one or more servers can include or implement at least a portion of the video editing system 900. For instance, the video editing system 900 can include an application running on the one or more servers or a portion of the video editing system 900 can be downloaded from the one or more servers. Additionally or alternatively, the video editing system 900 can include a web hosting application that allows the client device(s) to interact with content hosted at the one or more server(s)” where the computing devices utilized for the processing and computing may be “GPUs”, such that here the shared pool of computing resources can be “rapidly provisioned via virtualization” where these resources may be “GPUs” and these are ”securely divided between multiple customers” thus provisioning a virtual machine with access to a portion or slice of that graphics processing unit resource to the multiple users). Thus Aksit teaches known techniques applicable to the base system of Gui which is ready for improvement through solutions to known ways to implement video diffusion networks based on input prompts and pre-trained weights.
Therefore it would have been obvious for one of ordinary skill the art before the effective filing date of the invention to modify Cao by applying the known techniques of Aksit as doing so would be no more than application of a known technique to a base device ready for improvement which would yield predictable results and result in an improved system. The predictable result of the combination would be that the video generation method of Cao would be utilized and then implemented as in Aksit, where at least one of the above claimed steps would be performed on a virtual machine comprising a portion of a graphics processing unit to allow the end user device to avoid processing such data. This would result in an improved system as it would allow a user of any end user device to avoid the intensive video generation steps requiring specialized hardware while allowing to leverage the massive computing power of servers running specialized hardware not available to user devices as would be apparent to one having ordinary skill in the art as well as is suggested by Aksit (see Aksit, paragraph 0099 teaching “cloud computing environments. In this description, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be employed in the marketplace to offer ubiquitous and convenient on-demand access to the shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization and released with low management effort or service provider interaction, and then scaled accordingly”).
Claim(s) 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Cao in view of Jain et al3 (“Jain”).
Regarding claim 23, Cao teaches all that is required as applied to claim 1 above but fails to teach or suggest the self-attention map is computed by nulling logits based on the subject mask during a softmax operation. Rather, Cao teaches a mask based attention computation where the subject mask is incorporated within the attention function to restrict target to image queries to attend only to object-region keys of the source image (see section 4.2 as explained above). However, Cao does not teach the technique by which the mask is applied to make such a restriction so. Thus Cao stands as a base device upon which the claimed invention can be seen as an improvement through providing a technique for such a restriction based on the mask by nulling logits based on the subject mask during a softmax operation as this would provide a way to restrict the attention by choosing to give zero or null weight to one of the logits of the softmax operation.
In the same field of endeavor relating to diffusion based video generation based on text prompts and masking of objects, Jain teaches a technique which provides a mask related to a subject in an image prompt for guiding the generation of a shot of frames of a subject related to the prompt, and teaches that self-attention maps can be computed by nulling logits based on a subject mask during a softmax operation (see Jain, section 4.2 teaching “masked spatio-temporal mixed attention” where “We use additive masking for attention, i.e. for any query Q, key K, value V a binary 2D attention mask M” where
PNG
media_image1.png
143
480
media_image1.png
Greyscale
and “the additive mask M is such that it has a large negative value on the masked out entries in M, leading to the attention scores for such entries being small” such that here this nulls the logits in the softmax operation). Thus Jain provides a known technique applicable to the base system of Cao above.
Therefore it would have been obvious for one of ordinary skill in the art to modify Cao to utilize Jain’s masked attention technique when computing the self-attention maps in Cao as doing so would be no more than application of a known technique to a base device ready for improvement. Here while Jain’s masks are derived beforehand, their resulting mask is still applied in the same manner in a self-attention operation such that it could be utilized as the manner in which the attention is restricted by Cao’s mask from the intermediate data, such that the predictable result would be that the self-attention maps in Cao would be restricted in their attention queries through the nulling of the logits as in Cao. This would result in an improved system as it would ensure that the masked areas are paid attention to in the self-attention computation.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-3, and 5-21 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-7, 10, 15-22 of copending Application No. 18650726 in view of Cao.
Conflicting Application No. 18650726
Pending Application 18650761
Claim 1.
A computer-implemented method, comprising: receiving a text description related to a subject with two or more prompts describing scenes for generation of two or more images depicting the subject;
generating intermediate data associated with the two or more images by processing the text description with two or more prompts by at least one layer of a neural network according to pre-trained weights;
computing cross-image consistency data specific to the subject using the intermediate data; and
processing the cross-image consistency data by at least one remaining layer of the neural network according to the pre-trained weights to generate the two or more images.
Claim 1.
A computer-implemented method, comprising: obtaining a subject definition comprising a text description of a subject;
receiving a first prompt describing a first scene for generation of a first shot comprising two or more images depicting the subject;
generating intermediate data associated with the first shot by processing the first prompt by at least one layer of a neural network according to pre-trained weights;
computing cross-image consistency data by obtaining subject masks localizing the subject in the intermediate data using the intermediate data and the subject definition; and
computing a self-attention map by combining the subject masks with query tensors and keys; and
processing the cross-image consistency data by at least one remaining layer of the neural network according to the pre-trained weights to generate a video comprising at least the first shot.
Thus it can be seen that the instant claim 1 is a slightly different and with regard to necessitating generating “video”, narrower version of the conflicting claim and thus conflicting claim 1 is effectively a genus to the species recited in claim 1 of the instant application. The differences between the claims are the instant claims require generating a video whereas the conflicting claims require generation of two or more images but not necessarily a video. Furthermore the instant claims requires a first prompt with the conflicting claim requiring a first and second prompt, but in the context of a text description of a subject and whatever two or more prompts are related to the text description are for generation of two or more images depicting the subject. Thus in the case that a text description of a subject as in the instant claims contained multiple prompts related to generating the scene such as multiple text keywords or tokens then such reception of a first prompt would also be reception of a first and second prompt as in the conflicting claim. However, these remaining features of generating video instead of more broadly two or more images and receiving two or more prompts relating to a subject are taught by Cao as applied in the corresponding rejections of the corresponding limitations above. Thus modifying the conflicting claim 1 to arrive at the claimed invention using the applicable techniques taught above by Cao would have been obvious for one of ordinary skill in the art before the effective filing date of the invention as adding such features is known as explained above and doing so would yield predictable results and result in an improved system. Note that the abbreviated rationale above is provided in the interest of brevity and given that the claims are likely subject to further amendment. Independent claims 14 and 18 correspond to claim 1 as explained above and are rejected for the same reasons in this instance as claim 1 as well.
Note that in view of such modification, the dependent claims correspond and conflict according to the following table.
Conflicting Application No. 18650726
Pending Application 18650761
2
2
3
3
6
4
7
5
10
9
15
10
16
11
17
12
18
13
2
17
2
19
This is a provisional nonstatutory double patenting rejection.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 14 and 18 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant’s arguments on page 10 regarding Guo in view of Aksit are moot as the grounds of rejection has changed. However, the Examiner notes that Applicant’s argument regarding Aksit do not consider the full teaching and explanation given above nor consider the breadth of the claim language of claims 10 and 13. Thus claims 10 and 13 still rely upon Aksit which in combination with Cao renders the claims obvious. Applicant is not advised to pursue a strategy whereby attempts to distinguish the claims relate to such basic client/server execution options or to high level server computing techniques or intended use, unless the inventive concept of the claim is related to such techniques and is specifically tied to such execution options through the claim language.
Allowable Subject Matter
Claims 4 and 22 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the prior art of record fails to teach or suggest the respective claim limitations when considered as a whole.
Regarding claim 4, Cao fails to teach “interpolating between the cross-image consistency data produced by combining the subject masks and denoised intermediate data before processing the cross-image consistency data by the at least one remaining layer.” Rather such an interpolation step is not described or suggested in Cao and the Examiner is unable to find any teaching or suggestion of such a technique as recited. None of the cited prior art which is the closest prior art in the same field of endeavor, such as to Khachatryan or Guo, teaches or suggests the concept of interpolating between cross-image consistency data by combining subject masks and denoised intermediate data and each perform such techniques without performing an interpolation in such a manner in any way. Thus the claim contains allowable subject matter.
Regarding claim 22, Cao teaches all that is required as applied to claim 1 above but fails to teach wherein the subject masks are created by averaging cross-image self-attention maps related to the subject. Rather, Cao actually does teach at least “averaging” in connection with the subject masks, but crucially the data which is averaged and the processing stage at which the averaging takes place is not “by averaging cross-image self-attention maps related to the subject.” Cao teaches in section 4.2 “we average the cross-attention maps across all heads and layers” and “obtain the averaged cross-attention map for the token correlated to the foreground object”, however, these cross-attention maps are distinct from self-attention maps related to the subject as instead they are cross-attention maps that map spatial data to text data whereas the cross-image self-attention maps are maps that map spatial data to spatial data. The Examiner is further unable to find any teaching or suggestion of such technique wherein the subject masks are created by averaging cross-image self-attention maps related to the subject as in the claimed context.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SCOTT E SONNERS whose telephone number is (571)270-7504. The examiner can normally be reached Mon-Friday 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SCOTT E SONNERS/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
1 Cao M, Wang X, Qi Z, Shan Y, Qie X, Zheng Y. MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing. arXiv preprint arXiv:2304.08465. 2023 Apr 17.
2 US PGPUB No. 20250111866
3 Jain Y, Nasery A, Vineet V, Behl H. PEEKABOO: Interactive Video Generation via Masked-Diffusion. arXiv preprint arXiv:2312.07509. 2023 Dec 12.
Prosecution Insights