DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS), submitted on 9/24/2024 and 7/13/2026, are being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under pre-AIA 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims under pre-AIA 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were made absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was not commonly owned at the time a later invention was made in order for the examiner to consider the applicability of pre-AIA 35 U.S.C. 103(c) and potential pre-AIA 35 U.S.C. 102(e), (f) or (g) prior art under pre-AIA 35 U.S.C. 103(a).
Claims 1-3 are rejected under 35 U.S.C. 103 as being unpatentable over Ayush (US Patent 11,238,093 B2), (“Ayush”), in view of Geng et al. (US Patent 11,210,523 B2), (“Geng”).
Regarding claim 1, Ayush meets the claim limitations as follow.
A method (a method) [Ayush: col. 1, line 29] comprising: obtaining an input prompt describing a video scene ((textual descriptions about multiple videos are obtained) [Ayush: col. 11, line 34-35]; (a video is represented in the form of text, so a user can input a natural language query where textual descriptions, like the input query, can be retrieved. The sentence can be represented in the form of an encoded vector) [Ayush: col. 6, line 10-14]; (Input the video feature vector into an LSTM encoder) [Ayush: col. 7, line 33-34; Fig. 7]); generating a plurality of frame-wise token (Feature encoder 1315 may generate encoded video information based on the video feature vector) [Ayush: col. 15, line 1-2; col. 1, line 35-36; Figs. 3, 7, 13] – Note: The video information can represent tokens. For instance, BOS and EOS are examples of the frame-wise tokens) embeddings corresponding to a sequence of video frames, respectively ((Embedding component 1325 may embed the encoded video information using an embedding model) [Ayush: col. 15, line 37-38; col. 1, line 39-40; Figs. 9-11, 13-14]; (The input to feature encoder 1400 starts with the beginning of sentence <BOS> token which indicates the beginning of the sequence and the output of the LSTM 1405 is fed to the current unit to produce the next word and recursively the network is unfolded to obtain the output until the end of sentence <EOS> is produced which indicates the end of sequence and the network stops unfolding at the moment) [Ayush: col. 17, line 36-43; Fig. 14] – Note: Ayush discussed an illustration how the embedded tokens are used in the video), based on the input prompt (Generate an encoded video information based on the video feature vector) [Ayush: col. 1, line 35-36; Figs. 3, 7]; and generating (Video frames 405 are generated) [Ayush: col. 8, line 45], using a video generation model (recurrent neural network (RNN) encoder-decoder model with
Long Short-term Memory (LSTM) cells. The LSTM cells may be used to capture information related to the temporal order of the frames (as represented in different portions of the feature vector)) [Ayush: col. 3, line 53-58], a synthesized video depicting the video scene (A plurality of image-recognition model can be used to extract features from the video frames. The general feature extraction can be described by considering {x1, x2, ... , xn} as the input comprising of video frames, and the output is also a sequence {y1, y2, ... , yn}, which is described as, y,=f(x,) where f( ) is any suitable image recognition model) [Ayush: col. 8, line 22-26] – Note: Ayush discloses a plurality of image-recognition model which is used to select video frames that have appropriate description features), wherein the synthesized video comprises a plurality of images corresponding to the sequence of video frames (the set of frames contain the information used to generate the relevant text describing the frames. Thus, visual content of the video may be used to provide an automated method of describing the video. The frames are input to the neural image recognition model that can capture the features suggested for the neural language model to describe the video) [Ayush: col. 8, line 30-36].
Ayush does not explicitly disclose the following claim limitations (Emphasis added).
generating, using a video generation model, a synthesized video depicting the video scene.
However, in the same field of endeavor Geng further discloses the claim limitation and the deficient claim limitation as follows:
generating, using a video generation model, a synthesized video depicting the video scene (the audio-visual scene aware neural network 108 is trained to process features (e.g., scene graph representations) that represent scene in each video frame of the sequence of video frames to generate a response) [Geng: col. 7, line 14-17; Fig. 1].
generating a plurality of frame-wise token embeddings corresponding to a sequence of video frames (In the semantics-controlled transformer reasoning 410, the sentence sources that include the dialog history (H), the video caption (C), the query (Q1) and the reference answer (Attn) are embedded together using tokenization and word positional embedding layer) [Geng: col. 15, line 6-26; Fig. 4].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush with Geng to program the system to implement of Geng’s method.
Therefore, the combination of Ayush with Geng will enable the object classifiers of the video system can be used for different types of videos, irrespective of the video context, including the driving application and the detection in sport events [Geng: col. 4, line 18-47].
Regarding claim 2, Ayush meets the claim limitations as set forth in claim 1. Ayush further meets the claim limitations as follow.
encoding the input prompt ((encoded video information based on the video feature vector) [Ayush: col. 15, line 1-2; col. 1, line 35-36; Figs. 3, 7, 13]; (a video is represented in the form of text, so a user can input a natural language query where textual descriptions, like the input query, can be retrieved. The sentence can be represented in the form of an encoded vector) [Ayush: col. 6, line 10-14]) to obtain a plurality of token embeddings; generating one or more frame-specific embeddings for each of the sequence of video frames, respectively, based on the plurality of token embeddings (The input to feature encoder 1400 starts with the beginning of sentence <BOS> token which indicates the beginning of the sequence and the output of the LSTM 1405 is fed to the current unit to produce the next word and recursively the network is unfolded to obtain the output until the end of sentence <EOS> is produced which indicates the end of sequence and the network stops unfolding at the moment) [Ayush: col. 17, line 36-43; Fig. 14] – Note: Ayush described embedded tokens BOS and EOS for a video frame); and combining the plurality of token embeddings with the one or more frame-specific embeddings for each of the sequence of video frames to obtain the plurality of frame-wise token embeddings (The input to feature decoder 1410 starts with (BOS) token
which indicates the beginning of the sequence, and the output of the LSTM unit is fed to the current unit to produce the next word. The network is unfolded recursively to obtain the output until (EOS) is produced, which indicates the end of sequence; the network stops unfolding at this moment. The input word is converted into a word embedding vector by performing a lookup on a word embedding matrix We. The output words are then compared with actual annotated
data, and cross-entropy loss is calculated) [Ayush: col. 17, line 54-63; Fig. 14].
Ayush does not explicitly disclose the following claim limitations (Emphasis added).
encoding the input prompt to obtain a plurality of token embeddings.
However, in the same field of endeavor Geng further discloses the claim limitation and the deficient claim limitation as follows:
encoding the input prompt to obtain a plurality of token embeddings (In the semantics-controlled transformer reasoning 410, the sentence sources that include the dialog history (H), the video caption (C), the query (Q1) and the reference answer (Attn) are embedded together using tokenization and word positional embedding layer) [Geng: col. 15, line 6-26; Fig. 4].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush with Geng to program the system to implement of Geng’s method.
Therefore, the combination of Ayush with Geng will enable the object classifiers of the video system can be used for different types of videos, irrespective of the video context, including the driving application and the detection in sport events [Geng: col. 4, line 18-47].
Regarding claim 3, Ayush meets the claim limitations as set forth in claim 1. Ayush further meets the claim limitations as follow.
performing a cross-attention operation based on an intermediate representation (The video feature extraction component 1310 may include a CNN. In some cases, a CNN may enable processing of digital images with minimal pre-processing. A CNN may be characterized by the use of convolutional (or cross-correlational) hidden layers. These layers apply a convolution operation to the input before signaling the result to the next layer. Each convolutional node may process data for a limited field of input (i.e., the receptive field). During a forward pass of the CNN, filters at each layer may be convolved across the input volume, computing the dot product between the filter and the input. During the training process, the filters may be modified so that they activate when they detect a particular feature within the input) [Ayush: col. 17, line 8-18; Fig. 13] – Note: Ayush discloses that his system uses the cross correlation to generate frames that are contextually consistent with the text prompt while also reflecting the frame-specific variations) and a corresponding a frame-wise token embedding of the plurality of frame-wise token embeddings (embed the encoded video information using an embedding model) [Ayush: col. 15, line 37-38; col. 1, line 39-40; Figs. 9-11, 13-14]; (The input to feature encoder 1400 starts with the beginning of sentence <BOS> token which indicates the beginning of the sequence and the output of the LSTM 1405 is fed to the current unit to produce the next word and recursively the network is unfolded to obtain the output until the end of sentence <EOS> is produced which indicates the end of sequence and the network stops unfolding at the moment) [Ayush: col. 17, line 36-43; Fig. 14]).
In the same field of endeavor, Geng further discloses the cross-attention operation as follows:
performing a cross-attention operation based on an intermediate representation and a corresponding a frame-wise token embedding of the plurality of frame-wise token embeddings (Further, the sequence of scene graph representations 206a-206d corresponds to a sequence of features 208a, features 208b, features 208c and features 208d of the classified objects and determined relationships. The processor 104 is configured to utilize the audio-visual scene aware neural
network 108 to generate the sequence of scene graph representations as described in description of FIG. 1. Each scene graph representation of the sequence of scene graph representations 206a-206d represents the classified objects and the relationships among the objects in each video frame of the sequence of video frames 202a-202d. In the scene graph representation 206a, each node represents a classified object and each edge connecting two nodes represents the relationship among the classified objects. For instance, in the scene graph representation 206a, a subject node ( e.g., freezer) is directed to an object node (e.g., 13 cabinets). Further, the edge connecting the nodes is a directional edge such that the relationship among the objects is specified. The scene graph representation can be represented as, G={(xi, eij, xj) l xi, xj ЄV, eij Єε}, where V is vertex set which consists of single objects, E is edge set which consists of relations linking two objects and (xi, eij, xj) indicate that subject node xi and object node xj are connected by directed relation edge eij) [Geng: col. 9, line 54 – col. 10, line 10; Figs. 2A-3].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush with Geng to program the system to implement of Geng’s method.
Therefore, the combination of Ayush with Geng will enable the object classifiers of the video system can be used for different types of videos, irrespective of the video context, including the driving application and the detection in sport events [Geng: col. 4, line 18-47].
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Ayush (US Patent 11,238,093 B2), (“Ayush”), in view of Geng et al. (US Patent 11,210,523 B2), (“Geng”), in view of Xu et al. (US Patent 11,599,972 B1), (“Xu”).
Regarding claim 4, Ayush meets the claim limitations as set forth in claim 1. Ayush further meets the claim limitations as follow.
performing a diffusion process (The term "distance" between elements of the embedding space is used to refer to whether the representations of two sentences, two videos, or a sentence and a video are similar to each other in the embedding space. The distance may be calculated using the Euclidean distance between two vectors, cosine similarity between vectors, or using any other suitable algorithm for calculating similarity between vectors in a vector space) [Ayush: col. 4, line 6-13]; (A Euclidean distance between the video and sentence
vectors in the embedding space is used as a similarity metric and is denoted by d(x,y). A non-linear embedding using a deep neural network framework can be learned. The joint video-sentence embedding model 1505 has two branches as
shown in the figure, each consisting of fully connected layers with weight matrices W1 and V1, separated by nonlinear rectified linear (ReLU) activation units. Batch normalization is then applied before the last linear layer. Note
that ReLU is not applied after the last layer. The embeddings of videos and sentences are further normalized using L2 norm. Satisfactory results are received when two embedding layers per branch are kept) [Ayush: col. 18, line 19-31]) using the plurality of frame-wise token embeddings as guidance.
Ayush does not explicitly disclose the following claim limitations (Emphasis added).
performing a diffusion process using the plurality of frame-wise token embeddings as guidance.
However, in the same field of endeavor Geng further discloses the claim limitation and the deficient claim limitation as follows:
performing a diffusion process using the plurality of frame-wise token embeddings as guidance (A loss function (L) 326 is implemented between a predicted probability
distribution P of the feature vector 322 and a ground token distribution G of features 324 of reference answers. In each iteration, one word is generated and next word for the response is predicted using a co-attention transformer of the semantic-controlled transformer 300. Further, all next token probability distributions are collected in a batch to obtain the predicted probability distribution P. In a similar manner, ground token distribution G is obtained from
ground truth answers or responses to the query 306. Thus, the semantic-controlled transformer 300 learns the dialog model and generates the responses to the query 306.) [Geng: col. 14, line 10-31; Equation 13; Fig. 3].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush with Geng to program the system to implement of Geng’s method.
Therefore, the combination of Ayush with Geng will enable the object classifiers of the video system can be used for different types of videos, irrespective of the video context, including the driving application and the detection in sport events [Geng: col. 4, line 18-47].
However in the same field of endeavor, Xu further discloses diffusion process as follow:
performing a diffusion process (Diffusion models are a class of generative model, where in the training process, we incrementally add noise to a sample / image, and learn a function (the denoising function), that learns to remove this noise. In the reverse/generative process, we denoise that sample, starting from a sample of a standard normal. Some aspects of diffusion models will not be discussed in detail, such as the forward process or the sampling process, as these are explained in "Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239, 2020" and "Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super resolution via iterative refinement. arXiv preprint arXiv: 2104.07636, 2021" which are hereby incorporated by reference, The application of diffusion models to an AI based compression pipeline as discussed above is set out below. The decoder in the encoder-decoder pipeline may be replicated with a conditional diffusion decoder (CDD). An example of a CDD is described in "Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021". The aim of the CCD when applied in an AI based compression pipeline is to reconstruct the input image given the quantized latents over some number of timesteps T, starting from a sample from a standard normal conditioned with our latent. This is done through iteratively removing noise from the previous sample x, to get x,_1 until we reach x0 , which is our image to be decoded. The initial input to the CCD is a sample from a standard normal, conditioned with upsampled latents) [Xu: col. 7, line 44 – col. 8, line 8; Figs. 3, 6].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush and Geng with Xu to program the system to implement of Xu’s method.
Therefore, the combination of Ayush and Geng with Xu will enable the system to improve the compression process [Xu: col. 6, line 25-27].
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Ayush (US Patent 11,238,093 B2), (“Ayush”), in view of Geng et al. (US Patent 11,210,523 B2), (“Geng”), in view of Xu et al. (US Patent 11,599,972 B1), (“Xu”), in view of Lin (US Patent 7,053,613 B1), (“Lin”).
Regarding claim 5, Ayush and Geng meet the claim limitations as set forth in claim 1. In the same field of endeavor, Xu further discloses:
obtaining a plurality of noise inputs corresponding to the sequence of video frames, respectively (the denoising process may include a denoising function configured
to predict a noise vector; wherein the denoising function receives as input the first input training image with added noise, the data based on the latent representation and parameters describing a noise distribution; and the noise vector is applied to the output of the previous iterative step to obtain the output of the current iterative step. The parameters describing the noise distribution may
specify the variance of the noise distribution. The noise distribution may be a gaussian distribution) [Xu: col. 2, line 1-12; Fig. 3]; and generating a plurality of regularized noise inputs based on the plurality of noise inputs, respectively, wherein the plurality of regularized noise inputs have a temporally regularized distribution (Diffusion models are a class of generative model, where in the training process, we incrementally add noise to a sample/image, and learn a function (the denoising function), that learns to remove this noise. In the reverse/generative process, we denoise that sample, starting from a sample of a standard normal. Some aspects of diffusion models will not be discussed in detail, such as the forward process or the sampling process, as these are explained in "Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239, 2020" and "Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super resolution via iterative refinement. arXiv preprint arXiv: 2104.07636, 2021" which are hereby incorporated by reference, The application of diffusion models to an AI based compression pipeline as discussed above is set out below. The decoder in the encoder-decoder pipeline may be replicated with a conditional diffusion decoder (CDD). An example of a CDD is described in "Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021". The aim of the CCD when applied in an AI based compression pipeline is to reconstruct the input image given the quantized latents over some number of timesteps T, starting from a sample from a standard normal conditioned with our latent. This is done through iteratively removing noise from the previous sample x, to get x,_1 until we reach x0 , which is our image to be decoded. The initial input to the CCD is a sample from a standard normal, conditioned with upsampled
latents) [Xu: col. 7, line 44 – col. 8, line 8; Figs. 3, 6].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush and Geng with Xu to program the system to implement of Xu’s method.
Therefore, the combination of Ayush and Geng with Xu will enable the system to improve the compression process [Xu: col. 6, line 25-27].
In the same field of image processing, Lin further discloses benefits of the temporally regularized distribution as follow:
regularized noise inputs have a temporally regularized distribution (FIG. 7 shows the regularized and non-regularized reconstructed in vivo images and g-factor maps from the 3T scanner using the 8-channel array coil with 2.67-fold acceleration, and 2.00-fold acceleration. The g-factor maps showed noticeable local decreases in the added noise levels of the regularized reconstructed images. Similarly, regularization helped reduce noise in the temporal lobe in 2.67-fold acceleration (middle panel). In 4.00-fold acceleration, regularized reconstruction demonstrated decreased noise in the deep temporal lobe inside insular cortex. TABLE 2 summarizes the g-factor average, standard deviation, and median in the reconstructed anatomical images. As expected, more accelerated acquisitions resulted in higher g-factors in both regularized and non-regularized reconstructions. In 2.00-fold acceleration, g-factor average was suppressed from 1.07 to 0.72 by regularization (a 49% reduction). In 4.00-fold acceleration, g-factor associated noise reduction by regularization is 31% (non-regularized: 2.04, regularized: 1.52). Here, the advantages in SNR due to regularized reconstruction can be appreciated in the temporal lobe of the anatomical images (FIG. 8). In 2.00-fold acceleration, a banded noise region in the non-regularized reconstruction was minimized (FIGS. 7 and 8). The calculated L-curve is shown in FIG. 5 for a representative set of aliased pixels for the 2.0-fold accelerated case) [Lin: col. 8, line 30-54; Figs 7-8; col. 1, line 52-67].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush, Geng, and Xu with Lin to program the system to implement of Lin’s method.
Therefore, the combination of Ayush, Geng, and Xu with Lin will enable the system to improve the image quality [Lin: col. 8, line 30-54; Figs 7-8].
Regarding claim 6, Ayush meets the claim limitations as set forth in claim 1. Ayush further meets the claim limitations as follow.
generating a preliminary noise prediction; and generating a temporally regularized noise prediction based on the preliminary noise prediction and one or more temporally adjacent noise predictions (In some cases, an auto-encoder may learn a representation (i.e., the encoded video information) by training the network to ignore signal noise. This may result in significant dimensionality reduction. The encoded video information may include temporal information based on the temporal information included in the video feature vector. For example, the encoded video information may depend on the order of the video frames, thus capturing important information about the original video. For example, a frame depicting a man holding his chest and then a subsequent
frame showing the man had fallen on the ground may be represented differently than if the frame showing the man on the ground appearing first. A recurrent neural network (RNN)\ including a long short-term memory (LSTM) encoder may be used to capture this temporal information. RNNs have proven to be quite
successful in dealing with vanishing gradient problem over the sequence during training. However, other types of encoder may also be used) [Ayush: col. 10, line 28-46].
Ayush and Geng do not explicitly disclose the following claim limitations (Emphasis added).
generating a preliminary noise prediction;
However in the same field of endeavor, Xu further discloses:
generating a preliminary noise prediction (Diffusion models are a class of generative model, where in the training process, we incrementally add noise to a sample / image, and learn a function (the denoising function), that learns to remove this noise. In the reverse/generative process, we denoise that sample, starting from a sample of a standard normal. Some aspects of diffusion models will not be discussed in detail, such as the forward process or the sampling process, as these are explained in "Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239, 2020" and "Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super resolution via iterative refinement. arXiv preprint arXiv: 2104.07636, 2021" which are hereby incorporated by reference, The application of diffusion models to an AI based compression pipeline as discussed above is set out below. The decoder in the encoder-decoder pipeline may be replicated with a conditional diffusion decoder (CDD). An example of a CDD is described in "Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021". The aim of the CCD when applied in an AI based compression pipeline is to reconstruct the input image given the quantized latents over some number of timesteps T, starting from a sample from a standard normal conditioned with our latent. This is done through iteratively removing noise from the previous sample x, to get x,_1 until we reach x0 , which is our image to be decoded. The initial input to the CCD is a sample from a standard normal, conditioned with upsampled latents) [Xu: col. 7, line 44 – col. 8, line 8; Figs. 3, 6].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush and Geng with Xu to program the system to implement of Xu’s method.
Therefore, the combination of Ayush and Geng with Xu will enable the system to improve the compression process [Xu: col. 6, line 25-27].
In the same field of image processing, Lin further discloses benefits of the temporally regularized distribution as follow:
regularized noise inputs have a temporally regularized distribution (FIG. 7 shows the regularized and non-regularized reconstructed in vivo images and g-factor maps from the 3T scanner using the 8-channel array coil with 2.67-fold acceleration, and 2.00-fold acceleration. The g-factor maps showed noticeable local decreases in the added noise levels of the regularized reconstructed images. Similarly, regularization helped reduce noise in the temporal lobe in 2.67-fold acceleration (middle panel). In 4.00-fold acceleration, regularized reconstruction demonstrated decreased noise in the deep temporal lobe inside insular cortex. TABLE 2 summarizes the g-factor average, standard deviation, and median in the reconstructed anatomical images. As expected, more accelerated acquisitions resulted in higher g-factors in both regularized and non-regularized reconstructions. In 2.00-fold acceleration, g-factor average was suppressed from 1.07 to 0.72 by regularization (a 49% reduction). In 4.00-fold acceleration, g-factor associated noise reduction by regularization is 31% (non-regularized: 2.04, regularized: 1.52). Here, the advantages in SNR due to regularized reconstruction can be appreciated in the temporal lobe of the anatomical images (FIG. 8). In 2.00-fold acceleration, a banded noise region in the non-regularized reconstruction was minimized (FIGS. 7 and 8). The calculated L-curve is shown in FIG. 5 for a representative set of aliased pixels for the 2.0-fold accelerated case) [Lin: col. 8, line 30-54; Figs 7-8; col. 1, line 52-67].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush, Geng, and Xu with Lin to program the system to implement of Lin’s method.
Therefore, the combination of Ayush, Geng, and Xu with Lin will enable the system to improve the image quality [Lin: col. 8, line 30-54; Figs 7-8].
Claims 7-10 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Ayush (US Patent 11,238,093 B2), (“Ayush”), in view of Geng et al. (US Patent 11,210,523 B2), (“Geng”), in view of Orhon et al. (US Patent 11,392,799 B2), (“Orhon”).
Regarding claim 7, Ayush meets the claim limitations as set forth in claim 1. Ayush further meets the claim limitations as follow.
the video generation model is trained using a temporal consistency loss (The encoded video information may include temporal information based on the temporal information included in the video feature vector. For example, the encoded video information may depend on the order of the video frames, thus capturing important information about the original video. For example, a frame depicting a man holding his chest and then a subsequent frame showing the man had fallen on the ground may be represented differently than if the frame showing the man on the ground appearing first. A recurrent neural network (RNN)\ including a long short-term memory (LSTM) encoder may be used to capture this temporal information. RNNs have proven to be quite successful in dealing with vanishing gradient problem over the sequence during training. However, other types of encoder may also be used) [Ayush: col. 10, line 28-46].
Ayush and Geng do not explicitly disclose the following claim limitations (Emphasis added).
a temporal consistency loss;
However in the same field of endeavor, Orhon further discloses:
a temporal consistency loss (A temporal consistency loss term may be determined for each frame based on the current prediction for the first frame and the current predictions for the proceeding frames in the batch) [Orhon: col. 2, line 61-64; Abstract; Figs. 2-5].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush and Geng with Orhon to program the system to implement of Orhon’s method.
Therefore, the combination of Ayush and Geng with Orhon will improve the temporal ccosistent of deep neutral networks [Orhon: Title, col. 5, line 24-33].
Regarding claim 8, Ayush meets the claim limitations as follow.
A method (a method) [Ayush: col. 1, line 29] for training a machine learning model (train the system with video features and corresponding context information) [Ayush: col. 5, line 48-49; Fig. 1], the method comprising: obtaining a training set comprising a video ("video feature vector" refers to a vector of information obtained from a set of video frames. The feature information may include low level information such as edges, corners, blobs, ridges, and colors or high level information such as the presence of a person, plant, animal, vehicle, or building. The video feature vector may also encode meta-data about the video as well as information about the order of the frames. In some cases, each frame
corresponds to a unique set of values within the video feature vector. In some cases, the feature information is represented as a high-dimensional vector, and the meaning) [Ayush: col. 3, line 37-47] and a training prompt describing the video (train the system with video features and corresponding context information) [Ayush: col. 5, line 48-49; Fig. 1]; computing a temporal consistency loss based on the video and the training prompt (The input to feature decoder 1410 starts with (BOS) token which indicates the beginning of the sequence, and the output of the LSTM unit is fed to the current unit to produce the next word. The network is unfolded recursively to obtain the output until (EOS) is produced, which indicates the end of sequence; the network stops unfolding at this moment. The input word is converted into a word embedding vector by performing a lookup on a word embedding matrix We. The output words are then compared with actual annotated data, and cross-entropy loss is calculated) [Ayush: col. 17, line 54-63; Fig. 14]; and training a video generation model to generate a synthesized video from an input prompt (a content video retrieval system may include two phases: First, the oflline training phase to train the system with video features and corresponding context information from the captioned video data and the an online query processing phase for executing a query expressed in natural language and retrieving the top videos (sentence-to-video) or a video as the query and retrieving top sentences describing the context of the input video, where the video context matches with the context of the sentences) [Ayush: col. 5, line 46-55; Fig. 1] based on the temporal consistency loss.
In the same field of endeavor, Geng further discloses about the computing a temporal consistency loss based on the video and the training prompt as follows:
computing a temporal consistency loss based on the video and the training prompt (A loss function (L) 326 is implemented between a predicted probability
distribution P of the feature vector 322 and a ground token distribution G of features 324 of reference answers. In each iteration, one word is generated and next word for the response is predicted using a co-attention transformer of the semantic-controlled transformer 300. Further, all next token probability distributions are collected in a batch to obtain the predicted probability distribution P. In a similar manner, ground token distribution G is obtained from
ground truth answers or responses to the query 306. Thus, the semantic-controlled transformer 300 learns the dialog model and generates the responses to the query 306.) [Geng: col. 14, line 10-31; Equation 13; Fig. 3].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush with Geng to program the system to implement of Geng’s method.
Therefore, the combination of Ayush with Geng will enable the object classifiers of the video system can be used for different types of videos, irrespective of the video context, including the driving application and the detection in sport events [Geng: col. 4, line 18-47].
Ayush and Geng do not explicitly disclose the following claim limitations (Emphasis added).
a temporal consistency loss;
However in the same field of endeavor, Orhon further discloses:
a temporal consistency loss (A temporal consistency loss term may be determined for each frame based on the current prediction for the first frame and the current predictions for the proceeding frames in the batch) [Orhon: col. 2, line 61-64; Abstract; Figs. 2-5].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush and Geng with Orhon to program the system to implement of Orhon’s method.
Therefore, the combination of Ayush and Geng with Orhon will improve the temporal consistent of deep neutral networks [Orhon: Title, col. 5, line 24-33].
Regarding claim 9, Ayush and Geng meet the claim limitations as set forth in claim 8. Geng further meets the claim limitations as follow.
computing a difference in self-attention maps across video frames (In the node-based feature aggregation, for M node features, X = {x1, x2, ... , xM} in a scene graph representation (e.g., the scene graph representation 206a-206d), self-attention
is performed for each pair of linked nodes) [Geng: Please read more detail in col. 11, line 18 – col. 13, line 26; Fig. 2C-3], wherein the temporal consistency loss is based on the difference.
Ayush and Geng do not explicitly disclose the following claim limitations (Emphasis added).
wherein the temporal consistency loss is based on the difference.
However in the same field of endeavor, Orhon further discloses:
wherein the temporal consistency loss is based on the difference (A temporal consistency loss term may be determined for each frame based on the current prediction for the first frame and the current predictions for the proceeding frames in the batch) [Orhon: col. 2, line 61-64; Abstract; Figs. 2-5].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush and Geng with Orhon to program the system to implement of Orhon’s method.
Therefore, the combination of Ayush and Geng with Orhon will improve the temporal consistent of deep neutral networks [Orhon: Title, col. 5, line 24-33].
Regarding claim 10, Ayush meets the claim limitations as set forth in claim 8. Orhon further discloses:
obtaining a positive sample including two video frames of the video (A temporal consistency loss term may be determined for each frame based on the current prediction for the first frame and the current predictions for the proceeding frames in the batch) [Orhon: col. 2, line 61-64; Abstract; Figs. 2-5]; and obtaining a negative sample including a first video frame of the video and a second video frame from a different video (the flowchart continues at 260 and the training module 155 may continue training using a next set of frames, for example from the different video feed) [Orhon: col. 8, line 48-50; Figs. 2-5], wherein the temporal consistency loss is based on the positive sample and the negative sample (the flowchart continues at 260 and the training module 155 may continue training using a next set of frames, for example from the same video feed or a different video feed) [Orhon: col. 8, line 48-50; Figs. 2-5].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush and Geng with Orhon to program the system to implement of Orhon’s method.
Therefore, the combination of Ayush and Geng with Orhon will improve the temporal consistent of deep neutral networks [Orhon: Title, col. 5, line 24-33].
Regarding claim 12, Ayush and Geng meet the claim limitations as set forth in claim 8. Ayush and Geng further meet the claim limitations as follow.
generating a plurality of frame-wise token (Feature encoder 1315 may generate encoded video information based on the video feature vector) [Ayush: col. 15, line 1-2; col. 1, line 35-36; Figs. 3, 7, 13] embeddings corresponding to a sequence of video frames, respectively ((Embedding component 1325 may embed the encoded video information using an embedding model) [Ayush: col. 15, line 37-38; col. 1, line 39-40; Figs. 9-11, 13-14]; (The input to feature encoder 1400 starts with the
beginning of sentence <BOS> token which indicates the beginning of the sequence and the output of the LSTM 1405 is fed to the current unit to produce the next word and recursively the network is unfolded to obtain the output until the end of sentence <EOS> is produced which indicates the end of sequence and the network stops unfolding at the moment) [Ayush: col. 17, line 36-43; Fig. 14] – Note: Ayush discussed an illustration how the embedded tokens are used in the video), based on the training prompt (Generate an encoded video information based on the video feature vector) [Ayush: col. 1, line 35-36; Figs. 3, 7], wherein the temporal consistency loss is generated based on the plurality of frame-wise token embeddings (The input to feature decoder 1410 starts with (BOS) token
which indicates the beginning of the sequence, and the output of the LSTM unit is fed to the current unit to produce the next word. The network is unfolded recursively to obtain the output until (EOS) is produced, which indicates the end of sequence; the network stops unfolding at this moment. The input word is converted into a word embedding vector by performing a lookup on a word embedding matrix We. The output words are then compared with actual annotated
data, and cross-entropy loss is calculated) [Ayush: col. 17, line 54-63; Fig. 14]; (A loss function (L) 326 is implemented between a predicted probability
distribution P of the feature vector 322 and a ground token distribution G of features 324 of reference answers. In each iteration, one word is generated and next word for the response is predicted using a co-attention transformer of the semantic-controlled transformer 300. Further, all next token probability distributions are collected in a batch to obtain the predicted probability distribution P. In a similar manner, ground token distribution G is obtained from
ground truth answers or responses to the query 306. Thus, the semantic-controlled transformer 300 learns the dialog model and generates the responses to the query 306.) [Geng: col. 14, line 10-31; Equation 13; Fig. 3]).
Ayush and Geng do not explicitly disclose the following claim limitations (Emphasis added).
the temporal consistency loss.
However in the same field of endeavor, Orhon further discloses:
the temporal consistency loss (A temporal consistency loss term may be determined for each frame based on the current prediction for the first frame and the current predictions for the proceeding frames in the batch) [Orhon: col. 2, line 61-64; Abstract; Figs. 2-5].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush and Geng with Orhon to program the system to implement of Orhon’s method.
Therefore, the combination of Ayush and Geng with Orhon will improve the temporal consistent of deep neutral networks [Orhon: Title, col. 5, line 24-33].
Claims 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Ayush (US Patent 11,238,093 B2), (“Ayush”), in view of Geng et al. (US Patent 11,210,523 B2), (“Geng”), in view of Orhon et al. (US Patent 11,392,799 B2), (“Orhon”), in view of Xu et al. (US Patent 11,599,972 B1), (“Xu”), in view of Lin (US Patent 7,053,613 B1), (“Lin”).
Regarding claim 13, Ayush, Geng and Orhon meet the claim limitations as set forth in claim 8. In the same field of endeavor, Xu further discloses:
obtaining a plurality of noise inputs corresponding to the sequence of video frames, respectively (the denoising process may include a denoising function configured
to predict a noise vector; wherein the denoising function receives as input the first input training image with added noise, the data based on the latent representation and parameters describing a noise distribution; and the noise vector is applied to the output of the previous iterative step to obtain the output of the current iterative step. The parameters describing the noise distribution may
specify the variance of the noise distribution. The noise distribution may be a gaussian distribution) [Xu: col. 2, line 1-12; Fig. 3]; and generating a plurality of regularized noise inputs based on the plurality of noise inputs, respectively, wherein the plurality of regularized noise inputs have a temporally regularized distribution (Diffusion models are a class of generative model, where in the training process, we incrementally add noise to a sample/image, and learn a function (the denoising function), that learns to remove this noise. In the reverse/generative process, we denoise that sample, starting from a sample of a standard normal. Some aspects of diffusion models will not be discussed in detail, such as the forward process or the sampling process, as these are explained in "Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239, 2020" and "Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super resolution via iterative refinement. arXiv preprint arXiv: 2104.07636, 2021" which are hereby incorporated by reference, The application of diffusion models to an AI based compression pipeline as discussed above is set out below. The decoder in the encoder-decoder pipeline may be replicated with a conditional diffusion decoder (CDD). An example of a CDD is described in "Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021". The aim of the CCD when applied in an AI based compression pipeline is to reconstruct the input image given the quantized latents over some number of timesteps T, starting from a sample from a standard normal conditioned with our latent. This is done through iteratively removing noise from the previous sample x, to get x,_1 until we reach x0 , which is our image to be decoded. The initial input to the CCD is a sample from a standard normal, conditioned with upsampled
latents) [Xu: col. 7, line 44 – col. 8, line 8; Figs. 3, 6].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush, Geng and Orhon with Xu to program the system to implement of Xu’s method.
Therefore, the combination of Ayush, Geng and Orhon with Xu will enable the system to improve the compression process [Xu: col. 6, line 25-27].
In the same field of image processing, Lin further discloses benefits of the temporally regularized distribution as follow:
regularized noise inputs have a temporally regularized distribution (FIG. 7 shows the regularized and non-regularized reconstructed in vivo images and g-factor maps from the 3T scanner using the 8-channel array coil with 2.67-fold acceleration, and 2.00-fold acceleration. The g-factor maps showed noticeable local decreases in the added noise levels of the regularized reconstructed images. Similarly, regularization helped reduce noise in the temporal lobe in 2.67-fold acceleration (middle panel). In 4.00-fold acceleration, regularized reconstruction demonstrated decreased noise in the deep temporal lobe inside insular cortex. TABLE 2 summarizes the g-factor average, standard deviation, and median in the reconstructed anatomical images. As expected, more accelerated acquisitions resulted in higher g-factors in both regularized and non-regularized reconstructions. In 2.00-fold acceleration, g-factor average was suppressed from 1.07 to 0.72 by regularization (a 49% reduction). In 4.00-fold acceleration, g-factor associated noise reduction by regularization is 31% (non-regularized: 2.04, regularized: 1.52). Here, the advantages in SNR due to regularized reconstruction can be appreciated in the temporal lobe of the anatomical images (FIG. 8). In 2.00-fold acceleration, a banded noise region in the non-regularized reconstruction was minimized (FIGS. 7 and 8). The calculated L-curve is shown in FIG. 5 for a representative set of aliased pixels for the 2.0-fold accelerated case) [Lin: col. 8, line 30-54; Figs 7-8; col. 1, line 52-67].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush, Geng, Orhon and Xu with Lin to program the system to implement of Lin’s method.
Therefore, the combination of Ayush, Geng, Orhon and Xu with Lin will enable the system to improve the image quality [Lin: col. 8, line 30-54; Figs 7-8].
Regarding claim 14, Ayush meets the claim limitations as set forth in claim 8. Ayush further meets the claim limitations as follow.
generating a preliminary noise prediction; and generating a temporally regularized noise prediction based on the preliminary noise prediction and one or more temporally adjacent noise predictions (In some cases, an auto-encoder may learn a representation (i.e., the encoded video information) by training the network to ignore signal noise. This may result in significant dimensionality reduction. The encoded video information may include temporal information based on the temporal information included in the video feature vector. For example, the encoded video information may depend on the order of the video frames, thus capturing important information about the original video. For example, a frame depicting a man holding his chest and then a subsequent
frame showing the man had fallen on the ground may be represented differently than if the frame showing the man on the ground appearing first. A recurrent neural network (RNN)\ including a long short-term memory (LSTM) encoder may be used to capture this temporal information. RNNs have proven to be quite
successful in dealing with vanishing gradient problem over the sequence during training. However, other types of encoder may also be used) [Ayush: col. 10, line 28-46].
Ayush and Geng do not explicitly disclose the following claim limitations (Emphasis added).
generating a preliminary noise prediction;
However in the same field of endeavor, Xu further discloses:
generating a preliminary noise prediction (Diffusion models are a class of generative model, where in the training process, we incrementally add noise to a sample / image, and learn a function (the denoising function), that learns to remove this noise. In the reverse/generative process, we denoise that sample, starting from a sample of a standard normal. Some aspects of diffusion models will not be discussed in detail, such as the forward process or the sampling process, as these are explained in "Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239, 2020" and "Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super resolution via iterative refinement. arXiv preprint arXiv: 2104.07636, 2021" which are hereby incorporated by reference, The application of diffusion models to an AI based compression pipeline as discussed above is set out below. The decoder in the encoder-decoder pipeline may be replicated with a conditional diffusion decoder (CDD). An example of a CDD is described in "Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021". The aim of the CCD when applied in an AI based compression pipeline is to reconstruct the input image given the quantized latents over some number of timesteps T, starting from a sample from a standard normal conditioned with our latent. This is done through iteratively removing noise from the previous sample x, to get x,_1 until we reach x0 , which is our image to be decoded. The initial input to the CCD is a sample from a standard normal, conditioned with upsampled latents) [Xu: col. 7, line 44 – col. 8, line 8; Figs. 3, 6].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush and Geng with Xu to program the system to implement of Xu’s method.
Therefore, the combination of Ayush and Geng with Xu will enable the system to improve the compression process [Xu: col. 6, line 25-27].
In the same field of image processing, Lin further discloses benefits of the temporally regularized distribution as follow:
regularized noise inputs have a temporally regularized distribution (FIG. 7 shows the regularized and non-regularized reconstructed in vivo images and g-factor maps from the 3T scanner using the 8-channel array coil with 2.67-fold acceleration, and 2.00-fold acceleration. The g-factor maps showed noticeable local decreases in the added noise levels of the regularized reconstructed images. Similarly, regularization helped reduce noise in the temporal lobe in 2.67-fold acceleration (middle panel). In 4.00-fold acceleration, regularized reconstruction demonstrated decreased noise in the deep temporal lobe inside insular cortex. TABLE 2 summarizes the g-factor average, standard deviation, and median in the reconstructed anatomical images. As expected, more accelerated acquisitions resulted in higher g-factors in both regularized and non-regularized reconstructions. In 2.00-fold acceleration, g-factor average was suppressed from 1.07 to 0.72 by regularization (a 49% reduction). In 4.00-fold acceleration, g-factor associated noise reduction by regularization is 31% (non-regularized: 2.04, regularized: 1.52). Here, the advantages in SNR due to regularized reconstruction can be appreciated in the temporal lobe of the anatomical images (FIG. 8). In 2.00-fold acceleration, a banded noise region in the non-regularized reconstruction was minimized (FIGS. 7 and 8). The calculated L-curve is shown in FIG. 5 for a representative set of aliased pixels for the 2.0-fold accelerated case) [Lin: col. 8, line 30-54; Figs 7-8; col. 1, line 52-67].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush, Geng, Orhon and Xu with Lin to program the system to implement of Lin’s method.
Therefore, the combination of Ayush, Geng, Orhon and Xu with Lin will enable the system to improve the image quality [Lin: col. 8, line 30-54; Figs 7-8].
Claims 15-17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ayush (US Patent 11,238,093 B2), (“Ayush”), in view of Geng et al. (US Patent 11,210,523 B2), (“Geng”), in view of Lin (US Patent 7,053,613 B1), (“Lin”).
Regarding claim 15, Ayush and Geng meet the claim limitations as follow.
An apparatus (an apparatus) [Ayush: col. 1, line 29] comprising: at least one processor ((a processor) [Ayush: col. 6, line 67; Figs. 3, 13]; (a processor unit) [Ayush: col. 14, line 50; Fig. 13]); at least one memory (a memory unit) [Ayush: col. 14, line 50; Fig. 13] including instructions executable by the at least one processor (a processor executing a set of codes to control functional elements of an apparatus) [Ayush: col. 6, line 67 – col. 7, line 7; Figs. 3, 13]; and the apparatus (an apparatus) [Ayush: col. 1, line 29] further comprising a video generation model comprising parameters stored in the at least one memory (recurrent neural network (RNN) encoder-decoder model with Long Short-term Memory (LSTM) cells. The LSTM cells may be used to capture information related to the temporal order of the frames (as represented in different portions of the feature vector)) [Ayush: col. 3, line 53-58] and trained to generate a synthesized video (the audio-visual scene aware neural network 108 is trained to process features (e.g., scene graph representations) that represent scene in each video frame of the sequence of video frames to generate a response) [Geng: col. 7, line 14-17; Fig. 1] based on an input prompt ((a video is represented in the form of text, so a user can input a natural language query where textual descriptions, like the input query, can be retrieved. The sentence can be represented in the form of an encoded vector) [Ayush: col. 6, line 10-14]; (Generate an encoded video information based on the video feature vector) [Ayush: col. 1, line 35-36; Figs. 3, 7]), wherein the video generation model includes a mapping network configured to (neural network (RNN) encoder-decoder model) [Ayush: col. 3, line 54] generate a plurality of regularized noise inputs based on a plurality of noise inputs (In some cases, an auto-encoder may learn a representation (i.e., the encoded video information) by training the network to ignore signal noise. This may result in significant dimensionality reduction. The encoded video information may include temporal information based on the temporal information included in the video feature vector. For example, the encoded video information may depend on the order of the video frames, thus capturing important information about the original video. For example, a frame depicting a man holding his chest and then a subsequent frame showing the man had fallen on the ground may be represented differently than if the frame showing the man on the ground appearing first. A recurrent neural network (RNN)\ including a long short-term memory (LSTM) encoder may be used to capture this temporal information. RNNs have proven to be quite successful in dealing with vanishing gradient problem over the sequence during training. However, other types of encoder may also be used) [Ayush: col. 10, line 28-46], respectively, and wherein the plurality of regularized noise inputs have a temporally regularized distribution.
Ayush and Geng do not explicitly disclose the following claim limitations (Emphasis added).
the plurality of regularized noise inputs have a temporally regularized distribution.
However, in the same field of image processing, Lin further discloses the deficient limitation as follow:
the plurality of regularized noise inputs have a temporally regularized distribution (FIG. 7 shows the regularized and non-regularized reconstructed in vivo images and g-factor maps from the 3T scanner using the 8-channel array coil with 2.67-fold acceleration, and 2.00-fold acceleration. The g-factor maps showed noticeable local decreases in the added noise levels of the regularized reconstructed images. Similarly, regularization helped reduce noise in the temporal lobe in 2.67-fold acceleration (middle panel). In 4.00-fold acceleration, regularized reconstruction demonstrated decreased noise in the deep temporal lobe inside insular cortex. TABLE 2 summarizes the g-factor average, standard deviation, and median in the reconstructed anatomical images. As expected, more accelerated acquisitions resulted in higher g-factors in both regularized and non-regularized reconstructions. In 2.00-fold acceleration, g-factor average was suppressed from 1.07 to 0.72 by regularization (a 49% reduction). In 4.00-fold acceleration, g-factor associated noise reduction by regularization is 31% (non-regularized: 2.04, regularized: 1.52). Here, the advantages in SNR due to regularized reconstruction can be appreciated in the temporal lobe of the anatomical images (FIG. 8). In 2.00-fold acceleration, a banded noise region in the non-regularized reconstruction was minimized (FIGS. 7 and 8). The calculated L-curve is shown in FIG. 5 for a representative set of aliased pixels for the 2.0-fold accelerated case) [Lin: col. 8, line 30-54; Figs 7-8; col. 1, line 52-67].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush and Geng with Lin to program the system to implement of Lin’s method.
Therefore, the combination of Ayush and Geng with Lin will enable the system to improve the image quality [Lin: col. 8, line 30-54; Figs 7-8].
Regarding claim 16, Ayush meets the claim limitations as set forth in claim 15. Ayush further meets the claim limitations as follow.
a text encoder configured to encode the input prompt to obtain a plurality of token embeddings ((The present disclosure provides systems and methods that compare the semantic content of textual queries and video clips. For example, some embodiments of the present disclosure break a video into multiple frames, generate a feature vector from the frames based on the temporal relationship between them, and then embed the feature vector into a vector space along with a vector representing a search query. In some embodiments, the video feature vector is converted into a text caption prior to the embedding. In other embodiments, the video feature vector and a sentence vector are each embedded into a common space using a joint video sentence embedding model.) [Ayush: col. 3, line 12-23]; (Feature encoder 1315 may generate encoded video information based on the video feature vector) [Ayush: col. 15, line 1-2; col. 1, line 35-36; Figs. 3, 7, 13] – Note: The video information can represent tokens. For instance, BOS and EOS are examples of the frame-wise tokens).
Regarding claim 17, Ayush meets the claim limitations as set forth in claim 15. Ayush further meets the claim limitations as follow.
a frame-wise token generator configured to generate a plurality of frame-wise token embeddings corresponding to a sequence of video frames ((some embodiments of the present disclosure break a video into multiple frames, generate a feature vector from the frames based on the temporal relationship between them, and then embed the feature vector into a vector space along with a vector representing a search query. In some embodiments, the video feature vector is converted into a text caption prior to the embedding. In other embodiments, the video feature vector and a sentence vector are each embedded into a common space using a joint video sentence embedding model.) [Ayush: col. 3, line 12-23]; (Feature encoder 1315 may generate encoded video information based on the video feature vector) [Ayush: col. 15, line 1-2; col. 1, line 35-36; Figs. 3, 7, 13]; (The input to feature encoder 1400 starts with the beginning of sentence <BOS> token which indicates the beginning of the sequence and the output of the LSTM 1405 is fed to the current unit to produce the next word and recursively the network is unfolded to obtain the output until the end of sentence <EOS> is produced which indicates the end of sequence and the network stops unfolding at the moment) [Ayush: col. 17, line 36-43; Fig. 14] – Note: – Note: BOS and EOS are examples of the frame-wise tokens).
Regarding claim 20, Ayush and Geng meet the claim limitations as set forth in claim 15. Ayush and Geng further meet the claim limitations as follow.
the video generation model generates the synthesized video using a temporally regularized noise prediction (recurrent neural network (RNN) encoder-decoder model with Long Short-term Memory (LSTM) cells. The LSTM cells may be used to capture information related to the temporal order of the frames (as represented in different portions of the feature vector)) [Ayush: col. 3, line 53-58]; (the audio-visual scene aware neural network 108 is trained to process features (e.g., scene graph representations) that represent scene in each video frame of the sequence of video frames to generate a response) [Geng: col. 7, line 14-17; Fig. 1].
Ayush and Geng do not explicitly disclose the following claim limitations (Emphasis added).
using a temporally regularized noise prediction.
However, in the same field of image processing, Lin further discloses the deficient limitation as follow:
using a temporally regularized noise prediction (FIG. 7 shows the regularized and non-regularized reconstructed in vivo images and g-factor maps from the 3T scanner using the 8-channel array coil with 2.67-fold acceleration, and 2.00-fold acceleration. The g-factor maps showed noticeable local decreases in the added noise levels of the regularized reconstructed images. Similarly, regularization helped reduce noise in the temporal lobe in 2.67-fold acceleration (middle panel). In 4.00-fold acceleration, regularized reconstruction demonstrated decreased noise in the deep temporal lobe inside insular cortex. TABLE 2 summarizes the g-factor average, standard deviation, and median in the reconstructed anatomical images. As expected, more accelerated acquisitions resulted in higher g-factors in both regularized and non-regularized reconstructions. In 2.00-fold acceleration, g-factor average was suppressed from 1.07 to 0.72 by regularization (a 49% reduction). In 4.00-fold acceleration, g-factor associated noise reduction by regularization is 31% (non-regularized: 2.04, regularized: 1.52). Here, the advantages in SNR due to regularized reconstruction can be appreciated in the temporal lobe of the anatomical images (FIG. 8). In 2.00-fold acceleration, a banded noise region in the non-regularized reconstruction was minimized (FIGS. 7 and 8). The calculated L-curve is shown in FIG. 5 for a representative set of aliased pixels for the 2.0-fold accelerated case) [Lin: col. 8, line 30-54; Figs 7-8; col. 1, line 52-67].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush and Geng with Lin to program the system to implement of Lin’s method.
Therefore, the combination of Ayush and Geng with Lin will enable the system to improve the image quality [Lin: col. 8, line 30-54; Figs 7-8].
Claim 18 are rejected under 35 U.S.C. 103 as being unpatentable over Ayush (US Patent 11,238,093 B2), (“Ayush”), in view of Geng et al. (US Patent 11,210,523 B2), (“Geng”), in view of Lin (US Patent 7,053,613 B1), (“Lin”), in view of Xu et al. (US Patent 11,599,972 B1), (“Xu”).
Regarding claim 18, Ayush, Geng, and Lin meet the claim limitations as set forth in claim 15. Xu further meets the claim limitations as follow.
the video generation model includes a diffusion model (Diffusion models are a class of generative model, where in the training process, we incrementally add noise to a sample / image, and learn a function (the denoising function), that learns to remove this noise. In the reverse/generative process, we denoise that sample, starting from a sample of a standard normal. Some aspects of diffusion models will not be discussed in detail, such as the forward process or the sampling process, as these are explained in "Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239, 2020" and "Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super resolution via iterative refinement. arXiv preprint arXiv: 2104.07636, 2021" which are hereby incorporated by reference, The application of diffusion models to an AI based compression pipeline as discussed above is set out below. The decoder in the encoder-decoder pipeline may be replicated with a conditional diffusion decoder (CDD). An example of a CDD is described in "Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021". The aim of the CCD when applied in an AI based compression pipeline is to reconstruct the input image given the quantized latents over some number of timesteps T, starting from a sample from a standard normal conditioned with our latent. This is done through iteratively removing noise from the previous sample x, to get x,_1 until we reach x0 , which is our image to be decoded. The initial input to the CCD is a sample from a standard normal, conditioned with upsampled latents) [Xu: col. 7, line 44 – col. 8, line 8; Figs. 3, 6].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush, Geng, and Lin with Xu to program the system to implement of Xu’s method.
Therefore, the combination of Ayush, Geng, and Lin with Xu will enable the system to improve the compression process [Xu: col. 6, line 25-27].
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Ayush (US Patent 11,238,093 B2), (“Ayush”), in view of Geng et al. (US Patent 11,210,523 B2), (“Geng”), in view of Lin (US Patent 7,053,613 B1), (“Lin”), in view of Orhon et al. (US Patent 11,392,799 B2), (“Orhon”).
Regarding claim 19, Ayush meets the claim limitations as set forth in claim 15. Ayush further meets the claim limitations as follow.
the video generation model is trained using a temporal consistency loss (The encoded video information may include temporal information based on the temporal information included in the video feature vector. For example, the encoded video information may depend on the order of the video frames, thus capturing important information about the original video. For example, a frame depicting a man holding his chest and then a subsequent frame showing the man had fallen on the ground may be represented differently than if the frame showing the man on the ground appearing first. A recurrent neural network (RNN)\ including a long short-term memory (LSTM) encoder may be used to capture this temporal information. RNNs have proven to be quite successful in dealing with vanishing gradient problem over the sequence during training. However, other types of encoder may also be used) [Ayush: col. 10, line 28-46] based on a training video and a training prompt (train the system with video features and corresponding context information) [Ayush: col. 5, line 48-49; Fig. 1].
Ayush, Geng, and Lin do not explicitly disclose the following claim limitations (Emphasis added).
a temporal consistency loss;
However in the same field of endeavor, Orhon further discloses:
a temporal consistency loss (A temporal consistency loss term may be determined for each frame based on the current prediction for the first frame and the current predictions for the proceeding frames in the batch) [Orhon: col. 2, line 61-64; Abstract; Figs. 2-5].
It would have been obvious to one with an ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ayush, Geng, and Lin with Orhon to program the system to implement of Orhon’s method.
Therefore, the combination of Ayush, Geng, and Lin with Orhon will improve the temporal consistent of deep neutral networks [Orhon: Title, col. 5, line 24-33].
Allowable Subject Matter
11. Claim 11 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. This objection is given with a condition that all objections and rejections of related claims are addressed.
Reference Notice
Additional prior arts, included in the Notice of Reference Cited, made of record and not relied upon is considered pertinent to applicant's disclosure.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Philip Dang whose telephone number is (408) 918-7529. The examiner can normally be reached on Monday-Thursday between 8:30 am - 5:00 pm (PST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sath Perungavoor can be reached on 571-272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000./Philip P. Dang/Primary Examiner, Art Unit 2488
Prosecution Insights