DETAILED ACTION
Response to Arguments
Applicant's arguments filed with respect to claims 1-10, 13-18, and 21-24 have been fully considered but are moot in view of the new ground(s) of rejection. The rejections are necessitated due to claim amendments.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 13, and 14 are rejected under 35 U.S.C. 103 as being unpatentable over HUANG et al. (Pub. No. US 2021/0182616) in view of HUA et al. (Pub. No. US 2018/0060698).
Regarding claim 1, HUANG teaches method of training a video data generation neural network having a plurality of video generation network parameters [Para. 50 “training the neural network model may include adjusting values of one or more parameters/operators in the neural network model”], the method comprising: generating one or more sequences of training video frames (intermediate images) using the video data generation neural network in accordance with current values of the video data generation network parameters (nonlinear variation operator) [Para. 24 “Input the plurality of video frames through a neural network model so that the neural network model outputs intermediate images corresponding to the plurality of video frames”; Para. 26 “On each feature conversion layer, the electronic device performs, by using the nonlinear variation operator corresponding to the feature conversion layer, nonlinear variation on pixel values corresponding to pixels included in a feature map output by a previous layer and outputs a feature map on the current feature conversion layer”]; obtaining one or more sequences of target video frames (video frames) [Para. 20 “Obtain a plurality of video frames”]; and training the video data generation neural network (neural network model) using a video data embedding neural network (evaluation network model) configured to generate an embedding (feature map) of a video frame [Para. 64 “obtaining feature maps corresponding to the video frames and feature maps corresponding to the intermediate images, the feature maps being output by layers included in the evaluation network model; and determining the content losses between the intermediate images and the corresponding video frames according to the feature maps corresponding to the intermediate images and the feature maps corresponding to the corresponding video frames”; furthermore Para. 65 states “The evaluation network model is used for extracting an image feature of an input image”], the training comprising: generating a respective embedding (feature map) of each of the training video frames (intermediate images) by processing the training video frame using the video data embedding neural network (evaluation network model) [Para. 64. Furthermore, Para. 91 also states “Input the video frames and the intermediate images corresponding to the video frames into an evaluation network model, so that layers of the evaluation network model outputs feature maps of the video frames and feature maps of the intermediate images; obtain the feature maps of the video frames and the feature maps of the intermediate images; and determine a content loss between the intermediate images and the corresponding video frames according to the feature maps of the intermediate images and the feature maps of the video frames”]; generating a respective embedding (feature map) of each of the target video frames (video frames) by processing the target video frame using the video data embedding neural network [Para. 64. Furthermore, Para. 91 also states “Input the video frames and the intermediate images corresponding to the video frames into an evaluation network model, so that layers of the evaluation network model outputs feature maps of the video frames and feature maps of the intermediate images; obtain the feature maps of the video frames and the feature maps of the intermediate images; and determine a content loss between the intermediate images and the corresponding video frames according to the feature maps of the intermediate images and the feature maps of the video frames”]; determining/calculating a similarity (difference) between the respective embeddings (feature maps) of the training video frames and the respective embeddings of the target video frames [Para. 67 and 93]; and determining an update/adjust to the current values of the video data generation network parameters (non-linear variation operators) based on determining a gradient (changing rate) with respect to the video data generation network parameters of an objective function (training cost) that includes a term that depends on the similarity (difference) [Para. 67 states “After obtaining the feature maps corresponding to the intermediate images and the feature maps corresponding to the input video frames corresponding to the intermediate images, the electronic device calculates a difference between the pixel values of the corresponding pixel positions in the feature maps corresponding to the intermediate images and the feature maps corresponding to the corresponding video frames, to obtain a content difference matrix between the pixel values, and then performs a dimensionality reduction operation on the content difference matrix to obtain the content loss”; furthermore, Para. 92 teaches “Generate a training cost according to the time loss, the feature loss, and the content loss”. Para. 93 also teaches “Determine, according to a reverse sequence of layers included in the neural network model, a changing rate of the training cost with nonlinear variation operators corresponding to the layers; and adjust, according to the reverse sequence, the nonlinear variation operators corresponding to the layers included in the neural network model, so that the changing rate of the training cost with the adjusted nonlinear variation operators corresponding to the layers is reduced”].
However, HUANG doesn’t explicitly teach the rest of claim limitations.
HUA teaches combining the respective embeddings (feature sets) of a set of the training video frames to generate a combined embedding (aggregated feature set) of the set of the training video frames [Para. 15 “The feature sets {f.sub.k} 110 may be provided to the feature aggregator 104. The feature aggregator 104 aggregates the plurality of feature sets {f.sub.k} 110 to generate an aggregated feature set r 114 representing the plurality of feature sets {f.sub.k} 110”]; combining the respective embeddings (feature sets) of a set of the target video frames to generate a combined embedding (aggregated feature set) of the set of the target video frames [Para. 33 “In this case, two video processing systems such as the video processing system 100 of FIG. 1 with shared coefficients may be used. Respective videos may be provided to each of the two video processing systems, and verification decisions may be made.”; Para. 35 “At block 404, a plurality of feature sets may be generated based on the plurality of data sets received at block 402. Respective ones of the feature sets generated at block 404 may include features extracted from respective data sets corresponding to respective ones of the frames of the video.” And Para. 37 “The plurality of weighted feature sets may be combined to generate the aggregated feature set”;]; determining a collective similarity (same/different) between the combined embedding (R_I^L) of the set of the training video frames and the combined embedding (R_J^L) of the set of the target video frames [Para. 27 “the aggregated feature set r generated by the attention block 300 is the output of the feature aggregator”; Para. 33
PNG
media_image1.png
120
408
media_image1.png
Greyscale
].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify HUANG to the claim limitations, feature as taught by HUA; because the modification improves video face recognition by replacing blunt frame pooling method with a learned aggregation network that emphasizes informative frames and suppresses less useful ones, yielding a more accurate and scalable face representation from video.
Regarding claim 3, HUANG teaches wherein the video data generation neural network (neural network model) is configured to generate the training video frame (intermediate image) based on processing an input video frame (video frame; input video frame) in accordance with the current values of the video data generation network parameters (values of one or more parameters/operators in the neural network model; nonlinear variation operators corresponding to the feature conversion layers in the neural networks model) [Para. 50, “training the neural network model may include adjusting values of one or more parameters/operators in the neural network model”; Para. 86 “Respectively process the plurality of video frames through a neural network model so that the neural network model outputs corresponding intermediate images”; Para. 58 “The electronic device may then calculate a changing rate of the training cost with the nonlinear variation operators corresponding to the feature conversion layers in the neural network model, and adjust the nonlinear variation operators corresponding to the feature conversion layers in the neural network model according to the calculated change rate, so that the calculated change rate is reduced, and the neural network model is trained and optimized”].
Claims 13 and 14 are rejected for the same reasons as claim 1 above. Furthermore, HUANG teaches one or more computers, storage mediums/devices storing instructions that when executed by the one or more computers cause the one or more computers to perform the claim limitations [Para. 4-6].
Claims 2, 15 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over HUANG et al. (Pub. No. US 2021/0182616) in view of HUA et al. (Pub. No. US 2018/0060698) in view of Thomas Unterthiner (“Towards Accurate Generative Models of Video: A New Metric & Challeges”).
Regarding claims 2, 15, and 21, HUANG in view of HUA does not explicitly teach the claim the claim limitation.
However, Thomas teaches computing a Frechet Distance (frechet video distance) between the combined embedding (single embedding) of the set of the training video frames and the combined embedding (single embedding) of the set of the target video frame [Section 2 “Finally, the Fr´echet Inception Distance (FID; [14]) is computed according to Eq. 2 using the means and covariances obtained by fitting a multivariate Gaussian distribution to the recorded responses of the real, and generated samples”; “In this work we investigate several variations of a pre trained Inflated 3D Convnet (I3D; [5]), and name the re sulting metric the Fr´echet Video Distance (FVD)” and “we compared to a naive extension of FID for videos in which the Inception network (pre-trained on ImageNet [6]) is evaluated for each frame individually, and the resulting embeddings (or their pair-wise differences) are averaged to obtain a single embedding for each video”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify HUANG in view of HUA to the claim limitations, feature as taught by Thomas; because the modification enables the system to improve evaluation of video generative models by introducing FVD, a metric designed to better capture visual quality, temporal coherence and sample diversity in generated videos.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over HUANG et al. (Pub. No. US 2021/0182616) in view of HUA et al. (Pub. No. US 2018/0060698) in view of Kanumuri et al. (Pub. No. US 2009/0046995).
Regarding claim 4, HUANG in view of HUA doesn’t explicitly teach the claim limitation.
However, Kanumuri teaches wherein the target video frame (upsampled frame y) is an upsampled version of the input video frame (input frame X) [Para. 44 “processing logic upsamples input frame x to obtain upsampled frame y (processing block 201). The upsampling may be performed using an upsampling 2-D filter chosen to derive the upsampled version (y) of input frame x”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify HUANG in view of HUA to the claim limitations, feature as taught by Kanumuri; because the modification enables the system to turn low resolution video into higher resolution pixel adaptive transforms.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over HUANG et al. (Pub. No. US 2021/0182616) in view of HUA et al. (Pub. No. US 2018/0060698) in view of Zalewski (Pub. No. US 2008/0231751).
Regarding claim 5, HUANG in view of HUA doesn’t explicitly teach the claim limitation.
However, Zalewski teaches wherein the target video frame comprises an additional content item compared to the input video frame [Para. 39 and 19].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify HUANG in view of HUA to the claim limitations, feature as taught by Zalewski; because the modification enables the system to automatically decide where/when in a video to insert extra content (like ads) so it fits naturally into the scene.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over HUANG et al. (Pub. No. US 2021/0182616) in view of HUA et al. (Pub. No. US 2018/0060698) in view of Li (Pub. No. US 2020/0252611).
Regarding claim 6, HUANG in view of HUA doesn’t explicitly teach the claim limitation.
However, Li teaches wherein the target video frame (output video data) is a compressed version of the input video frame [Para. 42].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify HUANG in view of HUA to the claim limitations, feature as taught by Li; because the modification enables the system to perform efficient block-level bitrate control in a video encoder by dynamically selecting quantization parameters.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over HUANG et al. (Pub. No. US 2021/0182616) in view of HUA et al. (Pub. No. US 2018/0060698) in view of WANG et al. (Pub. No. US 2019/0325621).
Regarding claim 7, HUANG in view of HUA doesn’t explicitly teach the claim limitation.
However, WANG teaches wherein determining the update to the current values of the video data generation network parameters comprises: back propagating the gradient of the objective function (reconstruction error can then be back-propagated) through video data embedding network parameters (pre-trained VGG network/parameter) of the video data embedding neural network into the video data generation network parameters of video generation neural network (weight of the generator) [Para. 15].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify HUANG in view of HUA to the claim limitations, feature as taught by WANG; because the modification enables the system to reconstruct higher quality tomographic images from a raw reconstructed data.
Claims 8-10, 16-18, and 22-24 are rejected under 35 U.S.C. 103 as being unpatentable over HUANG et al. (Pub. No. US 2021/0182616) in view of HUA et al. (Pub. No. US 2018/0060698) in view of Ji et al. (Pub. No. US 2011/0182469).
Regarding claims 8, 16, and 22 HUANG in view of HUA doesn’t explicitly teach the claim limitation.
However, Ji teaches wherein the video data embedding network is part of a trained video processing neural network (3D CNN model) [Abstract, Para. 6, 20, fig. 1 and corresponding description].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify HUANG in view of HUA to the claim limitations, feature as taught by Ji; because the modification enables the system automatically recognize human actions in real world video using spatio-temporal feature directly from raw video frames.
Regarding claims 9, 17, and 23 HUANG in view of HUA doesn’t explicitly teach the claim limitation.
However, Ji teaches wherein the video processing neural network comprises one or more volumetric convolutional neural network layers (3D CNN architecture) each including a plurality of three- dimensional filters (3D KERNEL) [Para. 17, 14, fig.1 and corresponding description].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify HUANG in view of HUA to the claim limitations, feature as taught by Ji; because the modification enables the system automatically recognize human actions in real world video using spatio-temporal feature directly from raw video frames.
Regarding claims 10, 18, and 24 HUANG in view of HUA doesn’t explicitly teach the claim limitation.
However, Ji teaches wherein the video processing neural network further comprises an output subnetwork configured to generate a video processing network output by processing the embedding generated by the video data embedding neural network, the output subnetwork comprising at least an output layer [fig. 1, 2 and corresponding description].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify HUANG in view of HUA to the claim limitations, feature as taught by Ji; because the modification enables the system automatically recognize human actions in real world video using spatio-temporal feature directly from raw video frames.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOLOMON G BEZUAYEHU whose telephone number is (571)270-7452. The examiner can normally be reached on Monday-Friday 10 AM-8 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 888-786-0101 (IN USA OR CANADA) or 571-272-4000.
/SOLOMON G BEZUAYEHU/
Primary Examiner, Art Unit 2666