Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1 is rejected under 35 U.S.C. 103 as being unpatentable over Shacklett et al. (US 2022/0108421) in view of Xian et al. (Space-time Neural Irradiance Fields for Free-Viewpoint Video, Computer Vision and Pattern Recognition, 2021) in view of Chen et al. (NeRV: Neural Representations for Videos, Computer Vision and Pattern Recognition, 2021) in view of Parke et al. (Class-Incremental Learning for Action Recognition in Videos, Computer Vision and Pattern Recognition, March 2022).
Regarding claim 1, Shacklett et al. (hereinafter Shacklett) discloses a graphics processor (Shacklett, [0045], “graphics processing unit (GPU)”) comprising:
a system interconnect (Shacklett, [0085], “FIG. 8 is a block diagram illustrating an exemplary computer system”); and
a graphics processor cluster coupled with the system interconnect (Shacklett, [0170], “GPGPU 1530 can be linked directly to other instances of GPGPU 1530 to create a multi-GPU cluster to improve training speed for deep neural networks”), the graphics processor cluster including circuitry configured to (Shacklett, [0181], “one or more parallel processor(s) 1612 incorporate circuitry optimized for graphics and video processing, including, for example, video output circuitry, and constitutes a graphics processing unit (GPU)”) generate neural representation of an image (Shacklett, [0046], “this upscaled image 110 can be provided as input to an image reconstruction module 112 that can generate a high resolution, anti-aliased output image 116 using upscaled image 110 and previously generated image 122…image reconstruction module 112 may include one or more neural networks 114 used as part of an image reconstruction process”);
Shacklett does not expressly disclose “neural representations of a multi-view video”;
Xian et al. (hereinafter Xian) discloses a neural representations of a multi-view video (Xian, Fig. 1 illustrates a neural representation (neural radiance field) of a multi-view video).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the video content generation of Shacklett by incorporating the neural radiance field based novel image renderer of Xian. The motivation for doing so would been enabling generation of high-quality images from reference views.
Shacklett as modified by Xian does not expressly disclose “per-frame neural representations”;
Chen et al. (hereinafter Chen) discloses per-frame neural representations (Chen, 3.1 NeRV Architecture, [0001], “where the input is a frame index t and the output is the corresponding RGB image”. As shown in Fig. 2, the neural network receives a frame index as input and generate a corresponding output for that frame, as a result, providing a per-frame neural representation).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to generate the images of Shacklett using the neural representation for videos (NeRV) taught by Chen for encoding and rendering video frames. The motivation for doing so would have been improving image quality and flexibility.
Shacklett as modified by Xian and Chen does not expressly disclose “incremental training and transferal of weights”;
Park et al. (hereinafter Park) discloses incremental training and transferal of weights (Park, 3.2 Overview, [0001], “At the kth incremental step, a set of model parameters…is learned to mimic the feature representation given by the previous model”. Incremental learning that mimics features from a previously trained model is considered incremental training. In addition, Fig. 3 illustrates Distilling knowledge through the estimated importance map makes the model preserve important representations from the previous step. As can be seen in the figure, using model parameters learned in the prior steps is considered transferal of weights from the previously trained model).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Shacklett’s image generation system to generate images using incremental training and knowledge distillation process of Park. The motivation for doing so would have been enabling efficient generation of images while maintaining consistency and quality.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Shacklett et al. (US 2022/0108421) in view of Xian et al. in view of Chen et al. in view of Parke et al., as applied to claim 1, in further view of Zhang et al. (No-reference image quality assessment based on quality patches in real time, Image and Video Processing, 2018).
Regarding claim 2, Shacklett as modified by Xian and Chen with the same motivation from claim 1 discloses adaptively generate the per-frame neural representations (Chen, 3.1 NeRV Architecture, [0001], “where the input is a frame index t and the output is the corresponding RGB image”. As shown in Fig. 2, the neural network receives frames of a video and generates corresponding output frames, which is considered adaptively generate the per-frame neural representations);
Shacklett as modified by Xian, Chen and Park does not expressly disclose “on-the-fly image quality analysis”;
Zhang et al. (hereinafter Zhang) discloses on-the-fly image quality analysis (Zhang, 2.1 Extracting image quality patches, [0002], “get a real-time method of image quality assessment, the image quality is evaluated by using several image patches to represent the image quality”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to generate the neural representation of videos of Shacklett as modified by Xian and Chen based on the real-time image quality assessment method taught by Zhang. The motivation for doing so would have been dynamically evaluate and adjust image generation to improve visual quality.
Claims 3-4 are rejected under 35 U.S.C. 103 as being unpatentable over Shacklett et al. (US 2022/0108421) in view of Xian et al. in view of Chen et al. in view of Parke et al. in view of Zhang et al., as applied to claim 2, in further view of Isikdogan et al. (SemifreddoNets: Partially Frozen Neural Networks for Efficient Computer Vision Systems, Computer Vision and Pattern Recognition, 2020).
Regarding claim 3, Shacklett teaches a frame (Shacklett, [0045], “renderer 102 can receive input for one or more frames of a sequence”); Shacklett as modified by Xian, Chen and Park does not expressly disclose “dynamically partition a neural representation into a fixed portion and a trainable portion”;
Isikdogan et al. (hereinafter Isikdogan) discloses dynamically partition a neural representation into a fixed portion and a trainable portion (Isikdogan, Fig. 2 illustrates Frozen portion and trainable portion. Vertical weight freezing has the flexibility to both adjust to different types of input data and tasks. Many different types of vertical weight freezing schemes can be tailored to different kinds of needs).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Shacklett’s neural network to include the neural network architecture of Isikdogan, which comprises a partially frozen portion and trainable portion. The motivation for doing so would have been improving processing efficiency by reducing complexity while updating other portions of the network.
Regarding claim 4, Shacklett discloses a multilayer perceptron having a plurality of neural network layers (Shacklett, [0244], “neurons 2202 may be organized into one or more layers”. In addition, in paragraph [0360], “neural networks (e.g., auto-encoders, convolutional, recurrent, perceptrons, Long/Short Term Memory (LSTM)”);
Shacklett as modified by Xian, Chen, Park and Isikdogan with the same motivation from claim 3 discloses a first set of neural network layers associated with the fixed portion and a second set of neural network layers associated with the trainable portion (Isikdogan, Fig. 1 illustrates A high-level illustration of how the vertical freezing scheme in SemifreddoNets (right) differs from traditional layer-level parameter freezing approaches).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Shacklett et al. (US 2022/0108421) in view of Xian et al. in view of Chen et al. in view of Parke et al. in view of Zhang et al. in view of Isikdogan et al., as applied to claim 4, in further view of Xu et al. (3D-aware Image Synthesis via Learning Structural and Textural Representations, Computer Vision and Pattern Recognition, 2021).
Regarding claim 5, Shacklett teaches the frame; Shacklett as modified by Xian, Chen, Park and Isikdogan teaches the first set of neural network layers and the second set of neural network layers; Shacklett as modified by Xian, Chen, Park and Isikdogan does not expressly disclose “store color data and structure data”;
Xu et al. (hereinafter Xu) discloses store color data and structure data (Xu, 1 Introduction, [0004], “achieves 3D-aware image synthesis through explicitly learning a structural and a textural representation”. Textural representation including color data).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to configure the multi-layer neural network of Shacklett to store Xu’s structural and textural representation of images. The motivation for doing so would have been enhancing image synthesis quality.
Allowable Subject Matter
Claims 6-10 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYLE ZHAI whose telephone number is (571)270-3740. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ke Xiao can be reached at (571) 272 - 7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KYLE ZHAI/Primary Examiner, Art Unit 2611