Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 04/22/26 have been fully considered but they are not persuasive.
Applicant’s arguments with respect to claims 1, 23 and 28 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4 and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Shacklett et al. (US 2022/0108421) in view of Xian et al. (Space-time Neural Irradiance Fields for Free-Viewpoint Video, Computer Vision and Pattern Recognition, 2021) in view of Chen et al. (NeRV: Neural Representations for Videos, Computer Vision and Pattern Recognition, 2021) in view of Parke et al. (Class-Incremental Learning for Action Recognition in Videos, Computer Vision and Pattern Recognition, March 2022) in view of Isikdogan et al. (SemifreddoNets: Partially Frozen Neural Networks for Efficient Computer Vision Systems, Computer Vision and Pattern Recognition, 2020) in view of Aihara et al. (Motion dense sampling and component clustering for action recognition, Multimed Tools Appl, 2015).
Regarding claim 1, Shacklett et al. (hereinafter Shacklett) discloses a graphics processor (Shacklett, [0045], “graphics processing unit (GPU)”) comprising:
a system interconnect (Shacklett, [0085], “FIG. 8 is a block diagram illustrating an exemplary computer system”); and
a graphics processor cluster coupled with the system interconnect (Shacklett, [0170], “GPGPU 1530 can be linked directly to other instances of GPGPU 1530 to create a multi-GPU cluster to improve training speed for deep neural networks”), the graphics processor cluster including circuitry configured to (Shacklett, [0181], “one or more parallel processor(s) 1612 incorporate circuitry optimized for graphics and video processing, including, for example, video output circuitry, and constitutes a graphics processing unit (GPU)”) generate neural representation of an image (Shacklett, [0046], “this upscaled image 110 can be provided as input to an image reconstruction module 112 that can generate a high resolution, anti-aliased output image 116 using upscaled image 110 and previously generated image 122…image reconstruction module 112 may include one or more neural networks 114 used as part of an image reconstruction process”);
Shacklett does not expressly disclose “neural representations of a multi-view video”;
Xian et al. (hereinafter Xian) discloses a neural representations of a multi-view video (Xian, Fig. 1 illustrates a neural representation (neural radiance field) of a multi-view video).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the video content generation of Shacklett by incorporating the neural radiance field based novel image renderer of Xian. The motivation for doing so would been enabling generation of high-quality images from reference views.
Shacklett as modified by Xian does not expressly disclose “per-frame neural representations”;
Chen et al. (hereinafter Chen) discloses per-frame neural representations (Chen, 3.1 NeRV Architecture, [0001], “where the input is a frame index t and the output is the corresponding RGB image”. As shown in Fig. 2, the neural network receives a frame index as input and generate a corresponding output for that frame, as a result, providing a per-frame neural representation).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to generate the images of Shacklett using the neural representation for videos (NeRV) taught by Chen for encoding and rendering video frames. The motivation for doing so would have been improving image quality and flexibility.
Shacklett as modified by Xian and Chen does not expressly disclose “incremental training and transferal of weights”;
Park et al. (hereinafter Park) discloses incremental training and transferal of weights (Park, 3.2 Overview, [0001], “At the kth incremental step, a set of model parameters…is learned to mimic the feature representation given by the previous model”. Incremental learning that mimics features from a previously trained model is considered incremental training. In addition, Fig. 3 illustrates Distilling knowledge through the estimated importance map makes the model preserve important representations from the previous step. As can be seen in the figure, using model parameters learned in the prior steps is considered transferal of weights from the previously trained model).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Shacklett’s image generation system to generate images using incremental training and knowledge distillation process of Park. The motivation for doing so would have been enabling efficient generation of images while maintaining consistency and quality.
Shacklett teaches a frame (Shacklett, [0045], “receive input for one or more frames of a sequence”); Shacklett as modified by Xian, Chen and Park discloses adaptively generate (Park, 1 Introduction, [0004], “This paper presents a novel framework for classincremental learning for action recognition based on temporally attentive knowledge distillation”);
Shacklett as modified by Xian, Chen and Park does not expressly disclose “dynamically partition a neural representation for the frame into a fixed portion and a trainable portion”;
Isikdogan et al. (hereinafter Isikdogan) discloses dynamically partition a neural representation into a fixed portion and a trainable portion (Isikdogan, Fig. 2 illustrates Frozen portion and trainable portion. Vertical weight freezing has the flexibility to both adjust to different types of input data and tasks. Many different types of vertical weight freezing schemes can be tailored to different kinds of needs);
train the trainable portion for a neural representation (Isikdogan, 3.2 Backbone model architecture, [0001], “whereas the trainable parts are trained separately for each given dataset and task”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Shacklett’s neural network to include the neural network architecture of Isikdogan, which comprises a partially frozen portion and trainable portion. The motivation for doing so would have been improving processing efficiency by reducing complexity while updating other portions of the network.
Shacklett as modified by Xian, Chen, Park and Isikdogan does not expressly disclose “determine, for a frame, regions of the frame that include motion relative to a previous frame”;
Aihara et al. (hereinafter Aihara) discloses determine, for a frame, regions of the frame that include motion relative to a previous frame (Aihara, 3.1 Separation of action region from video frames, [0001], “The features extracted from foreground region are informative for action recognition. In order to achieve the separation, Motion Mask (MM) is automatically generated in our method”. Fig. 3 illustrates regions of the frame that include motion relative to a previous frame);
generate a mask to identify the regions of the frame that include the motion (Aihara, Fig. 3 illustrates motion mask).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the motion-based mask generation of Aihara into the trainable neural network system of Shacklett as modified by Xian, Chen, Park and Isikdogan so that training is performed using detected regions including motion information. The motivation for doing so would have been improving processing accuracy and efficiency.
Regarding claim 4, Shacklett discloses a multilayer perceptron having a plurality of neural network layers (Shacklett, [0244], “neurons 2202 may be organized into one or more layers”. In addition, in paragraph [0360], “neural networks (e.g., auto-encoders, convolutional, recurrent, perceptrons, Long/Short Term Memory (LSTM)”);
Shacklett as modified by Xian, Chen, Park and Isikdogan with the same motivation from claim 3 discloses a first set of neural network layers associated with the fixed portion and a second set of neural network layers associated with the trainable portion (Isikdogan, Fig. 1 illustrates A high-level illustration of how the vertical freezing scheme in SemifreddoNets (right) differs from traditional layer-level parameter freezing approaches).
Regarding claim 28, Shacklett discloses a data processing system (Shacklett, [0004], “FIG. 1 illustrates an image generation system”) comprising:
a memory device (Shacklett, [0394], “a non-transitory computer-readable storage medium stores instructions”); and
a graphics processor coupled with the memory device (Shacklett, Fig. 8).
The remaining limitations recite in claim 28 are similar in scope to the functions recited in claim 1 and therefore are rejected under the same rationale.
Claims 2 and 29 are rejected under 35 U.S.C. 103 as being unpatentable over Shacklett et al. (US 2022/0108421) in view of Xian et al. in view of Chen et al. in view of Parke et al., as applied to claims 1 and 28, in further view of Zhang et al. (No-reference image quality assessment based on quality patches in real time, Image and Video Processing, 2018).
Regarding claim 2, Shacklett as modified by Xian and Chen with the same motivation from claim 1 discloses adaptively generate the per-frame neural representations (Chen, 3.1 NeRV Architecture, [0001], “where the input is a frame index t and the output is the corresponding RGB image”. As shown in Fig. 2, the neural network receives frames of a video and generates corresponding output frames, which is considered adaptively generate the per-frame neural representations);
Shacklett as modified by Xian, Chen and Park does not expressly disclose “on-the-fly image quality analysis”;
Zhang et al. (hereinafter Zhang) discloses on-the-fly image quality analysis (Zhang, 2.1 Extracting image quality patches, [0002], “get a real-time method of image quality assessment, the image quality is evaluated by using several image patches to represent the image quality”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to generate the neural representation of videos of Shacklett as modified by Xian and Chen based on the real-time image quality assessment method as taught by Zhang. The motivation for doing so would have been dynamically evaluate and adjust image generation to improve visual quality.
Regarding claim 29, claim 29 recites function that is similar in scope to the function recited in claim 2 and therefore is rejected under the same rationale.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Shacklett et al. (US 2022/0108421) in view of Xian et al. in view of Chen et al. in view of Parke et al. in view of Zhang et al. in view of Isikdogan et al., as applied to claim 4, in further view of Xu et al. (3D-aware Image Synthesis via Learning Structural and Textural Representations, Computer Vision and Pattern Recognition, 2021).
Regarding claim 5, Shacklett teaches the frame; Shacklett as modified by Xian, Chen, Park and Isikdogan teaches the first set of neural network layers and the second set of neural network layers; Shacklett as modified by Xian, Chen, Park and Isikdogan does not expressly disclose “store color data and structure data”;
Xu et al. (hereinafter Xu) discloses store color data and structure data (Xu, 1 Introduction, [0004], “achieves 3D-aware image synthesis through explicitly learning a structural and a textural representation”. Textural representation including color data).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to configure the multi-layer neural network of Shacklett to store Xu’s structural and textural representation of images. The motivation for doing so would have been enhancing image synthesis quality.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Shacklett et al. (US 2022/0108421) in view of Xian et al. in view of Chen et al. in view of Parke et al. in view of Isikdogan et al. in view of Aihara et al., as applied to claim 4, in further view of in view of Zhi et al. (MGSampler: An Explainable Sampling Strategy for Video Action Recognition, Computer Vision and Pattern Recognition, 2021).
Regarding claim 21, Shacklett as modified by Xian, Chen, Parke, Isikdogan and Aihara with the same motivation from claim 1 discloses the regions of the frame associated with the mask (Aihara, Fig. 9)
Shacklett as modified by Xian, Chen, Parke, Isikdogan and Aihara does not expressly disclose “adjust sampling”;
Zhi et al. (hereinafter Zhi) discloses adjusting sampling (Zhi, 1 Introduction, [0005], “we can perform adaptive sampling over the entire video by randomly picking a representative frame from each segment”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the regions in a frame associated with the mask of Aihara by incorporating the concept of adaptive sampling as taught by Zhi. The motivation for doing so would have been reducing training time and computation cost.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Shacklett et al. (US 2022/0108421) in view of Xian et al. in view of Chen et al. in view of Parke et al. in view of Isikdogan et al. in view of Aihara et al., as applied to claim 4, in further view of in view of Dharur et al. (US 2020/0193609).
Regarding claim 22, Shacklett as modified by Xian, Chen, Parke, Isikdogan and Aihara with the same motivation from claim 1 discloses motion relative to a previous frame based on a determination that an optical flow (Aihara, 3.1 Separation of action region from video frames, [0001], “Detection of a foreground region is done based on optical flow information. To generate MM, we decide how many successive optical flow frames should be used first”);
Shacklett as modified by Xian, Chen, Parke, Isikdogan and Aihara does not expressly disclose “exceeds a threshold”;
Dharur et al. discloses exceeds a threshold (Dharur, [0007], “if the motion vectors between a current frame and a previous frame indicate a change in movement (or an amount of movement) between frames that is above a certain motion threshold, image segmentation can be performed on a target frame”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to determine optical flow between frames of Aihara using motion vectors between frames above a certain threshold as taught by Dharur in order to improve motion estimation accuracy.
Claims 23 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Holte et al. (3D Human Action Recognition for Multi-View Camera Systems, IEEE, 2011) in view of Isikdogan et al. (SemifreddoNets: Partially Frozen Neural Networks for Efficient Computer Vision Systems, Computer Vision and Pattern Recognition, 2020) in view of Aihara et al. (Motion dense sampling and component clustering for action recognition, Multimed Tools Appl, 2015) in view of Zhi et al. (MGSampler: An Explainable Sampling Strategy for Video Action Recognition, Computer Vision and Pattern Recognition, 2021) in view of Shacklett et al. (US 2022/0108421).
Regarding claim 23, Holte et al. (hereinafter Holte) discloses a method (Holte, Fig. 1) comprising:
determining, for a frame of a multi-view video, regions of the frame that include motion relative to a previous frame of the multi-view video (Holte, II Multi-view motion detection, [0001], “We detect motion in Multi-frames…using a 3D version of optical flow to produce velocity annotated point clouds… (3D optical flow). Afterwards we combine the estimated 3D optical flow for each view into a 3D motion vector field”);
Holte discloses a frame (Holte, II Multi-view motion detection, [0001], “We detect motion in Multi-frames”); Holte does not expressly disclose “dynamically partitioning a neural representation into a fixed portion and a trainable portion”;
Isikdogan et al. (hereinafter Isikdogan) discloses dynamically partition a neural representation into a fixed portion and a trainable portion (Isikdogan, Fig. 2 illustrates Frozen portion and trainable portion. Vertical weight freezing has the flexibility to both adjust to different types of input data and tasks. Many different types of vertical weight freezing schemes can be tailored to different kinds of needs);
train the trainable portion for a neural representation (Isikdogan, 3.2 Backbone model architecture, [0001], “whereas the trainable parts are trained separately for each given dataset and task”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the human action recognition system for a multi-view camera environment of Holte to include partially frozen neural networks as taught by Isikdogan. The motivation for doing so would have been improving processing efficiency by reducing complexity while updating other portions of the network.
Holtz as modified by Isikdogan does not expressly disclose “generating a mask to identify the regions of the frame that include the motion”;
Aihara et al. (hereinafter Aihara) discloses generating a mask to identify the regions of the frame that include the motion (Aihara, 3.1 Separation of action region from video frames, [0001], “The features extracted from foreground region are informative for action recognition. In order to achieve the separation, Motion Mask (MM) is automatically generated in our method”. Fig. 3 illustrates regions of the frame that include motion relative to a previous frame);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the motion-based mask generation of Aihara into the trainable neural network system of Holtz as modified by Isikdogan so that training is performed using detected regions including motion information. The motivation for doing so would have been improving processing accuracy and efficiency.
Holtz as modified by Isikdogan and Aihara teaches training to the regions of the frame associated with the mask; Holtz as modified by Isikdogan and Aihara does not expressly disclose “adjusting sampling”;
Zhi et al. (hereinafter Zhi) discloses adjusting sampling (Zhi, 1 Introduction, [0005], “we can perform adaptive sampling over the entire video by randomly picking a representative frame from each segment”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the training process of Isikdogan by incorporating the concept of adaptive sampling as taught by Zhi. The motivation for doing so would have been reducing training time and computation cost.
Holtz as modified by Isikdogan, Aihara and Zhi does not expressly disclose “using processing hardware”;
Shacklett discloses using processing hardware (Shacklett, [0085], “a processor that may include execution units to execute an instruction”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to perform the training of Isikdogan using the hardware processing of Shacklett in order to improve training speed and overall system performance.
Regarding claim 26, Holte as modified by Isikdogan with the same motivation from claim 23 discloses training is performed (Isikdogan, 3.2 Backbone model architecture, [0001], “whereas the trainable parts are trained separately for each given dataset and task”);
Holte as modified by Isikdogan and Aihara with the same motivation from claim 23 discloses applying the mask to select regions of the frame (Aihara, Figs. 9 and 15).
Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over Holte et al. in view of Isikdogan et al. in view of Aihara et al. in view of Zhi et al. in view of Shacklett et al. (US 2022/0108421), as applied to claim 23, in further view of Dharur et al. (US 2020/0193609).
Regarding claim 24, Holte discloses motion comprises determining that an optical flow between the frame and the previous frame (Holte, II Multi-view motion detection, [0001], “We detect motion in Multi-frames…using a 3D version of optical flow to produce velocity annotated point clouds… (3D optical flow). Afterwards we combine the estimated 3D optical flow for each view into a 3D motion vector field”. In addition, in paragraph [0002], “Optical Flow Estimation in Multi-Frames. Optical flow is the pattern of apparent motion in a visual scene caused by the relative motion between an observer and the scene”);
Holte as modified by Isikdogan, Aihara, Zhi and Shacklett does not expressly disclose “exceeds a threshold”;
Dharur et al. discloses exceeds a threshold (Dharur, [0007], “if the motion vectors between a current frame and a previous frame indicate a change in movement (or an amount of movement) between frames that is above a certain motion threshold, image segmentation can be performed on a target frame”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to determine optical flow between frames of Holte using motion vectors between frames above a certain threshold as taught by Dharur in order to improve motion estimation accuracy.
Claim 25 is rejected under 35 U.S.C. 103 as being unpatentable over Holte et al. in view of Isikdogan et al. in view of Aihara et al. in view of Zhi et al. in view of Shacklett et al. (US 2022/0108421), as applied to claim 23, in further view of Xu (US 2005/0058344).
Regarding claim 25, Holte as modified by Isikdogan and Aihara with the same motivation from claim 23 discloses generation of the mask (Aihara, 3.1 Separation of action region from video frames, [0001], “The features extracted from foreground region are informative for action recognition. In order to achieve the separation, Motion Mask (MM) is automatically generated in our method”);
Holte as modified by Isikdogan, Aihara, Zhi and Shacklett does not expressly disclose “enabling or disabling generation of the mask on a per-video or per-frame basis”;
Xu discloses enabling or disabling on a per-frame basis (Xu, [0017], “the segmentations of video frame 20, video frame 25, and video frame 30 are skipped…The number of skipped frames (e.g., video frame 20, video frame 25, video frame 30) may vary, for example, based on the computational power of the computing device performing the video segmentation component 40”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to generate the mask in Aihara using the concept of the skipping function as taught by Xu in order to improve processing efficiency.
Claim 27 is rejected under 35 U.S.C. 103 as being unpatentable over Holte et al. in view of Isikdogan et al. in view of Aihara et al. in view of Zhi et al. in view of Shacklett et al. (US 2022/0108421), as applied to claim 23, in further view of Abramov (US 2020/0380763).
Regarding claim 27, Holte teaches a second frame and a first frame (Holte, II Multi-view motion detection, [0001], “We detect motion in Multi-frames”); Holte as modified by Isikdogan, Aihara, Zhi and Shacklett does not expressly disclose “reducing a number of training iterations based on a rendering quality”;
Abramov discloses reducing a number of training iterations based on a rendering quality (Abramov, [0034], “The machine learning model(s) 110 may be used in an iterative process that predicts successively higher quality denoised images until one or more completion criteria are met…so the uncertainty map 120 may be used to allocate additional samples to pixels with higher uncertainty, as explained in more detail herein”. The process continues until completion criteria are satisfied; the number of iterations is dependent on the rendering quality achieved during the process).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to process the frames of Holte using an iterative process that continues until a predetermined criteria are met, the number of iterations is based on rendering quality as taught by Abramov. The motivation for doing so would have been improving processing efficiency by avoiding unnecessary iterations.
Allowable Subject Matter
Claims 7-10 and 30-32 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYLE ZHAI whose telephone number is (571)270-3740. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ke Xiao can be reached at (571) 272 - 7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KYLE ZHAI/Primary Examiner, Art Unit 2611
Prosecution Insights