Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-4, 8-11, and 15-18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wu et al. (NPL, “Video Quality Assessment Based on Swin Transformer with Spatio-Temporal Feature Fusion and Data Augmentation”, published 8/14/2023) as evidenced by Liu et al. (NPL, “Swin Transformer V2: Scaling Up Capacity and Resolution”, published 2022), hereinafter Liu V2.
Regarding claim 1, Wu teaches a method for evaluating video quality, comprising: performing frame sampling on a target video to obtain a plurality of video frames (Pg. 3, Col. 1, “Given a video whose number of frames and frame rate is N and r, we split the video into M = N/r video segments for feature extraction”); cropping at least one sub-image out of each of the plurality of video frames to obtain a plurality of video sub-images (Pg. 5, Col. 1, “In the testing stage, the input frames are resized to 320 × 320, and the “torchvision.transforms.TenCrop” function is used to crop 10 image patches with a resolution 256 × 256”); inputting the plurality of video sub-images respectively into a quality evaluation model to output quality evaluation sub-information respectively corresponding to the video sub-images, wherein the quality evaluation model comprises a self-attention network with a moving window (Eq. 1, Pg. 3, Col. 1, “For each segment Si (i represents the index of the segment), one frame is randomly sampled from each segment for spatial feature extraction while the whole segment is employed for temporal feature extraction.”; “Frame-level spatial feature is expressed as
PNG
media_image1.png
34
315
media_image1.png
Greyscale
where SFik indicates the extracted spatial features from the k-th sampled frame Fik of segment Si, ⊕ stands for the concatenation operation, GAP(·) represents the global average pooling operation, Lj (Fik) stands for the feature maps obtained from j-th last transformer block of Swin Transformer V2, and Conv1 denotes 1 × 1 convolution operation.”, The cropped frames are evaluated for spatial features to use as quality evaluation sub-information. Further, Wu discloses using Swin Transformer V2, explicitly citing Liu V2 as the foundational blueprint for this architecture (Pg. 3, Col. 1, “We use deep semantic information as a video quality representation by utilizing the features extracted from the last two Transformer blocks of Swin Transformer V2 [23]” (reference 23 is LiuV2; LiuV2 is what one of ordinary skill in the art to mean by “Swin Transformer V2”).), as well as describing that it is an upgraded version of the classical Swin Transformer (Pg. 2, Col. 1). Liu V2 also discloses maintaining “the stage, block, and channel settings of the original Swin Transformer for 4 configurations of Swin Transformer V2”, showing that V2 carries forward the structural constraints of the original Swin Transformer (Liu V1). Fig. 3, (See Liu V1 in pertinent prior art), reprinted below,
PNG
media_image2.png
233
675
media_image2.png
Greyscale
shows that the architecture includes self-attention networks with a moving window); and fusing the quality evaluation sub-information respectively corresponding to the video sub-images to obtain quality evaluation information corresponding to the target video (Fig. 1, reprinted below,
PNG
media_image3.png
504
671
media_image3.png
Greyscale
, shows the sub-information corresponding to video sub-images being fused for quality evaluation, also see Eq. 1 above).
Regarding claim 2, Wu as evidenced by Liu V2 teaches all of the elements of claim 1, as stated above, as well as wherein cropping the at least one sub-image out of each of the plurality of video frames comprises: cropping the plurality of video frames respectively according to a fixed size and a random area scheme to obtain at least one video sub-image corresponding to each video frame (Pg. 4, Col. 2, “In spatial domain, each input frame is resized to 320 × 320 and randomly cropped a patch with a resolution of 256 × 256.”, although directed towards training, cropping video frames according to a fixed size and random area scheme is disclosed).
Regarding claim 3, Wu as evidenced by Liu V2 teaches all of the elements of claim 1, as stated above, as well as wherein the quality evaluation model comprises a plurality of cascaded self-attention networks with moving windows (See Liu V1, Fig. 3, where a plurality of self-attention networks with moving windows are used), a feature merging module and a regression module (Fig. 2), wherein an output of the plurality of cascaded self-attention networks with moving windows is connected with the feature merging module (Fig. 1, reprinted above), and the self-attention networks with moving windows comprise at least two serially connected self-attention blocks with moving windows (See Liu V1, Fig. 3); and the self-attention networks with moving windows are configured for performing feature extraction on input data (Fig. 1, reprinted above), the feature merging module is configured for concatenating features output by the plurality of self-attention networks with moving windows after size transformation of the features (Pg. 3, Col. 2, “Rather than simply merging features from different layers, we introduce a 1 × 1 convolutional layer as shown in Fig. 2 to deepen the spatial features extracted from the intermediate stages of the pre trained network.”), and the regression module is configured for performing regression processing on the concatenated features in terms of the quality evaluation information (Fig. 1, reprinted above).
Regarding claim 4, Wu as evidenced by Liu V2 teaches all of the elements of claim 3, as stated above, as well as wherein the quality evaluation model further comprises: a patch partition module, a linear embedding module, and a patch merging module disposed between two adjacent self-attention networks with moving windows; wherein the patch partition module is connected with the linear embedding module, and the linear embedding module is connected with a first self-attention network with a moving window; and the patch partition module is configured for performing patch partitioning on the input video sub-images, the linear embedding module is configured for performing linear feature embedding on partitioned patches, and the patch merging module is configured for performing patch merging processing on features output by the self-attention networks with moving windows (See Liu V1, Fig. 3, reprinted above, where a patch partition module feeds into a linear embedding module which is connected to a self-attention network with a moving window).
Claim 8 corresponds to claim 1 and is rejected under the same analysis.
Claim 9 corresponds to claim 2 and is rejected under the same analysis.
Claim 10 corresponds to claim 3 and is rejected under the same analysis.
Claim 11 corresponds to claim 4 and is rejected under the same analysis.
Claim 15 corresponds to claim 1 and is rejected under the same analysis.
Claim 16 corresponds to claim 2 and is rejected under the same analysis.
Claim 17 corresponds to claim 3 and is rejected under the same analysis.
Claim 18 corresponds to claim 4 and is rejected under the same analysis.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 8-11, and 15-18 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. (NPL, “Video Quality Assessment Based on Swin Transformer with Spatio-Temporal Feature Fusion and Data Augmentation”, published 8/14/2023) and Liu et al. (NPL, “Swin Transformer V2: Scaling Up Capacity and Resolution”, published 2022), hereinafter Liu V2. As outlined above, Wu et al. appears to explicitly use the Swin Transformer of Liu V2. No apparent rationale is needed other than that the principal reference instructs one of ordinary skiill in the art to use Liu V2 which appears to be the same as the publication referenced. However, to the extent that the description is from two references, it would have been obvious to one of ordinary skill in the art to follow the directions given in the Wu et al. paper and use Liu V2 for its stated purpose.
Claim(s) 5-7, 12-14, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wu as evidenced by Liu V2 in view of Ying et al. (NPL, “Patch-VQ: ‘Patching Up’ the Video Quality Problem”, published 2022).
Regarding claim 5, Wu as evidenced by Liu V2 teaches all of the elements of claim 1, as stated above, as well as wherein a training method of the quality evaluation model comprises: performing frame sampling on a video sample to obtain a plurality of video frame samples; cropping at least one sub-image out of each of the plurality of video frames to obtain a plurality of video sub-image samples; and training, based on the predicted quality evaluation information, the quality evaluation model (Pg. 4, Col. 1, Section 3.3.1, “Training Data Augmentation”, Training is performed, see analysis of claim 1 above).
Wu does not explicitly disclose inputting the plurality of video sub-image samples into a quality evaluation model to output predicted quality evaluation information respectively corresponding to the video sub-image samples. However, they do disclose determine quality evaluation sub-information for each of the sub-image samples.
Ying teaches inputting the plurality of video sub-image samples into a quality evaluation model to output predicted quality evaluation information respectively corresponding to the video sub-image samples (Table 3, Pg. 7, Col. 1, “Table 3 sheds light into the capability of the compared models in predicting local quality. The two PVQ models delivered the best performance on all three types of v-patches”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wu as evidenced by Liu V2 to incorporate the teachings of Ying to include inputting the plurality of video sub-image samples into a quality evaluation model to output predicted quality evaluation information respectively corresponding to the video sub-image samples. Wu discloses a method for video quality evaluation that takes patches of videos for feature extraction and fuses them together to obtain a final quality determination, however they do not perform a quality determination on individual patches. Ying teaches a method for video quality evaluation that uses patches of videos to obtain a local quality determination. Wu also includes Ying as related art. One of ordinary skill in the art would have understood that evaluating the quality of an individual patch provides local information that may be relevant for global quality determinations, as disclosed by Ying (Pg. 2, Col. 1). Implementing the method of Ying into the system of Wu would have been a predictable variation using known techniques.
Regarding claim 6, Wu as modified above teaches all of the elements of claim 5, as stated above as well as wherein sampling the frames for the video sample to obtain the plurality of video frame samples, comprises: performing frame sampling on the sample video with a preset sampling frequency to obtain the plurality of video frame samples (Pg. 2, Col. 1, “Specifically, the input frames are resized and randomly cropped with a fixed resolution, and then randomly extracted from each video segment with a fixed sampling frequency to maintain temporal correlation.”).
Regarding claim 7, Wu as modified above teaches all of the elements of claim 5, as stated above as well as wherein training, based on the predicted quality evaluation information, the quality evaluation model comprises: determining a loss function based on real quality evaluation information and the predicted quality evaluation information of the plurality of video sub-image samples (Ying; Fig. 9, Pg. 6, Col. 1, “The entire dataset of videos, v patches, and human annotations was divided into a training and two test sets.”); and training the quality evaluation model based on the loss function (Ying; Pg. 13, Section C, “For training PVQ (Sec. 4), we used the Adam optimizer…”).
Claim 12 corresponds to claim 5 and is rejected under the same analysis.
Claim 13 corresponds to claim 6 and is rejected under the same analysis.
Claim 14 corresponds to claim 7 and is rejected under the same analysis.
Claim 19 corresponds to claim 5 and is rejected under the same analysis.
Claim 20 corresponds to claim 6 and is rejected under the same analysis.
Conclusion
Pertinent Prior Art: Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows [Review of Swin Transformer: Hierarchical Vision Transformer using Shifted Windows]. arXiv:2103.14030 [Cs]. https://arxiv.org/abs/2103.14030, hereinafter Liu V1. Provides structural details of the neural network architecture used by Wu, as Liu V2 explicitly notes that its core architecture is built upon and remains identical to Liu V1 regarding the relevant limitations.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID A WAMBST whose telephone number is (703)756-1750. The examiner can normally be reached M-F 9-6:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID ALEXANDER WAMBST/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights