DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending regarding this application.
Priority
The present application claims foreign priority benefits from CN202211018116.9 filed on
08/24/2022. The certified copies of the priority documents were electronically retrieved on 04/29/2024.
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/02/2024 and 03/31/2025 are
considered and attached.
Claim Objections
Claim 17 is objected to because of the following informalities:
Claim 17 should have a period at the end of the claim. Please add a “.” to the end of claim 17.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
The term “basic” in claims 3, 4, 11, 12, 19, and 20 is a relative term which renders the claim indefinite. The term “basic” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Applicant’s specification discusses “performing basic feature extraction” as recited in the above claims in para. [0079], [0150]-[0159], and [0231]-[0232]. However, none of these sections clarify how the “feature extraction” as claimed in claims 3, 4, 11, 12, 19, and 20 is “basic”. Therefore, the specification fails to properly define “basic feature extraction”. Furthermore, the term “basic” is a relative term that creates confusion regarding its definition in the context of the above feature extraction. “basic” is not a term commonly used to describe the process of “feature extraction” in the field of image analysis. As such, the specification does not provide a standard for ascertaining the requisite degree of the term “basic” within the phrase “basic feature extraction”, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention term.
Applicant recites basic mobile network(s) in claims 3, 11, and 19. The specification clearly specifies that “a MobileNet-v3-small model is used as a spatial-domain feature extraction model” in para. [0154]. See also para. [0079]. However, it is unclear whether applicant means to imply the term “basic” in the context of the claimed mobile network(s) necessitates a special definition to be read into the claims, wherein the term “basic mobile networks” specifically refers to the basic networks of the MobileNet-v3-small model. Nowhere in the specification does the applicant clarify the relativity of the term “basic” in the context of the mobile networks, nor does the specification define a specific definition of the phrase “basic mobile network”. Additionally, “basic” is not a term commonly used in the field of image analysis to describe mobile networks. As such, the term “basic mobile networks” presents a lack of clarity regarding how the relative term “basic” alters the interpretation of the phrase “basic mobile networks”.
As such, claims 3, 4, 11, 12, 19, and 20 are rejected under 112(b).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 9, 13, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (CN 110837842 A, see attached English translation for citations), hereinafter Zhang, in view of Niu et al. (CN 112954312 A, see attached English translation for citations), hereinafter Niu.
Regarding claim 1, Zhang teaches a video quality assessment method (Zhang teaches “a method for video quality assessment” in para. [0128]), performed by a computer device (Zhang teaches that a computer can carry out the method in para. [0093]), comprising:
playing an online video stream (Zhang teaches extracting a video from a video stream in para. [0295]);
obtaining a time-domain feature of a corresponding unit of duration based on video playback fluency detected within at least one unit of duration in a process of playing the online video stream (Zhang teaches “video frame images can be intercepted from a video stream according to a certain frequency, so that the video to be trained is obtained” in para. [0295], wherein each video is “uniformly sampl[ed] according to a time sequence to form a subset of X” in para. [0266]; see also para. [0129]. Zhang additionally teaches “the feature vector set is input to a time sequence inference module in the video quality assessment model, and a plurality of feature vectors of different scales are output by the time sequence inference module, where the different scales refer to different numbers of image frames corresponding to the scale feature vectors” in para. [0191]-[0192]. See also that “use an obtained feature map as a network input for subsequently extracting the spatial domain features and the temporal domain features” in para. [0181]; here, the individual features within the feature vector are interpreted as equivalent to the time-domain feature(s));
extracting video frames from the online video stream, and separately extracting a spatial-domain feature from each extracted video frame (Zhang teaches “a characteristic diagram sequence is extracted from the video quality evaluation model to be trained, wherein the video to be trained comprises M frames of images, the characteristic diagram sequence comprises M characteristic diagrams, and the characteristic diagrams and the images have corresponding relations” in para. [0296]);
obtaining a time-domain feature vector based on the time-domain feature of the unit of duration (Zhang teaches “generating a time domain feature vector based on the first scale feature vector and the second scale feature vector” in para. [0192], wherein “the timing inference module outputs temporal feature vectors corresponding to the feature vector set” in para. [0134], and the temporal feature vector is interpreted as equivalent to the time-domain feature vector and the time-domain feature vector is based on the scale feature vector, which is the time-domain feature of the unit of duration as further shown in para. [0272]-[0274]. See also para. [0259]), and obtaining a spatial-domain feature vector based on a corresponding spatial-domain feature of each video frame (Zhang teaches “acquiring a space domain characteristic vector corresponding to the characteristic vector set through a video quality evaluation mode” as shown in para. [0187] and “acquiring a characteristic diagram sequence of a video to be evaluated through a convolutional neural network, wherein the characteristic diagram sequence comprises a plurality of characteristic diagrams, the characteristic diagrams and the images have corresponding relations” as shown in para. [0178], wherein the sequence comprises M characteristic diagrams as shown in para. [0057]; see also para. [0133]-[0134] and para. [0202]); and
performing (Zhang teaches “a spatial domain prediction score may be calculated based on the spatial domain feature vector, a temporal domain prediction score may be calculated based on the temporal domain feature vector, the spatial domain prediction score and the temporal domain prediction score are input to a fusion unit in the video quality assessment model, and the fusion unit outputs a target assessment score” in para. [0136], wherein the target assessment score is interpreted as equivalent to the video quality assessment value).
While Zhang teaches a spatial-domain feature vector and the time-domain feature vector and determining a video quality assessment value based on fused spatial-domain and time-domain scores (see above), Zhang fails to teach performing feature fusion processing on the spatial-domain feature vector and the time-domain feature vector to obtain a fusion feature vector, and determining a video quality assessment value of the online video stream based on the fusion feature vector.
However, Niu teaches performing feature fusion processing on the spatial-domain feature vector and the time-domain feature vector to obtain a fusion feature vector, and determining a video quality assessment value of the online video stream based on the fusion feature vector (Niu teaches a method for fusing spatio-temporal characteristics which includes “computing a spatial attention map of spatio-temporal features” as shown in para. [0083] and para. [0087], wherein the fusion result is used to generate a fusion vector Fv which is then used to “obtain[] a sub-video quality score through full connection layer regression” as shown in para. [0088]).
Zhang and Niu are both considered to be analogous to the claimed invention because they are in the same field of video quality assessment through determining spatial and temporal features. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang to incorporate the teachings of Niu and include “performing feature fusion processing on the spatial-domain feature vector and the time-domain feature vector to obtain a fusion feature vector, and determining a video quality assessment value of the online video stream based on the fusion feature vector”. The motivation for doing so would have been to “effectively improve the efficiency and performance of evaluating the quality of the reference-free video”, as suggested by Niu in para. [0008]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang with Niu to obtain the invention specified in claim 1.
Regarding claim 5, Zhang and Niu teach the video quality assessment method according to claim 1, wherein obtaining the spatial-domain feature vector based on the corresponding spatial-domain feature of each video frame comprises:
performing one of the following processing:
performing splicing processing on the corresponding spatial-domain feature of the video frame to obtain the spatial-domain feature vector;
performing averaging processing on the corresponding spatial-domain feature of the video frame to obtain the spatial-domain feature vector (Zhang teaches that “after each frame of image in the video to be evaluated is processed, 10 second feature vectors to be processed can be obtained, and then the 10 second feature vectors to be processed are averaged on each element, so that a space domain feature vector is obtained” in para. [0202]); or
performing pooling processing on the corresponding spatial-domain feature of the video frame to obtain the spatial-domain feature vector.
Note: Only one limitation above need be found in the prior art due to the “one of” and “or” language in the claim.
Regarding claim 9, Zhang teaches a video quality assessment apparatus, comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:
video playback code configured to cause at least one of the at least one processor to play an online video stream (Zhang teaches extracting a video from a video stream in para. [0295]);
time-domain detection code configured to cause at least one of the at least one processor to obtain a time-domain feature of a corresponding unit of duration based on video playback fluency detected within at least one unit of duration in a process of playing the online video stream (Zhang teaches “video frame images can be intercepted from a video stream according to a certain frequency, so that the video to be trained is obtained” in para. [0295], wherein each video is “uniformly sampl[ed] according to a time sequence to form a subset of X” in para. [0266]; see also para. [0129]. Zhang additionally teaches “the feature vector set is input to a time sequence inference module in the video quality assessment model, and a plurality of feature vectors of different scales are output by the time sequence inference module, where the different scales refer to different numbers of image frames corresponding to the scale feature vectors” in para. [0191]-[0192]. See also that “use an obtained feature map as a network input for subsequently extracting the spatial domain features and the temporal domain features” in para. [0181]; here, the individual features within the feature vector are interpreted as equivalent to the time-domain feature(s));
spatial-domain detection code configured to cause at least one of the at least one processor to extract video frames from the online video stream, and separately extract a spatial-domain feature from each extracted video frame, the time-domain detection code being further configured to cause at least one of the at least one processor to obtain a time-domain feature vector based on the time-domain feature of the unit of duration, and the spatial-domain detection code being further configured to cause at least one of the at least one processor to obtain a spatial-domain feature vector based on a corresponding spatial-domain feature of each video frame (Zhang teaches “a characteristic diagram sequence is extracted from the video quality evaluation model to be trained, wherein the video to be trained comprises M frames of images, the characteristic diagram sequence comprises M characteristic diagrams, and the characteristic diagrams and the images have corresponding relations” in para. [0296]); and
quality assessment code configured to cause at least one of the at least one processor to perform (Zhang teaches “a spatial domain prediction score may be calculated based on the spatial domain feature vector, a temporal domain prediction score may be calculated based on the temporal domain feature vector, the spatial domain prediction score and the temporal domain prediction score are input to a fusion unit in the video quality assessment model, and the fusion unit outputs a target assessment score” in para. [0136], wherein the target assessment score is interpreted as equivalent to the video quality assessment value).
While Zhang teaches a spatial-domain feature vector and the time-domain feature vector and determining a video quality assessment value based on fused spatial-domain and time-domain scores (see above), Zhang fails to teach performing feature fusion processing on the spatial-domain feature vector and the time-domain feature vector to obtain a fusion feature vector, and determining a video quality assessment value of the online video stream based on the fusion feature vector.
However, Niu teaches performing feature fusion processing on the spatial-domain feature vector and the time-domain feature vector to obtain a fusion feature vector, and determining a video quality assessment value of the online video stream based on the fusion feature vector (Niu teaches a method for fusing spatio-temporal characteristics which includes “computing a spatial attention map of spatio-temporal features” as shown in para. [0083] and para. [0087], wherein the fusion result is used to generate a fusion vector Fv which is then used to “obtain[] a sub-video quality score through full connection layer regression” as shown in para. [0088]).
Zhang and Niu are both considered to be analogous to the claimed invention because they are in the same field of video quality assessment through determining spatial and temporal features. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang to incorporate the teachings of Niu and include “to perform feature fusion processing on the spatial-domain feature vector and the time-domain feature vector to obtain a fusion feature vector, and determine a video quality assessment value of the online video stream based on the fusion feature vector”. The motivation for doing so would have been to “effectively improve the efficiency and performance of evaluating the quality of the reference-free video”, as suggested by Niu in para. [0008]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang with Niu to obtain the invention specified in claim 9.
Regarding claim 13, Zhang and Niu teach the video quality assessment apparatus according to claim 9, wherein the spatial-domain detection code is further configured to cause at least one of the at least one processor to:
perform one of the following processing:
perform splicing processing on the corresponding spatial-domain feature of the video frame to obtain the spatial-domain feature vector;
perform averaging processing on the corresponding spatial-domain feature of the video frame to obtain the spatial-domain feature vector (Zhang teaches that “after each frame of image in the video to be evaluated is processed, 10 second feature vectors to be processed can be obtained, and then the 10 second feature vectors to be processed are averaged on each element, so that a space domain feature vector is obtained” in para. [0202]); or
perform pooling processing on the corresponding spatial-domain feature of the video frame to obtain the spatial-domain feature vector.
Note: Only one limitation above need be found in the prior art due to the “one of” and “or” language in the claim.
Regarding claim 17, Zhang teaches a non-transitory computer-readable storage medium (Zhang teaches “the memory 520 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device” in para. [0407]), storing computer code which, when executed by at least one processor, causes the at least one processor to (Zhang teaches “the memory 520 may be used to store software programs and modules, and the processor 580 executes various functional applications and data processing of the mobile phone by operating the software programs and modules stored in the memory 520” in para. [0407]) at least:
play an online video stream (Zhang teaches extracting a video from a video stream in para. [0295]);
obtain a time-domain feature of a corresponding unit of duration based on video playback fluency detected within at least one unit of duration in a process of playing the online video stream (Zhang teaches “video frame images can be intercepted from a video stream according to a certain frequency, so that the video to be trained is obtained” in para. [0295], wherein each video is “uniformly sampl[ed] according to a time sequence to form a subset of X” in para. [0266]; see also para. [0129]. Zhang additionally teaches “the feature vector set is input to a time sequence inference module in the video quality assessment model, and a plurality of feature vectors of different scales are output by the time sequence inference module, where the different scales refer to different numbers of image frames corresponding to the scale feature vectors” in para. [0191]-[0192]. See also that “use an obtained feature map as a network input for subsequently extracting the spatial domain features and the temporal domain features” in para. [0181]; here, the individual features within the feature vector are interpreted as equivalent to the time-domain feature(s));
extract video frames from the online video stream, and separately extract a spatial-domain feature from each extracted video frame (Zhang teaches “a characteristic diagram sequence is extracted from the video quality evaluation model to be trained, wherein the video to be trained comprises M frames of images, the characteristic diagram sequence comprises M characteristic diagrams, and the characteristic diagrams and the images have corresponding relations” in para. [0296]);
obtain a time-domain feature vector based on the time-domain feature of the unit of duration (Zhang teaches “generating a time domain feature vector based on the first scale feature vector and the second scale feature vector” in para. [0192], wherein “the timing inference module outputs temporal feature vectors corresponding to the feature vector set” in para. [0134], and the temporal feature vector is interpreted as equivalent to the time-domain feature vector and the time-domain feature vector is based on the scale feature vector, which is the time-domain feature of the unit of duration as further shown in para. [0272]-[0274]. See also para. [0259]), and obtain a spatial-domain feature vector based on a corresponding spatial-domain feature of each video frame (Zhang teaches “acquiring a space domain characteristic vector corresponding to the characteristic vector set through a video quality evaluation mode” as shown in para. [0187] and “acquiring a characteristic diagram sequence of a video to be evaluated through a convolutional neural network, wherein the characteristic diagram sequence comprises a plurality of characteristic diagrams, the characteristic diagrams and the images have corresponding relations” as shown in para. [0178], wherein the sequence comprises M characteristic diagrams as shown in para. [0057]; see also para. [0133]-[0134] and para. [0202]); and
perform (Zhang teaches “a spatial domain prediction score may be calculated based on the spatial domain feature vector, a temporal domain prediction score may be calculated based on the temporal domain feature vector, the spatial domain prediction score and the temporal domain prediction score are input to a fusion unit in the video quality assessment model, and the fusion unit outputs a target assessment score” in para. [0136], wherein the target assessment score is interpreted as equivalent to the video quality assessment value).
While Zhang teaches a spatial-domain feature vector and the time-domain feature vector and determining a video quality assessment value based on fused spatial-domain and time-domain scores (see above), Zhang fails to teach performing feature fusion processing on the spatial-domain feature vector and the time-domain feature vector to obtain a fusion feature vector, and determining a video quality assessment value of the online video stream based on the fusion feature vector.
However, Niu teaches performing feature fusion processing on the spatial-domain feature vector and the time-domain feature vector to obtain a fusion feature vector, and determining a video quality assessment value of the online video stream based on the fusion feature vector (Niu teaches a method for fusing spatio-temporal characteristics which includes “computing a spatial attention map of spatio-temporal features” as shown in para. [0083] and para. [0087], wherein the fusion result is used to generate a fusion vector Fv which is then used to “obtain[] a sub-video quality score through full connection layer regression” as shown in para. [0088]).
Zhang and Niu are both considered to be analogous to the claimed invention because they are in the same field of video quality assessment through determining spatial and temporal features. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang to incorporate the teachings of Niu and include “performing feature fusion processing on the spatial-domain feature vector and the time-domain feature vector to obtain a fusion feature vector, and determining a video quality assessment value of the online video stream based on the fusion feature vector”. The motivation for doing so would have been to “effectively improve the efficiency and performance of evaluating the quality of the reference-free video”, as suggested by Niu in para. [0008]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang with Niu to obtain the invention specified in claim 17.
Claims 2, 10, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (CN 110837842 A, see attached English translation for citations), hereinafter Zhang, in view of Niu et al. (CN 112954312 A, see attached English translation for citations), hereinafter Niu and Zhang (CN 109922334 A, see English translation for citations), hereinafter Zhang ‘334.
Regarding claim 2, Zhang and Niu teach the video quality assessment method according to claim 1, further comprising:
determining (Zhang teaches a video assessment period wherein “the video to be trained comprises M frames of images” in para. [0296]);
wherein obtaining the time-domain feature vector comprises:
based on the quantity of video frames extracted within the video assessment period(Zhang teaches “acquiring a space domain characteristic vector and a time domain characteristic vector according to the first characteristic vector” in para. [0300], wherein the first characteristic vector has M first characteristic vectors as shown in para. [0297]); and
wherein determining the video quality assessment value comprises:
determining the video quality assessment value within the video assessment period based on the fusion feature vector within the video assessment period (While Zhang teaches a video assessment period (as shown above) and determining the video quality assessment value (see claim 1), Niu teaches a method for fusing spatio-temporal characteristics which includes “computing a spatial attention map of spatio-temporal features” as shown in para. [0083] and para. [0087], wherein the fusion result is used to generate a fusion vector Fv which is then used to “obtain[] a sub-video quality score through full connection layer regression” as shown in para. [0088] and “taking the real quality score of each sub-video as the real quality score of the corresponding video” as shown in para. [0067], wherein the sub-video is interpreted as equivalent to the claimed video assessment period). Similar motivations as applied to claim 1 can be applied here.
Zhang and Niu fail to teach determining whether a quantity of video frames extracted within a video assessment period reaches a set quantity threshold.
However, Zhang ‘334 teaches determining whether a quantity of video frames extracted within a video assessment period reaches a set quantity threshold (Zhang ‘334 teaches determining a frame number upper limit value, and determining frame/video features based on the total number of frames reaching and the frame number upper limit value as shown in para. [0040]-[0041]).
Zhang, Niu, and Zhang ‘334 are all considered to be analogous to the claimed invention because they are in the same field of analyzing video quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu) to incorporate the teachings of Zhang ‘334 and include “determining whether a quantity of video frames extracted within a video assessment period reaches a set quantity threshold”. The motivation for doing so would have been to accurately determine whether a target video is low quality view frequently over a specified amount of time, as suggested by Zhang ‘334 in para. [0040]-[0043]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang and Niu with Zhang ‘334 to obtain the invention specified in claim 2.
Regarding claim 10, Zhang and Niu teach the video quality assessment apparatus according to claim 9, wherein the program code further comprises detection code configured to cause at least one of the at least one processor to determine (Zhang teaches a video assessment period wherein “the video to be trained comprises M frames of images” in para. [0296]);
wherein the time-domain detection code is further configured to cause at least one of the at least one processor to, based on the quantity of video frames extracted within the video assessment period(Zhang teaches “acquiring a space domain characteristic vector and a time domain characteristic vector according to the first characteristic vector” in para. [0300], wherein the first characteristic vector has M first characteristic vectors as shown in para. [0297]); and
wherein the quality assessment code is further configured to cause at least one of the at least one processor to determine the video quality assessment value within the video assessment period based on the fusion feature vector within the video assessment period (While Zhang teaches a video assessment period (as shown above) and determining the video quality assessment value (see claim 1), Niu teaches a method for fusing spatio-temporal characteristics which includes “computing a spatial attention map of spatio-temporal features” as shown in para. [0083] and para. [0087], wherein the fusion result is used to generate a fusion vector Fv which is then used to “obtain[] a sub-video quality score through full connection layer regression” as shown in para. [0088] and “taking the real quality score of each sub-video as the real quality score of the corresponding video” as shown in para. [0067], wherein the sub-video is interpreted as equivalent to the claimed video assessment period). Similar motivations as applied to claim 9 can be applied here.
Zhang and Niu fail to teach determining whether a quantity of video frames extracted within a video assessment period reaches a set quantity threshold.
However, Zhang ‘334 teaches determining whether a quantity of video frames extracted within a video assessment period reaches a set quantity threshold (Zhang ‘334 teaches determining a frame number upper limit value, and determining frame/video features based on the total number of frames reaching and the frame number upper limit value as shown in para. [0040]-[0041]).
Zhang, Niu, and Zhang ‘334 are all considered to be analogous to the claimed invention because they are in the same field of analyzing video quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu) to incorporate the teachings of Zhang ‘334 and include “to determine whether a quantity of video frames extracted within a video assessment period reaches a set quantity threshold”. The motivation for doing so would have been to accurately determine whether a target video is low quality view frequently over a specified amount of time, as suggested by Zhang ‘334 in para. [0040]-[0043]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang and Niu with Zhang ‘334 to obtain the invention specified in claim 10.
Regarding claim 18, Zhang and Niu teach the non-transitory computer-readable medium according to claim 17, wherein the computer code further causes the at least one processor to at least:
determine (Zhang teaches a video assessment period wherein “the video to be trained comprises M frames of images” in para. [0296]);
wherein the obtain the time-domain feature vector comprises:
based on the quantity of video frames extracted within the video assessment period(Zhang teaches “acquiring a space domain characteristic vector and a time domain characteristic vector according to the first characteristic vector” in para. [0300], wherein the first characteristic vector has M first characteristic vectors as shown in para. [0297]); and
wherein determining the video quality assessment value comprises:
determining the video quality assessment value within the video assessment period based on the fusion feature vector within the video assessment period (While Zhang teaches a video assessment period (as shown above) and determining the video quality assessment value (see claim 1), Niu teaches a method for fusing spatio-temporal characteristics which includes “computing a spatial attention map of spatio-temporal features” as shown in para. [0083] and para. [0087], wherein the fusion result is used to generate a fusion vector Fv which is then used to “obtain[] a sub-video quality score through full connection layer regression” as shown in para. [0088] and “taking the real quality score of each sub-video as the real quality score of the corresponding video” as shown in para. [0067], wherein the sub-video is interpreted as equivalent to the claimed video assessment period). Similar motivations as applied to claim 17 can be applied here.
Zhang and Niu fail to teach determining whether a quantity of video frames extracted within a video assessment period reaches a set quantity threshold.
However, Zhang ‘334 teaches determining whether a quantity of video frames extracted within a video assessment period reaches a set quantity threshold (Zhang ‘334 teaches determining a frame number upper limit value, and determining frame/video features based on the total number of frames reaching and the frame number upper limit value as shown in para. [0040]-[0041]).
Zhang, Niu, and Zhang ‘334 are all considered to be analogous to the claimed invention because they are in the same field of analyzing video quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu) to incorporate the teachings of Zhang ‘334 and include “determining whether a quantity of video frames extracted within a video assessment period reaches a set quantity threshold”. The motivation for doing so would have been to accurately determine whether a target video is low quality view frequently over a specified amount of time, as suggested by Zhang ‘334 in para. [0040]-[0043]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang and Niu with Zhang ‘334 to obtain the invention specified in claim 18.
Claims 3, 11, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (CN 110837842 A, see attached English translation for citations), hereinafter Zhang, in view of Niu et al. (CN 112954312 A, see attached English translation for citations), hereinafter Niu and Kim et al. (U.S. Publication No. 2022/0309344 A1), hereinafter Kim.
Regarding claim 3, Zhang and Niu teach the video quality assessment method according to claim 1, wherein separately extracting the spatial-domain feature from each extracted video frame comprises:
performing the following processing on any one of the video frames:
performing basic feature extraction on the video frame by using a convolutional neural network to obtain an initial feature map of the video frame (Zhang teaches “a feature map sequence of a video to be evaluated is acquired through a convolutional neural network” in para. [0185]);
inputting the initial feature map to a plurality of consecutive basic mobile networks (Zhang teaches that “in consideration of processing efficiency and accuracy, the light-weight MobileNet-v2 can be adopted as the framework network” in para. [0183], wherein MobileNet is made up of multiple mobile networks, and the feature map sequence is input into the MobileNet as shown in para. [0181]-[0182]);
determining the spatial-domain feature based on a feature map outputted by a last basic mobile network (Zhang teaches “use an obtained feature map as a network input for subsequently extracting the spatial domain features” wherein “the last processing module also includes a Global Average Pooling (GAP) layer” in para. [0181], and “acquiring a feature vector set corresponding to the feature map sequence through a global average pooling GAP layer” in para. [0179] (the feature vector set is interpreted as the feature map outputted by the last basic mobile network, as the GAP layer is part of the last processing module (network)), wherein the feature vector set is the spatial domain feature see also para. [0183] wherein the skeleton network may be MobileNet).
While Zhang teaches “MobileNet-v2 applies Depth-wise separable convolution (DSC) on the network structure instead of the normal convolution, and the amount of computation of the Depth separable convolution can be reduced to about 1/9 of the conventional convolution” in para. [0183]), Zhang and Niu fail to teach performing the following processing by using each basic mobile network of the plurality of consecutive basic mobile networks: performing dimensionality augmentation processing on a feature map inputted to the basic mobile network to obtain a first intermediate feature map, performing depthwise separable convolution processing on the first intermediate feature map to obtain a second intermediate feature map, performing attention mechanism-based processing on the second intermediate feature map to obtain a third intermediate feature map, and performing residual processing on the third intermediate feature map and the feature map inputted to the basic mobile network to obtain a feature map outputted by the basic mobile network.
However, Kim teaches performing the following processing by using each basic mobile network of the plurality of consecutive basic mobile networks:
performing dimensionality augmentation processing on a feature map inputted to the basic mobile network to obtain a first intermediate feature map (Kim teaches that input 105 processed with convolution 110 to generate feature maps 115, which undergo dimension reduction 120, resulting in feature maps 125 (first intermediate feature map); see Fig 1 and para. [0026]-[0030]),
performing depthwise separable convolution processing on the first intermediate feature map to obtain a second intermediate feature map (Kim teaches that feature maps 125 are processed with convolution operation 130 (depthwise separable convolution) to obtain (second intermediate) feature maps 135; see Fig 1 and para. [0031]-[0032]),
performing attention mechanism-based processing on the second intermediate feature map to obtain a third intermediate feature map (Kim teaches that feature maps 135 are broadcasted to the frequency dimension (attention mechanism) 137 to obtain maps 140; Fig 1 and para. [0033]), and
performing residual processing on the third intermediate feature map and the feature map inputted to the basic mobile network to obtain a feature map outputted by the basic mobile network (Kim teaches that a residual connection 150 is used to broadcast the input tensor 105, which is augmented with feature maps 140 by operation (residual processing) 145 to output data 155; see Fig 1 and para. [0034]).
Zhang, Niu, and Kim are all considered to be analogous to the claimed invention because they are in the same field of analyzing audio/visual quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu) to incorporate the teachings of Kim and include “performing the following processing by using each basic mobile network of the plurality of consecutive basic mobile networks: performing dimensionality augmentation processing on a feature map inputted to the basic mobile network to obtain a first intermediate feature map, performing depthwise separable convolution processing on the first intermediate feature map to obtain a second intermediate feature map, performing attention mechanism-based processing on the second intermediate feature map to obtain a third intermediate feature map, and performing residual processing on the third intermediate feature map and the feature map inputted to the basic mobile network to obtain a feature map outputted by the basic mobile network”. The motivation for doing so would have been that “the residual connection 150 allows the system to retain two-dimensional features of the input, despite the dimension reduction operation 120” and “expand temporal output to the frequency dimension”, as suggested by Kim in para. [0034] and para. [0024], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang and Niu with Kim to obtain the invention specified in claim 3.
Regarding claim 11, Zhang and Niu teach the video quality assessment apparatus according to claim 9, wherein the spatial-domain detection code is further configured to cause at least one of the at least one processor to:
perform the following processing on any one of the video frames:
perform basic feature extraction on the video frame by using a convolutional neural network to obtain an initial feature map of the video frame (Zhang teaches “a feature map sequence of a video to be evaluated is acquired through a convolutional neural network” in para. [0185]);
input the initial feature map to a plurality of consecutive basic mobile networks (Zhang teaches that “in consideration of processing efficiency and accuracy, the light-weight MobileNet-v2 can be adopted as the framework network” in para. [0183], wherein MobileNet is made up of multiple mobile networks, and the feature map sequence is input into the MobileNet as shown in para. [0181]-[0182]);
determine the spatial-domain feature based on a feature map outputted by a last basic mobile network (Zhang teaches “use an obtained feature map as a network input for subsequently extracting the spatial domain features” wherein “the last processing module also includes a Global Average Pooling (GAP) layer” in para. [0181], and “acquiring a feature vector set corresponding to the feature map sequence through a global average pooling GAP layer” in para. [0179] (the feature vector set is interpreted as the feature map outputted by the last basic mobile network, as the GAP layer is part of the last processing module (network)), wherein the feature vector set is the spatial domain feature see also para. [0183] wherein the skeleton network may be MobileNet).
While Zhang teaches “MobileNet-v2 applies Depth-wise separable convolution (DSC) on the network structure instead of the normal convolution, and the amount of computation of the Depth separable convolution can be reduced to about 1/9 of the conventional convolution” in para. [0183]), Zhang and Niu fail to teach to perform the following processing by using each basic mobile network of the plurality of consecutive basic mobile networks: perform dimensionality augmentation processing on a feature map inputted to the basic mobile network to obtain a first intermediate feature map, perform depthwise separable convolution processing on the first intermediate feature map to obtain a second intermediate feature map, perform attention mechanism-based processing on the second intermediate feature map to obtain a third intermediate feature map, and perform residual processing on the third intermediate feature map and the feature map inputted to the basic mobile network to obtain a feature map outputted by the basic mobile network.
However, Kim teaches perform the following processing by using each basic mobile network of the plurality of consecutive basic mobile networks:
perform dimensionality augmentation processing on a feature map inputted to the basic mobile network to obtain a first intermediate feature map (Kim teaches that input 105 processed with convolution 110 to generate feature maps 115, which undergo dimension reduction 120, resulting in feature maps 125 (first intermediate feature map); see Fig 1 and para. [0026]-[0030]),
perform depthwise separable convolution processing on the first intermediate feature map to obtain a second intermediate feature map (Kim teaches that feature maps 125 are processed with convolution operation 130 (depthwise separable convolution) to obtain (second intermediate) feature maps 135; see Fig 1 and para. [0031]-[0032]),
perform attention mechanism-based processing on the second intermediate feature map to obtain a third intermediate feature map (Kim teaches that feature maps 135 are broadcasted to the frequency dimension (attention mechanism) 137 to obtain maps 140; Fig 1 and para. [0033]), and
perform residual processing on the third intermediate feature map and the feature map inputted to the basic mobile network to obtain a feature map outputted by the basic mobile network (Kim teaches that a residual connection 150 is used to broadcast the input tensor 105, which is augmented with feature maps 140 by operation (residual processing) 145 to output data 155; see Fig 1 and para. [0034]).
Zhang, Niu, and Kim are all considered to be analogous to the claimed invention because they are in the same field of analyzing audio/visual quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu) to incorporate the teachings of Kim and include to “perform the following processing by using each basic mobile network of the plurality of consecutive basic mobile networks: perform dimensionality augmentation processing on a feature map inputted to the basic mobile network to obtain a first intermediate feature map, perform depthwise separable convolution processing on the first intermediate feature map to obtain a second intermediate feature map, perform attention mechanism-based processing on the second intermediate feature map to obtain a third intermediate feature map, and perform residual processing on the third intermediate feature map and the feature map inputted to the basic mobile network to obtain a feature map outputted by the basic mobile network”. The motivation for doing so would have been that “the residual connection 150 allows the system to retain two-dimensional features of the input, despite the dimension reduction operation 120” and “expand temporal output to the frequency dimension”, as suggested by Kim in para. [0034] and para. [0024], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang and Niu with Kim to obtain the invention specified in claim 11.
Regarding claim 19, Zhang and Niu teach the non-transitory computer-readable storage medium according to claim 17, wherein the separately extract the spatial-domain feature from each extracted video frame comprises:
performing basic feature extraction on the video frame by using a convolutional neural network to obtain an initial feature map of the video frame (Zhang teaches “a feature map sequence of a video to be evaluated is acquired through a convolutional neural network” in para. [0185]);
inputting the initial feature map to a plurality of consecutive basic mobile networks (Zhang teaches that “in consideration of processing efficiency and accuracy, the light-weight MobileNet-v2 can be adopted as the framework network” in para. [0183], wherein MobileNet is made up of multiple mobile networks, and the feature map sequence is input into the MobileNet as shown in para. [0181]-[0182]);
determining the spatial-domain feature based on a feature map outputted by a last basic mobile network (Zhang teaches “use an obtained feature map as a network input for subsequently extracting the spatial domain features” wherein “the last processing module also includes a Global Average Pooling (GAP) layer” in para. [0181], and “acquiring a feature vector set corresponding to the feature map sequence through a global average pooling GAP layer” in para. [0179] (the feature vector set is interpreted as the feature map outputted by the last basic mobile network, as the GAP layer is part of the last processing module (network)), wherein the feature vector set is the spatial domain feature see also para. [0183] wherein the skeleton network may be MobileNet).
While Zhang teaches “MobileNet-v2 applies Depth-wise separable convolution (DSC) on the network structure instead of the normal convolution, and the amount of computation of the Depth separable convolution can be reduced to about 1/9 of the conventional convolution” in para. [0183]), Zhang and Niu fail to teach performing the following processing by using each basic mobile network of the plurality of consecutive basic mobile networks: performing dimensionality augmentation processing on a feature map inputted to the basic mobile network to obtain a first intermediate feature map, performing depthwise separable convolution processing on the first intermediate feature map to obtain a second intermediate feature map, performing attention mechanism-based processing on the second intermediate feature map to obtain a third intermediate feature map, and performing residual processing on the third intermediate feature map and the feature map inputted to the basic mobile network to obtain a feature map outputted by the basic mobile network.
However, Kim teaches performing the following processing by using each basic mobile network of the plurality of consecutive basic mobile networks:
performing dimensionality augmentation processing on a feature map inputted to the basic mobile network to obtain a first intermediate feature map (Kim teaches that input 105 processed with convolution 110 to generate feature maps 115, which undergo dimension reduction 120, resulting in feature maps 125 (first intermediate feature map); see Fig 1 and para. [0026]-[0030]),
performing depthwise separable convolution processing on the first intermediate feature map to obtain a second intermediate feature map (Kim teaches that feature maps 125 are processed with convolution operation 130 (depthwise separable convolution) to obtain (second intermediate) feature maps 135; see Fig 1 and para. [0031]-[0032]),
performing attention mechanism-based processing on the second intermediate feature map to obtain a third intermediate feature map (Kim teaches that feature maps 135 are broadcasted to the frequency dimension (attention mechanism) 137 to obtain maps 140; Fig 1 and para. [0033]), and
performing residual processing on the third intermediate feature map and the feature map inputted to the basic mobile network to obtain a feature map outputted by the basic mobile network (Kim teaches that a residual connection 150 is used to broadcast the input tensor 105, which is augmented with feature maps 140 by operation (residual processing) 145 to output data 155; see Fig 1 and para. [0034]).
Zhang, Niu, and Kim are all considered to be analogous to the claimed invention because they are in the same field of analyzing audio/visual quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu) to incorporate the teachings of Kim and include “performing the following processing by using each basic mobile network of the plurality of consecutive basic mobile networks: performing dimensionality augmentation processing on a feature map inputted to the basic mobile network to obtain a first intermediate feature map, performing depthwise separable convolution processing on the first intermediate feature map to obtain a second intermediate feature map, performing attention mechanism-based processing on the second intermediate feature map to obtain a third intermediate feature map, and performing residual processing on the third intermediate feature map and the feature map inputted to the basic mobile network to obtain a feature map outputted by the basic mobile network”. The motivation for doing so would have been that “the residual connection 150 allows the system to retain two-dimensional features of the input, despite the dimension reduction operation 120” and “expand temporal output to the frequency dimension”, as suggested by Kim in para. [0034] and para. [0024], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang and Niu with Kim to obtain the invention specified in claim 19.
Claims 4, 12, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (CN 110837842 A, see attached English translation for citations), hereinafter Zhang, in view of Niu et al. (CN 112954312 A, see attached English translation for citations), hereinafter Niu, Kim et al. (U.S. Publication No. 2022/0309344 A1), hereinafter Kim, and Zhang et al. (CN 110807757 A, see English translation for citations), hereinafter Zhang ‘757.
Regarding claim 4, Zhang, Niu, and Kim teach the video quality assessment method according to claim 3.
Zhang further teaches wherein the performing basic feature extraction on the video frame by using the convolutional neural network comprises:
obtaining image data (Zhang teaches obtaining image data according to a plurality of RGB channels in para. [0170]); and
performing basic feature extraction on the image data (Zhang teaches acquiring a feature map based on a convolutional neural network in para. [0029]; see also para. [0179]-[0181] wherein this process occurs based on the RGB image).
Zhang, Niu, and Kim fail to teach obtaining image data corresponding to a gray-scale value channel from image data.
However, Zhang ‘757 teaches obtaining image data corresponding to a gray-scale value channel from image data (Zhang ‘757 teaches “the server may convert the first image into a grayscale image, perform Canny edge detection on the grayscale image through two threshold parameters, and output the binarized first gradient feature map” in para. [0130], wherein “the process of solving the image quality score for any image frame is converted into the process of solving the image quality score for the Y-channel [(i.e. grayscale channel), as shown in para. [0157])] image of any image frame, so that chromatic values irrelevant to sharpness can be filtered” as shown in para. [0158]).
Zhang, Niu, Kim, and Zhang ‘757 are all considered to be analogous to the claimed invention because they are in the same field of analyzing audio/visual quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu and Kim) to incorporate the teachings of Zhang ‘757 and include “obtaining image data corresponding to a gray-scale value channel from image data”. The motivation for doing so would have been “so that chromatic values irrelevant to sharpness can be filtered, and a large amount of calculation of the chromatic values is avoided, thereby greatly simplifying the calculated amount of the artificial intelligence-based image quality evaluation process”, as suggested by Zhang ‘757 in para. [0158]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang, Niu, and Kim with Zhang ‘757 to obtain the invention specified in claim 4.
Regarding claim 12, Zhang, Niu, and Kim teach the video quality assessment apparatus according to claim 11.
Zhang further teaches wherein the spatial-domain detection code is further configured to cause at least one of the at least one processor to:
obtain image data (Zhang teaches obtaining image data according to a plurality of RGB channels in para. [0170]); and
perform basic feature extraction on the image data (Zhang teaches acquiring a feature map based on a convolutional neural network in para. [0029]; see also para. [0179]-[0181] wherein this process occurs based on the RGB image).
Zhang, Niu, and Kim fail to teach obtain image data corresponding to a gray-scale value channel from image data.
However, Zhang ‘757 teaches to obtain image data corresponding to a gray-scale value channel from image data (Zhang ‘757 teaches “the server may convert the first image into a grayscale image, perform Canny edge detection on the grayscale image through two threshold parameters, and output the binarized first gradient feature map” in para. [0130], wherein “the process of solving the image quality score for any image frame is converted into the process of solving the image quality score for the Y-channel [(i.e. grayscale channel), as shown in para. [0157])] image of any image frame, so that chromatic values irrelevant to sharpness can be filtered” as shown in para. [0158]).
Zhang, Niu, Kim, and Zhang ‘757 are all considered to be analogous to the claimed invention because they are in the same field of analyzing audio/visual quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu and Kim) to incorporate the teachings of Zhang ‘757 and include to “obtain image data corresponding to a gray-scale value channel from image data”. The motivation for doing so would have been “so that chromatic values irrelevant to sharpness can be filtered, and a large amount of calculation of the chromatic values is avoided, thereby greatly simplifying the calculated amount of the artificial intelligence-based image quality evaluation process”, as suggested by Zhang ‘757 in para. [0158]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang, Niu, and Kim with Zhang ‘757 to obtain the invention specified in claim 12.
Regarding claim 20, Zhang, Niu, and Kim teach the non-transitory computer-readable storage medium according to claim 19.
Zhang further teaches wherein the perform basic feature extraction on the video frame by using the convolutional neural network comprises:
obtaining image data (Zhang teaches obtaining image data according to a plurality of RGB channels in para. [0170]); and
performing basic feature extraction on the image data (Zhang teaches acquiring a feature map based on a convolutional neural network in para. [0029]; see also para. [0179]-[0181] wherein this process occurs based on the RGB image).
Zhang, Niu, and Kim fail to teach obtaining image data corresponding to a gray-scale value channel from image data.
However, Zhang ‘757 teaches obtaining image data corresponding to a gray-scale value channel from image data (Zhang ‘757 teaches “the server may convert the first image into a grayscale image, perform Canny edge detection on the grayscale image through two threshold parameters, and output the binarized first gradient feature map” in para. [0130], wherein “the process of solving the image quality score for any image frame is converted into the process of solving the image quality score for the Y-channel [(i.e. grayscale channel), as shown in para. [0157])] image of any image frame, so that chromatic values irrelevant to sharpness can be filtered” as shown in para. [0158]).
Zhang, Niu, Kim, and Zhang ‘757 are all considered to be analogous to the claimed invention because they are in the same field of analyzing audio/visual quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu and Kim) to incorporate the teachings of Zhang ‘757 and include “obtaining image data corresponding to a gray-scale value channel from image data”. The motivation for doing so would have been “so that chromatic values irrelevant to sharpness can be filtered, and a large amount of calculation of the chromatic values is avoided, thereby greatly simplifying the calculated amount of the artificial intelligence-based image quality evaluation process”, as suggested by Zhang ‘757 in para. [0158]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang, Niu, and Kim with Zhang ‘757 to obtain the invention specified in claim 20.
Claims 6 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (CN 110837842 A, see attached English translation for citations), hereinafter Zhang, in view of Niu et al. (CN 112954312 A, see attached English translation for citations), hereinafter Niu and Wu et al. (U.S. Publication No. 2015/0348251 A1), hereinafter Wu.
Regarding claim 6, Zhang and Niu teach the video quality assessment method according to claim 1.
While Zhang teaches wherein obtaining the time-domain feature of the corresponding unit of duration based on video playback fluency detected within at least one unit of duration (see claim 1) and “the time domain distortion features which are common in real-time video call and can describe motion blur and the like caused by frame blocking, sudden brightness change and severe jitter are effectively extracted through the time sequence reasoning module” as shown in para. [0123], and marking the unit of duration based on the video playback fluency as shown in para. [0200]-[0201], Zhang and Niu fail to teach wherein obtaining the time-domain feature of the corresponding unit of duration based on video playback fluency detected within at least one unit of duration comprises: performing the following processing on each unit of duration: determining the video playback fluency within the unit of duration based on a difference between a video frame played in the unit of duration and a video frame played in at least one historical unit of duration closest to the unit of duration; and marking the unit of duration based on the video playback fluency to obtain the time-domain feature of the corresponding unit of duration, the time-domain feature of the unit of duration being marked as a first value based on a determination that the video playback fluency is smooth, and the time-domain feature of the unit of duration being marked as a second value based on a determination that the video playback fluency is freezing.
However, Wu teaches wherein obtaining the time-domain feature of the corresponding unit of duration based on video playback fluency detected within at least one unit of duration comprises:
performing the following processing on each unit of duration:
determining the video playback fluency within the unit of duration based on a difference between a video frame played in the unit of duration and a video frame played in at least one historical unit of duration closest to the unit of duration (Wu teaches determining video playback fluency of a current frame (i.e. the unit of duration) based on a difference between a current frame and a previous frame as described in para. [0091] and FIGs. 3-5); and
marking the unit of duration based on the video playback fluency to obtain the time-domain feature of the corresponding unit of duration (Wu teaches determining flags for a frame wherein the flags represent video playback fluency for a specific frame, wherein the flag can be interpreted as equivalent to the claimed time-domain feature in para. [0084]), the time-domain feature of the unit of duration being marked as a first value based on a determination that the video playback fluency is smooth (Wu teaches “Ffrz , Fjit and Fgst are respectively flag bits of a frozen frame, a jitter frame and a ghosting frame of a current frame, and one and only one of the three flag bits is 1, and other flag bits are all 0, 1 representing that there is a corresponding type of distortion in an evaluated video frame, and 0 representing that there is no corresponding type of distortion in the evaluated video frame” in para. [0084], wherein the time-domain feature of the unit of duration is marked as 0 when the frame is not a freeze frame (i.e. when the playback fluency is smooth)), and the time-domain feature of the unit of duration being marked as a second value based on a determination that the video playback fluency is freezing (Wu teaches “Ffrz , Fjit and Fgst are respectively flag bits of a frozen frame, a jitter frame and a ghosting frame of a current frame, and one and only one of the three flag bits is 1, and other flag bits are all 0, 1 representing that there is a corresponding type of distortion in an evaluated video frame, and 0 representing that there is no corresponding type of distortion in the evaluated video frame” in para. [0084], wherein the time-domain feature of the unit of duration is marked as 1 when the frame is a freeze frame (i.e. when the playback fluency is freezing)).
Zhang, Niu, and Wu are all considered to be analogous to the claimed invention because they are in the same field of analyzing video quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu) to incorporate the teachings of Wu and include “wherein obtaining the time-domain feature of the corresponding unit of duration based on video playback fluency detected within at least one unit of duration comprises: performing the following processing on each unit of duration: determining the video playback fluency within the unit of duration based on a difference between a video frame played in the unit of duration and a video frame played in at least one historical unit of duration closest to the unit of duration; and marking the unit of duration based on the video playback fluency to obtain the time-domain feature of the corresponding unit of duration, the time-domain feature of the unit of duration being marked as a first value based on a determination that the video playback fluency is smooth, and the time-domain feature of the unit of duration being marked as a second value based on a determination that the video playback fluency is freezing”. The motivation for doing so would have been to that “the quality of the video in time domain is evaluated according to this technical index in the embodiment of the disclosure, and the influence of the moveability on the quality of the video is measured by analysing the attribute of the index, thereby improving the accuracy of the evaluation”, as suggested by Wu in para. [0044]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang and Niu with Wu to obtain the invention specified in claim 6.
Regarding claim 14, Zhang and Niu teach the video quality assessment apparatus according to claim 9, wherein the time-domain detection code is further configured to cause at least one of the at least one processor to:.
While Zhang teaches “the time domain distortion features which are common in real-time video call and can describe motion blur and the like caused by frame blocking, sudden brightness change and severe jitter are effectively extracted through the time sequence reasoning module” as shown in para. [0123], and marking the unit of duration based on the video playback fluency as shown in para. [0200]-[0201], Zhang and Niu fail to teach perform the following processing on each unit of duration: determine the video playback fluency within the unit of duration based on a difference between a video frame played in the unit of duration and a video frame played in at least one historical unit of duration closest to the unit of duration; and mark the unit of duration based on the video playback fluency to obtain the time-domain feature of the corresponding unit of duration, the time-domain feature of the unit of duration being marked as a first value based on a determination that the video playback fluency is smooth, and the time-domain feature of the unit of duration being marked as a second value based on a determination that the video playback fluency is freezing.
However, Wu teaches to:
perform the following processing on each unit of duration:
determine the video playback fluency within the unit of duration based on a difference between a video frame played in the unit of duration and a video frame played in at least one historical unit of duration closest to the unit of duration (Wu teaches determining video playback fluency of a current frame (i.e. the unit of duration) based on a difference between a current frame and a previous frame as described in para. [0091] and FIGs. 3-5); and
mark the unit of duration based on the video playback fluency to obtain the time-domain feature of the corresponding unit of duration (Wu teaches determining flags for a frame wherein the flags represent video playback fluency for a specific frame, wherein the flag can be interpreted as equivalent to the claimed time-domain feature in para. [0084]), the time-domain feature of the unit of duration being marked as a first value based on a determination that the video playback fluency is smooth (Wu teaches “Ffrz , Fjit and Fgst are respectively flag bits of a frozen frame, a jitter frame and a ghosting frame of a current frame, and one and only one of the three flag bits is 1, and other flag bits are all 0, 1 representing that there is a corresponding type of distortion in an evaluated video frame, and 0 representing that there is no corresponding type of distortion in the evaluated video frame” in para. [0084], wherein the time-domain feature of the unit of duration is marked as 0 when the frame is not a freeze frame (i.e. when the playback fluency is smooth)), and the time-domain feature of the unit of duration being marked as a second value based on a determination that the video playback fluency is freezing (Wu teaches “Ffrz , Fjit and Fgst are respectively flag bits of a frozen frame, a jitter frame and a ghosting frame of a current frame, and one and only one of the three flag bits is 1, and other flag bits are all 0, 1 representing that there is a corresponding type of distortion in an evaluated video frame, and 0 representing that there is no corresponding type of distortion in the evaluated video frame” in para. [0084], wherein the time-domain feature of the unit of duration is marked as 1 when the frame is a freeze frame (i.e. when the playback fluency is freezing)).
Zhang, Niu, and Wu are all considered to be analogous to the claimed invention because they are in the same field of analyzing video quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu) to incorporate the teachings of Wu and include “perform the following processing on each unit of duration: determine the video playback fluency within the unit of duration based on a difference between a video frame played in the unit of duration and a video frame played in at least one historical unit of duration closest to the unit of duration; and mark the unit of duration based on the video playback fluency to obtain the time-domain feature of the corresponding unit of duration, the time-domain feature of the unit of duration being marked as a first value based on a determination that the video playback fluency is smooth, and the time-domain feature of the unit of duration being marked as a second value based on a determination that the video playback fluency is freezing”. The motivation for doing so would have been to that “the quality of the video in time domain is evaluated according to this technical index in the embodiment of the disclosure, and the influence of the moveability on the quality of the video is measured by analysing the attribute of the index, thereby improving the accuracy of the evaluation”, as suggested by Wu in para. [0044]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang and Niu with Wu to obtain the invention specified in claim 14.
Claims 7 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (CN 110837842 A, see attached English translation for citations), hereinafter Zhang, in view of Niu et al. (CN 112954312 A, see attached English translation for citations), hereinafter Niu and Wang et al. (U.S. Publication No. 2015/0130952 A1), hereinafter Wang.
Regarding claim 7, Zhang and Niu teach the video quality assessment method according to claim 1, wherein obtaining the time-domain feature vector comprises:
arranging each of the time-domain features in an order of the corresponding unit of duration to obtain a time-domain feature sequence (Zhang teaches “the feature vector set is input to a time sequence inference module in the video quality assessment model, and a plurality of feature vectors of different scales are output by the time sequence inference module, where the different scales refer to different numbers of image frames corresponding to the scale feature vectors” in para. [0191]-[0192]);
obtaining the time-domain feature vector based on the respective values of the plurality of time-domain feature parameters (Zhang teaches “a method for determining a time domain feature vector is introduced, and based on consideration of inter-frame sequential logic, by establishing a model for a multi-scale inter-frame sequential logic relationship of a video, information of time domain distortion is extracted” in para. [0259], and “averaging according to the second-scale feature vector to be processed, the fourth-scale feature vector to be processed and the fifth-scale feature vector to be processed to obtain a time domain feature vector” in para. [0271]; see also para. [0194], wherein the time domain distortion is obtained by analyzing time domain distortion characteristics in the context of the time sequence connection).
Zhang and Niu fail to teach collecting statistic about respective corresponding values of a plurality of time-domain feature parameters from the time-domain feature sequence.
However, Wang teaches collecting statistic about respective corresponding values of a plurality of time-domain feature parameters from the time-domain feature sequence (Wang teaches “the system calculates the duration (in terms of number of frames) of each freeze event 33” wherein “the freeze event features include the count of the freeze events 35 for the whole video transmission as well as the mean, maximum, and standard deviation of the duration 39 of each freeze event” in para. [0040]. These statistical values are interpreted as equivalent to the claimed statistics regarding the time-domain feature parameters. See also applicant’s description of the above process in para. [0164]-[0170]).
Zhang, Niu, and Wang are all considered to be analogous to the claimed invention because they are in the same field of analyzing video quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu) to incorporate the teachings of Wang and include to “collect statistic about respective corresponding values of a plurality of time-domain feature parameters from the time-domain feature sequence”. The motivation for doing so would have been to “help[] avoid false positives freeze frames”, as suggested by Wang in para. [0038] and “explicitly consider[ing] the differences in the video content for more accurate video quality metrics”, as suggested by Wang in para. [0008]. Additionally, Wang suggests the invention “allows a quantifiable measurement of frame freezing and its relation to the subjective quality of the video presentation” in para. [0009]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang and Niu with Wang to obtain the invention specified in claim 7.
Regarding claim 15, Zhang and Niu teach the video quality assessment apparatus according to claim 9, wherein the time-domain detection code is further configured to cause at least one of the at least one processor to:
arrange each of the time-domain features in an order of the corresponding unit of duration to obtain a time-domain feature sequence (Zhang teaches “the feature vector set is input to a time sequence inference module in the video quality assessment model, and a plurality of feature vectors of different scales are output by the time sequence inference module, where the different scales refer to different numbers of image frames corresponding to the scale feature vectors” in para. [0191]-[0192]);
obtain the time-domain feature vector based on the respective values of the plurality of time-domain feature parameters (Zhang teaches “a method for determining a time domain feature vector is introduced, and based on consideration of inter-frame sequential logic, by establishing a model for a multi-scale inter-frame sequential logic relationship of a video, information of time domain distortion is extracted” in para. [0259], and “averaging according to the second-scale feature vector to be processed, the fourth-scale feature vector to be processed and the fifth-scale feature vector to be processed to obtain a time domain feature vector” in para. [0271]; see also para. [0194], wherein the time domain distortion is obtained by analyzing time domain distortion characteristics in the context of the time sequence connection).
Zhang and Niu fail to teach collecting statistic about respective corresponding values of a plurality of time-domain feature parameters from the time-domain feature sequence.
However, Wang teaches collecting statistic about respective corresponding values of a plurality of time-domain feature parameters from the time-domain feature sequence (Wang teaches “the system calculates the duration (in terms of number of frames) of each freeze event 33” wherein “the freeze event features include the count of the freeze events 35 for the whole video transmission as well as the mean, maximum, and standard deviation of the duration 39 of each freeze event” in para. [0040]. These statistical values are interpreted as equivalent to the claimed statistics regarding the time-domain feature parameters. See also applicant’s description of the above process in para. [0164]-[0170]).
Zhang, Niu, and Wang are all considered to be analogous to the claimed invention because they are in the same field of analyzing video quality. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu) to incorporate the teachings of Wang and include to “collect statistic about respective corresponding values of a plurality of time-domain feature parameters from the time-domain feature sequence”. The motivation for doing so would have been to “help[] avoid false positives freeze frames”, as suggested by Wang in para. [0038] and “explicitly consider[ing] the differences in the video content for more accurate video quality metrics”, as suggested by Wang in para. [0008]. Additionally, Wang suggests the invention “allows a quantifiable measurement of frame freezing and its relation to the subjective quality of the video presentation” in para. [0009]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang and Niu with Wang to obtain the invention specified in claim 15.
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (CN 110837842 A, see attached English translation for citations), hereinafter Zhang, in view of Niu et al. (CN 112954312 A, see attached English translation for citations), hereinafter Niu and Cheng et al. (CN 109615064 A, see attached English translation for citations), hereinafter Cheng.
Regarding claim 8, Zhang and Niu teach the video quality assessment method according to claim 1, wherein performing feature fusion processing on the spatial-domain feature vector and the time-domain feature vector to obtain the fusion feature vector, and determining the video quality assessment value of the online video stream based on the fusion feature vector comprises:
performing feature scaling processing on the time-domain feature vector to obtain a scaled time-domain feature vector (Zhang teaches that “a method for determining a time domain feature vector is provided, that is, a first function is used to generate a first scale to-be-processed feature vector corresponding to a first scale feature vector, a second function is used to generate a second scale to-be-processed feature vector corresponding to the first scale to-be-processed feature vector, a first function is used to generate a third scale to-be-processed feature vector corresponding to the second scale feature vector, a second function is used to generate a fourth scale to-be-processed feature vector corresponding to the third scale to-be-processed feature vector, and finally, a time domain feature vector is determined according to the second scale to-be-processed feature vector and the fourth scale to-be-processed feature vector” in para. [0275]; see also para. [0191]-[0192]), a dimension of the scaled time-domain feature vector being consistent with a dimension of the spatial-domain feature vector (Zhang teaches “a time domain feature vector 1 × 5, wherein 5 represents 5 dimensions” in para. [0275] which is the same as the number of dimensions in the spatial feature vector as shown in para. [0248]);
inputting the fusion feature vector into a full connection layer, and performing full connection processing on the fusion feature vector by using the full connection layer to obtain the video quality assessment value (Niu teaches inputting the fusion vector into a full connection layer regression in order to obtain the sub-video quality score in para. [0088]). Similar motivation as applied to claim 1 can be applied here.
While Niu teaches obtaining the fusion feature vector based on a (Niu teaches a process of developing the fusion vector wherein the temporal-spatial feature vector is subjected to pooling (which is broadly interpreted as equivalent to scaling in para. [0088], wherein the temporal-spatial vector is derived from a spatiotemporal feature map which is determined based on a summation and splicing technique applied to the temporal domain features and the spatial information of the feature maps in para. [0079], wherein the summation can broadly be interpreted as a difference, as the output of the summation is a map which highlights the two different vectors), Zhang and Niu fail to teach obtaining the fusion feature vector based on a difference between the spatial-domain feature vector and the scaled time-domain feature vector.
However, Cheng teaches obtaining the fusion feature vector based on a difference between the spatial-domain feature vector and the scaled time-domain feature vector (Cheng teaches “the spatial position feature vector and time contextual feature vector are passed through” a merge combining layer wherein space-time feature subtraction occurs in order to generate a new fusion feature vector that is subsequently sent to a full connection layer in para. [0035]).
Zhang, Niu, and Cheng are all considered to be analogous to the claimed invention because they are in the same field of analyzing videos and performing feature fusion. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu) to incorporate the teachings of Cheng and include “obtaining the fusion feature vector based on a difference between the spatial-domain feature vector and the scaled time-domain feature vector”. The motivation for doing so would have been to that feature fusion “can increase net The connection between nervous layer and subsequent nervous layer before network” which “enables characteristic information more rapidly to flow in a network”, as suggested by Cheng in para. [0004]. See also para. [0006]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang and Niu with Cheng to obtain the invention specified in claim 8.
Regarding claim 16, Zhang and Niu teach the video quality assessment apparatus according to claim 9, wherein the quality assessment code is further configured to cause at least one of the at least one processor to:
perform feature scaling processing on the time-domain feature vector to obtain a scaled time-domain feature vector (Zhang teaches that “a method for determining a time domain feature vector is provided, that is, a first function is used to generate a first scale to-be-processed feature vector corresponding to a first scale feature vector, a second function is used to generate a second scale to-be-processed feature vector corresponding to the first scale to-be-processed feature vector, a first function is used to generate a third scale to-be-processed feature vector corresponding to the second scale feature vector, a second function is used to generate a fourth scale to-be-processed feature vector corresponding to the third scale to-be-processed feature vector, and finally, a time domain feature vector is determined according to the second scale to-be-processed feature vector and the fourth scale to-be-processed feature vector” in para. [0275]; see also para. [0191]-[0192]), a dimension of the scaled time-domain feature vector being consistent with a dimension of the spatial-domain feature vector (Zhang teaches “a time domain feature vector 1 × 5, wherein 5 represents 5 dimensions” in para. [0275] which is the same as the number of dimensions in the spatial feature vector as shown in para. [0248]); and
input the fusion feature vector into a full connection layer, and perform full connection processing on the fusion feature vector by using the full connection layer to obtain the video quality assessment value (Niu teaches inputting the fusion vector into a full connection layer regression in order to obtain the sub-video quality score in para. [0088]). Similar motivation as applied to claim 9 can be applied here.
While Niu teaches obtaining the fusion feature vector based on a (Niu teaches a process of developing the fusion vector wherein the temporal-spatial feature vector is subjected to pooling (which is broadly interpreted as equivalent to scaling in para. [0088], wherein the temporal-spatial vector is derived from a spatiotemporal feature map which is determined based on a summation and splicing technique applied to the temporal domain features and the spatial information of the feature maps in para. [0079], wherein the summation can broadly be interpreted as a difference, as the output of the summation is a map which highlights the two different vectors), Zhang and Niu fail to teach obtaining the fusion feature vector based on a difference between the spatial-domain feature vector and the scaled time-domain feature vector.
However, Cheng teaches obtaining the fusion feature vector based on a difference between the spatial-domain feature vector and the scaled time-domain feature vector (Cheng teaches “the spatial position feature vector and time contextual feature vector are passed through” a merge combining layer wherein space-time feature subtraction occurs in order to generate a new fusion feature vector that is subsequently sent to a full connection layer in para. [0035]).
Zhang, Niu, and Cheng are all considered to be analogous to the claimed invention because they are in the same field of analyzing videos and performing feature fusion. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Zhang (as modified by Niu) to incorporate the teachings of Cheng and include to “obtain the fusion feature vector based on a difference between the spatial-domain feature vector and the scaled time-domain feature vector”. The motivation for doing so would have been to that feature fusion “can increase net The connection between nervous layer and subsequent nervous layer before network” which “enables characteristic information more rapidly to flow in a network”, as suggested by Cheng in para. [0004]. See also para. [0006]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Zhang and Niu with Cheng to obtain the invention specified in claim 16.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to KYLA G ALLEN whose telephone number is (703)756-5315. The examiner can
normally be reached M-F 7:30am - 4:30pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use
the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
John Villecco can be reached on (571) 272-7319. The fax phone number for the organization where this
application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users. To
file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or
571-272-1000.
/Kyla Guan-Ping Tiao Allen/
Examiner, Art Unit 2661
/JOHN VILLECCO/Supervisory Patent Examiner, Art Unit 2661
Prosecution Insights