DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yue Li et al. [US 20220394308 A1] in view of Feng Wu et al. [US 20210227243 A1].
Regarding claim 1, Yue teaches:
1. A decoding method (i.e. The present disclosure is generally related to image and video coding and decoding- ¶0002… The disclosed examples may also be applicable to future video coding standards, future video codecs, or as a post-processing method outside of an encoding/decoding process- ¶0057… The coding flow of a typical video coder/decoder (a.k.a., codec) is discussed- ¶0079, fig. 5… Furthermore, while certain video processing operations are referred to as “coding” operations or tools, it will be appreciated that the coding tools or operations are used at an encoder and corresponding decoding tools or operations that reverse the results of the coding will be performed by a decoder- ¶0719), comprising:
decoding a bitstream to determine a related syntax element of a current coding tree unit (i.e. Further, the presence of (e.g., application of) NN filter models may be controlled through syntax elements at various levels. For example, the syntax element(s) that indicate whether to apply a NN filter may be at a first level (e.g., in a sequence parameter set (SPS) and/or a sequence header of a video unit). Syntax element(s) that indicate whether to apply a NN filter may also be at a second level (e.g., a picture header, a picture parameter set (PPS), and/or a slice header of the video unit). Still further, syntax element(s) that indicate whether to apply a NN filter may be at a third level (e.g., the syntax element is indicated for a patch of the video unit, a CTU of the video unit, a CTB of the video unit, a block of the video unit, a subpicture of the video unit, a tile of the video unit, or a region of the video unit- ¶0004); determining a geometric transformation type (i.e. FIG. 9 shows examples of geometry transformation-based adaptive loop filter (GALF) filter shapes- ¶0039… the filter shape- ¶0147) of the current coding tree unit according (i.e. An index is signaled at the picture level to indicate the filter shape used for the luma component- ¶0147... Three geometric transformations, including diagonal, vertical flip, and rotation- ¶0164) to the related syntax element (i.e. Filter parameters signaling is discussed. In the JEM, GALF filter parameters are signalled for the first CTU, i.e., after the slice header and before the SAO parameters of the first CTU. Up to 25 sets of luma filter coefficients could be signalled. To reduce bits overhead, filter coefficients of different classification can be merged. Also, the GALF coefficients of reference pictures are stored and allowed to be reused as GALF coefficients of a current picture. The current picture may choose to use GALF coefficients stored for the reference pictures and bypass the GALF coefficients signalling. In this case, only an index to one of the reference pictures is signalled, and the stored GALF coefficients of the indicated reference picture are inherited for the current picture- ¶0173… A flag is signalled to indicate whether GALF is applied to the luma component of a CU. For chroma component, whether GALF is applied or not is indicated at picture level only- ¶0176);
determining reference sample information of the current coding tree unit, wherein the reference sample information at least comprises predicted sample information and/or reconstructed sample information of the current coding tree unit (i.e. a reconstruction unit 2112 to reconstruct the encoded block for use as a reference picture- ¶0734… Reconstruction unit 2112 may add the reconstructed residual video block to corresponding samples from one or more predicted video blocks generated by the prediction unit 2102 to produce a reconstructed video block associated with the current block for storage in the buffer 2113- ¶0749);
performing geometric transformation on the reference sample information of the current coding tree unit according to the geometric transformation type, to obtain geometric transformed reference sample information (i.e. the coefficients of the adaptive filter f (k, l) can still be sent for this class in which case the coefficients of the filter which will be applied to the reconstructed image are sum of both sets of coefficients- ¶0175… At the decoder side, when GALF is enabled for a block, each sample R(i, j) within the block is filtered, resulting in sample value R′ (i, j) as shown below, where L denotes filter length, fm,n represents filter coefficient, and f (k, l) denotes the decoded filter coefficients- ¶0177);
inputting the geometric transformed reference sample information of the current coding tree unit to a neural network based in-loop filter model for filtering, to output filtered reconstructed sample information (i.e. FIG. 12A is an example architecture 1200 of the proposed CNN filter, and FIG. 12B is an example of construction 1250 of residual block (ResBlock). In most of deep convolutional neural networks, residual blocks are utilized as the basic module and stacked several times to construct the final network wherein in one example, the residual block is obtained by combining a convolutional layer, a ReLU/PReLU activation function and a convolutional layer as shown in FIG. 12B- ¶0206… the distorted reconstruction frames are fed into CNN and processed by the CNN model whose parameters are already determined in the training stage. The input samples to the CNN can be reconstructed samples before or after DB, or reconstructed samples before or after SAO, or reconstructed samples before or after ALF- ¶0207… the present disclosure provides one or more neural network (NN) filter models trained as part of an in-loop filtering technology or filtering technology used in a post-processing stage for reducing the distortion incurred during compression. In addition, samples with different characteristics may be processed by different NN filter models- ¶0209).
However, Yue does not teach explicitly:
performing inverse geometric transformation on the filtered reconstructed sample information according to the geometric transformation type, to obtain final reconstructed sample information of the current coding tree unit.
In the same field of endeavor, Feng teaches:
performing inverse geometric transformation on the filtered reconstructed sample information according to the geometric transformation type, to obtain final reconstructed sample information of the current coding tree unit (i.e. an inverse interpolation module, configured to input, to a third interpolation filter, the second sub-pixel picture on which a flipping operation is performed, to obtain a first picture, and perform an inverse operation of the flipping operation on the first picture to obtain a second picture, where the second interpolation filter and the third interpolation filter share a filter parameter- ¶0072,… FIG. 6D is an illustrative schematic diagram of another interpolation filter training procedure according to an embodiment of this application- ¶0153).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Yue teaches with the teachings Feng to improve prediction accuracy of motion information of a picture block, and further improve coding performance (Feng- ¶0007).
Regarding claim 2, Yue and Feng teach all the limitations of claim 1 and Yue further teaches:
comprising: determining, according to a first syntax element, whether a current image block in which the current coding tree unit is located uses a neural network based in-loop filtering technology with performing the geometric transformation on an input (i.e. Further, the presence of (e.g., application of) NN filter models may be controlled through syntax elements at various levels. For example, the syntax element(s) that indicate whether to apply a NN filter may be at a first level (e.g., in a sequence parameter set (SPS) and/or a sequence header of a video unit). Syntax element(s) that indicate whether to apply a NN filter may also be at a second level (e.g., a picture header, a picture parameter set (PPS), and/or a slice header of the video unit). Still further, syntax element(s) that indicate whether to apply a NN filter may be at a third level (e.g., the syntax element is indicated for a patch of the video unit, a CTU of the video unit, a CTB of the video unit, a block of the video unit, a subpicture of the video unit, a tile of the video unit, or a region of the video unit- ¶0004).
Regarding claim 3, Yue and Feng teach all the limitations of claim 2 and Yue further teaches:
wherein the first syntax element comprises at least one of following: an image sequence level first syntax element, used to indicate whether an image sequence uses the neural network based in-loop filtering technology with performing the geometric transformation on the input; an image level first syntax element, used to indicate whether an image uses the neural network based in-loop filtering technology with performing the geometric transformation on the input; a slice level first syntax element, used to indicate whether a slice uses the neural network based in-loop filtering technology with performing the geometric transformation on the input; or a coding tree unit level first syntax element, used to indicate whether a coding tree unit uses the neural network based in-loop filtering technology with performing the geometric transformation on the input(i.e. Further, the presence of (e.g., application of) NN filter models may be controlled through syntax elements at various levels. For example, the syntax element(s) that indicate whether to apply a NN filter may be at a first level (e.g., in a sequence parameter set (SPS) and/or a sequence header of a video unit). Syntax element(s) that indicate whether to apply a NN filter may also be at a second level (e.g., a picture header, a picture parameter set (PPS), and/or a slice header of the video unit). Still further, syntax element(s) that indicate whether to apply a NN filter may be at a third level (e.g., the syntax element is indicated for a patch of the video unit, a CTU of the video unit, a CTB of the video unit, a block of the video unit, a subpicture of the video unit, a tile of the video unit, or a region of the video unit- ¶0004… Further, examples of this description are directed to controlling the presence of (e.g., application of) NN filter models through syntax elements at various levels. For example, the syntax element(s) that indicate whether to apply a NN filter may be at a first level (e.g., in a sequence parameter set (SPS) and/or a sequence header of a video unit). Syntax element(s) that indicate whether to apply a NN filter may also be at a second level (e.g., a picture header, a picture parameter set (PPS), and/or a slice header of the video unit). Still further, syntax element(s) that indicate whether to apply a NN filter may be at a third level (e.g., the syntax element is indicated for a patch of the video unit, a CTU of the video unit, a CTB of the video unit, a block of the video unit, a subpicture of the video unit, a tile of the video unit, or a region of the video unit- ¶0211).
Regarding claim 4, Yue and Feng teach all the limitations of claim 2 and Yue further teaches:
comprising: determining, according to a second syntax element, the geometric transformation type of the coding tree unit in the current image block, when determining, according to the first syntax element, that the current image block in which the current coding tree unit is located uses the neural network based in-loop filtering technology with performing the geometric transformation on the input (i.e. An index is signaled at the picture level to indicate the filter shape used for the luma component- ¶0147... Three geometric transformations, including diagonal, vertical flip, and rotation- ¶0164… Filter parameters signaling is discussed. In the JEM, GALF filter parameters are signalled for the first CTU, i.e., after the slice header and before the SAO parameters of the first CTU. Up to 25 sets of luma filter coefficients could be signalled. To reduce bits overhead, filter coefficients of different classification can be merged. Also, the GALF coefficients of reference pictures are stored and allowed to be reused as GALF coefficients of a current picture. The current picture may choose to use GALF coefficients stored for the reference pictures and bypass the GALF coefficients signalling. In this case, only an index to one of the reference pictures is signalled, and the stored GALF coefficients of the indicated reference picture are inherited for the current picture- ¶0173… A flag is signalled to indicate whether GALF is applied to the luma component of a CU. For chroma component, whether GALF is applied or not is indicated at picture level only- ¶0176… when GALF is enabled for a block, each sample R(i, j) within the block is filtered, resulting in sample value R′ (i, j) as shown below, where L denotes filter length, fm,n represents filter coefficient, and f (k, l) denotes the decoded filter coefficients- ¶0177).
Regarding claim 5, Yue and Feng teach all the limitations of claim 4 and Yue further teaches:
wherein the second syntax element comprises one of following: an image sequence level second syntax element, used to indicate a geometric transformation type of all coding tree units in an image sequence; an image level second syntax element, used to indicate a geometric transformation type of all coding tree units in an image; a slice level second syntax element, used to indicate a geometric transformation type of all coding tree units in a slice; or a coding tree unit level second syntax element, used to indicate a geometric transformation type of a coding tree unit (i.e. Filter parameters signaling is discussed. In the JEM, GALF filter parameters are signalled for the first CTU, i.e., after the slice header and before the SAO parameters of the first CTU- ¶0173).
Regarding claim 6, Yue and Feng teach all the limitations of claim 2 and Yue further teaches:
determining, according to a third syntax element, whether the current image block uses a neural network based in-loop filtering technology(i.e. in any of the preceding aspects, another implementation of the aspect provides that the syntax element is coded based on a context model that is selected based on a number of allowed NN filters, wherein a filter model index for a color component of the video unit is configured to specify one of K context models, and wherein the one of the K context models is specified as a minimum of K−1 and binIdx, wherein binIdx is an index of a bin to be coded- ¶0018).
Regarding claim 7, Yue and Feng teach all the limitations of claim 6 and Yue further teaches:
wherein the third syntax element comprises at least one of following:
an image sequence level third syntax element, used to indicate whether an image sequence uses the neural network based in-loop filtering technology; an image level third syntax element, used to indicate whether an image uses the neural network based in-loop filtering technology; a slice level third syntax element, used to indicate whether a slice uses the neural network based in-loop filtering technology; or a coding tree unit level third syntax element, used to indicate whether a coding tree unit uses the neural network based in-loop filtering technology (i.e. the presence of (e.g., application of) NN filter models may be controlled through syntax elements at various levels. For example, the syntax element(s) that indicate whether to apply a NN filter may be at a first level (e.g., in a sequence parameter set (SPS) and/or a sequence header of a video unit). Syntax element(s) that indicate whether to apply a NN filter may also be at a second level (e.g., a picture header, a picture parameter set (PPS), and/or a slice header of the video unit). Still further, syntax element(s) that indicate whether to apply a NN filter may be at a third level (e.g., the syntax element is indicated for a patch of the video unit, a CTU of the video unit, a CTB of the video unit, a block of the video unit, a subpicture of the video unit, a tile of the video unit, or a region of the video unit- ¶0004).
Regarding claim 8, Yue and Feng teach all the limitations of claim 2 and Yue further teaches:
determining, according to a fourth syntax element, whether the current image block is allowed to use the neural network based in-loop filtering technology with performing the geometric transformation on the input (i.e. in any of the preceding aspects, another implementation of the aspect provides that the syntax element is coded based on a context model that is selected based on a number of allowed NN filters, wherein a filter model index for a color component of the video unit is configured to specify one of K context models, and wherein the one of the K context models is specified as a minimum of K−1 and binIdx, wherein binIdx is an index of a bin to be coded- ¶0018).
Regarding claim 9, Yue and Feng teach all the limitations of claim 8 and Yue further teaches:
wherein the fourth syntax element comprises at least one of following: an image sequence level fourth syntax element, used to indicate whether an image sequence is allowed to use the neural network based in-loop filtering technology with performing the geometric transformation on the input; an image level fourth syntax element, used to indicate whether an image is allowed to use the neural network based in-loop filtering technology with performing the geometric transformation on the input; a slice level fourth syntax element, used to indicate whether a slice is allowed to use the neural network based in-loop filtering technology with performing the geometric transformation on the input; or a coding tree unit level fourth syntax element, used to indicate whether a coding tree unit is allowed to use the neural network based in-loop filtering technology with performing the geometric transformation on the input (i.e. in any of the preceding aspects, another implementation of the aspect provides that a first level comprises a sequence level and a syntax element indicated in the first level is indicated in a sequence parameter set (SPS) and/or a sequence header of the video unit; a second level comprises a picture level and a syntax element indicated in the second level is indicated in a picture header, a picture parameter set (PPS), and/or a slice header of the video unit; and a third level comprises a subpicture level a syntax element indicated in the third level is indicated for a patch of the video unit, a coding tree unit (CTU) of the video unit, a coding tree block (CTB) of the video unit, a block of the video unit, a subpicture of the video unit, a tile of the video unit, a slice of the video unit, or a region of the video unit- ¶0007).
Regarding claim 10, Yue and Feng teach all the limitations of claim 2 and Yue further teaches:
determining, according to a fifth syntax element, whether the current image block is allowed to use a neural network based in-loop filtering technology (i.e. in any of the preceding aspects, another implementation of the aspect provides that the syntax element is coded based on a context model that is selected based on a number of allowed NN filters, wherein a filter model index for a color component of the video unit is configured to specify one of K context models, and wherein the one of the K context models is specified as a minimum of K−1 and binIdx, wherein binIdx is an index of a bin to be coded- ¶0018).
Regarding claim 11, Yue and Feng teach all the limitations of claim 10 and Yue further teaches:
wherein the fifth syntax element comprises at least one of the following:
an image sequence level fifth syntax element, used to indicate whether an image sequence is allowed to use the neural network based in-loop filtering technology; an image level fifth syntax element, used to indicate whether an image is allowed to use the neural network based in-loop filtering technology; a slice level fifth syntax element, used to indicate whether a slice is allowed to use the neural network based in-loop filtering technology; or a coding tree unit level fifth syntax element, used to indicate whether a coding tree unit is allowed to use the neural network based in-loop filtering technology (i.e. in any of the preceding aspects, another implementation of the aspect provides that a first level comprises a sequence level and a syntax element indicated in the first level is indicated in a sequence parameter set (SPS) and/or a sequence header of the video unit; a second level comprises a picture level and a syntax element indicated in the second level is indicated in a picture header, a picture parameter set (PPS), and/or a slice header of the video unit; and a third level comprises a subpicture level a syntax element indicated in the third level is indicated for a patch of the video unit, a coding tree unit (CTU) of the video unit, a coding tree block (CTB) of the video unit, a block of the video unit, a subpicture of the video unit, a tile of the video unit, a slice of the video unit, or a region of the video unit- ¶0007… Further, examples of this description are directed to controlling the presence of (e.g., application of) NN filter models through syntax elements at various levels. For example, the syntax element(s) that indicate whether to apply a NN filter may be at a first level (e.g., in a sequence parameter set (SPS) and/or a sequence header of a video unit). Syntax element(s) that indicate whether to apply a NN filter may also be at a second level (e.g., a picture header, a picture parameter set (PPS), and/or a slice header of the video unit). Still further, syntax element(s) that indicate whether to apply a NN filter may be at a third level (e.g., the syntax element is indicated for a patch of the video unit, a CTU of the video unit, a CTB of the video unit, a block of the video unit, a subpicture of the video unit, a tile of the video unit, or a region of the video unit- ¶0211… FIG. 16 illustrates an embodiment of a video bitstream 1600. As used herein the video bitstream 1600 may also be referred to as a coded video bitstream, a bitstream, or variations thereof. As shown in FIG. 16, the bitstream 1600 comprises one or more of the following: decoding capability information (DCI) 1602, a video parameter set (VPS) 1604, a sequence parameter set (SPS) 1606, a picture parameter set (PPS) 1608, a picture header (PH) 1612, and a picture 1614. Each of the DCI 1602, the VPS 1604, the SPS 1606, and the PPS 1608 may be generically referred to as a parameter set. In an embodiment, other parameter sets not shown in FIG. 16 may also be included in the bitstream 1600 such as, for example, an adaption parameter set (APS), which is a syntax structure containing syntax elements that apply to zero or more slices as determined by zero or more syntax elements found in slice headers- ¶0219).
Regarding claim 12, Yue and Feng teach all the limitations of claim 1 and Yue further teaches:
wherein the reference sample information further comprises a constant parameter, and the method further comprises: determining, according to a sixth syntax element, whether to adjust the constant parameter of a current image block in which the current coding tree unit is located; determining, according to a seventh syntax element, an adjusted constant parameter of the current image block in which the current coding tree unit is located when determining, according to the sixth syntax element, to adjust the constant parameter of the current image block in which the current coding tree unit is located; and adjusting the constant parameter according to an adjustment parameter, and inputting the adjusted constant parameter to a neural network based in-loop filtering technology (i.e. The current CNN-based loop filtering has the following problems. First, an individual CNN model is trained for each quality level (e.g., a quantization parameter (QP), a constant rate factor (CRF), or a bitrate), which results in a large number of CNN models. Second, when QP is taken as the input of neural networks, it is first tiled into a two-dimensional array with the same size as the block to be filtered, and then treated as an additional input plane. Accordingly, the QP information may not be fully exploited. Third, when constructing a candidate list containing multiple models for a coding block, the characteristics of the coding block have not been fully explored- ¶0208… The present disclosure also elaborates how to design a unified NN filter model by feeding at least one indicator, which may be related to quality level (e.g., QP, CRF, or bitrate) and thus is a quality-level indicator (QI), as an input to the NN filter, and how to construct a candidate list containing multiple models for a coding block by taking the coding statistics of the block into account. Other examples of this description are directed to combining NN filter models with non-deep learning-based filtering (NDLF) methods- ¶0210… b. In one example, the quality-level indicator of a video unit is the QP associated with the sequence (e.g., the initial QP signalled in SPS) that the video unit belongs to- ¶0239).
Regarding claim 13, Yue and Feng teach all the limitations of claim 12 and Yue further teaches:
wherein the sixth syntax element comprises at least one of following: an image sequence level sixth syntax element, used to indicate whether to adjust the constant parameter of a coding tree unit in an image sequence; an image level sixth syntax element, used to indicate whether to adjust the constant parameter of a coding tree unit in an image; a slice level sixth syntax element, used to indicate whether to adjust the constant parameter of a coding tree unit in a slice; or a coding tree unit level sixth syntax element, used to indicate whether to adjust the constant parameter of a coding tree unit; and the seventh syntax element includes one of following: an image sequence level seventh syntax element, used to indicate an adjusted constant parameter for all coding tree units in an image sequence; a seventh syntax element of an image level, used to indicate an adjusted constant parameter for all coding tree units in an image; a slice level seventh syntax element, used to indicate an adjusted constant parameter for all coding tree units in a slice; or a coding tree unit level seventh syntax element, used to indicate an adjusted constant parameter for a coding tree unit (i.e. b. In one example, the quality-level indicator of a video unit is the QP associated with the sequence (e.g., the initial QP signalled in SPS) that the video unit belongs to- ¶0239.
Regarding claim 14, Yue and Feng teach all the limitations of claim 2 and Yue further teaches:
wherein the current image block comprises at least one of following: a current image sequence, a current image, a current slice, or a current coding tree unit (i.e. Filter parameters signaling is discussed. In the JEM, GALF filter parameters are signalled for the first CTU, i.e., after the slice header and before the SAO parameters of the first CTU- ¶0173).
Regarding claim 15, Yue and Feng teach all the limitations of claim 2 and Yue further teaches:
comprising: in a case that a time domain level of the current coding tree unit is greater than or equal to a time domain level threshold, determining the geometric transformation type of the current coding tree unit according to the related syntax element; in a case that the time domain level of the current coding tree unit is less than the time domain level threshold, determining not to use a neural network based in-loop filtering technology with performing the geometric transformation on an input (i.e. position dependent threshold table is selected from two tables (i.e., Tc7 and Tc3 tabulated below) that are provided to decoder as a side information- ¶0129… For the P or Q boundaries being filtered with a short symmetrical filter, position dependent threshold of lower magnitude is applied- ¶0133… In one example, one candidate in the candidate list is the NN filter model corresponding to the candidate for coding intra slices if the ratio of intra-coded samples is no smaller than or greater than a threshold- ¶0262… For a low temporal layer, the third candidate is replaced by the intra-slice NN filter model if the ratio of intra-coded samples is no smaller than a threshold- ¶0435… Intra-slice NN filter may be used for an inter-slice. For a low temporal layer, the inter-slice NN filter input with the third auxiliary parameter candidate is replaced by the intra-slice NN filter model if the ratio of intra-coded samples is no smaller than a threshold- ¶0447… For low temporal layers, the third candidate is replaced by the intra-slice NN filter model if the ratio of intra-coded samples is no smaller than a threshold. For high temporal layers, the third candidate, i.e. the q−10 model is replaced by the q+5 model- ¶0464).
Regarding claim 16, Yue and Feng teach all the limitations of claim 1 and Yue further teaches:
wherein the reference sample information further comprises a constant parameter and a non-constant parameter of the current tree coding unit; the constant parameter comprises at least one of the following: a quantization parameter, or an image type or a slice type corresponding to the current coding tree unit (i.e. The current CNN-based loop filtering has the following problems. First, an individual CNN model is trained for each quality level (e.g., a quantization parameter (QP), a constant rate factor (CRF), or a bitrate), which results in a large number of CNN models. Second, when QP is taken as the input of neural networks, it is first tiled into a two-dimensional array with the same size as the block to be filtered, and then treated as an additional input plane. Accordingly, the QP information may not be fully exploited. Third, when constructing a candidate list containing multiple models for a coding block, the characteristics of the coding block have not been fully explored- ¶0208… The present disclosure also elaborates how to design a unified NN filter model by feeding at least one indicator, which may be related to quality level (e.g., QP, CRF, or bitrate) and thus is a quality-level indicator (QI), as an input to the NN filter, and how to construct a candidate list containing multiple models for a coding block by taking the coding statistics of the block into account. Other examples of this description are directed to combining NN filter models with non-deep learning-based filtering (NDLF) methods- ¶0210… b. In one example, the quality-level indicator of a video unit is the QP associated with the sequence (e.g., the initial QP signalled in SPS) that the video unit belongs to- ¶0239); and the non-constant parameter includes at least one of the following: boundary strength information of the current coding tree unit, partitioning information of the current coding tree unit, or reconstructed sample information of a coding tree unit that is corresponding to the current coding tree unit and is in a reference image (i.e. The boundary strength calculation is discussed. For a transform block boundary/coding subblock boundary, if it is located in the 8×8 grid, the transform block boundary/coding subblock boundary may be filterd and the setting of bS[xD.sub.i] [yD.sub.j] (wherein [xD.sub.i] [yD.sub.j] denotes the coordinate) for this edge is defined in Tabel 1 and Table 2, respectively- ¶0085… In the first decision, boundary strength (bS) is modified for chroma filtering and the conditions are checked sequentially.- ¶0114… For each P or Q boundary filtered with asymmetrical filter, depending on the result of decision-making process in the boundary strength calculation, position dependent threshold table is selected from two tables (i.e., Tc7 and Tc3 tabulated below) that are provided to decoder as a side information- ¶0129).
Regarding claim 17, Yue and Feng teach all the limitations of claim 1 and Yue further teaches:
wherein the geometric transformation type comprises one of following: diagonal flip, horizontal flip, vertical flip, or rotation by a preset angle (i.e. FIG. 10 shows relative coordinates 1000 for the 5×5 diamond filter support-diagonal, vertical flip, and rotation, respectively (from left to right)- ¶0162).
Regarding claim 18, Yue and Feng teach all the limitations of claim 1 and Yue further teaches:
wherein the current coding tree unit is a largest coding unit, or is obtained by changing a size of a largest coding unit (i.e. CTUs in a picture 400 are discussed with reference to FIGS. 4A-4C. Suppose the CTB/largest coding unit (LCU) size indicated by M×N (typically M is equal to N, as defined in HEVC/VVC), and for a CTB located at picture (or tile or slice or other kinds of types, picture border is taken as an example) border, K×L samples are within picture border wherein either K<M or L<N. For those CTBs 402 as depicted in FIG. 4A-4C, the CTB size is still equal to M×N, however, the bottom boundary/right boundary of the CTB is outside the picture 400- ¶0078).
Regarding claim 19, Yue teaches:
19. An encoding method (i.e. The present disclosure is generally related to image and video coding and decoding- ¶0002… The disclosed examples may also be applicable to future video coding standards, future video codecs, or as a post-processing method outside of an encoding/decoding process- ¶0057… The coding flow of a typical video coder/decoder (a.k.a., codec) is discussed- ¶0079, fig. 5… Furthermore, while certain video processing operations are referred to as “coding” operations or tools, it will be appreciated that the coding tools or operations are used at an encoder and corresponding decoding tools or operations that reverse the results of the coding will be performed by a decoder- ¶0719), comprising:
determining reference sample information of a current coding tree unit, wherein the reference sample information at least comprises predicted sample information and/or reconstructed sample information of the current coding tree unit (i.e. a reconstruction (REC) component 524. The REC component 524 is able to output images to the DF 502- ¶0081);
performing geometric transformation on the reference sample information of the current coding tree unit according to candidate geometric transformation types, to obtain geometric transformed reference sample information (i.e. the coefficients of the adaptive filter f (k, l) can still be sent for this class in which case the coefficients of the filter which will be applied to the reconstructed image are sum of both sets of coefficients- ¶0175… At the decoder side, when GALF is enabled for a block, each sample R(i, j) within the block is filtered, resulting in sample value R′ (i, j) as shown below, where L denotes filter length, fm,n represents filter coefficient, and f (k, l) denotes the decoded filter coefficients- ¶0177);;
inputting the geometric transformed reference sample information of the current coding tree unit to a neural network based in-loop filter model for filtering, to output filtered reconstructed sample information (i.e. FIG. 12A is an example architecture 1200 of the proposed CNN filter, and FIG. 12B is an example of construction 1250 of residual block (ResBlock). In most of deep convolutional neural networks, residual blocks are utilized as the basic module and stacked several times to construct the final network wherein in one example, the residual block is obtained by combining a convolutional layer, a ReLU/PReLU activation function and a convolutional layer as shown in FIG. 12B- ¶0206… the distorted reconstruction frames are fed into CNN and processed by the CNN model whose parameters are already determined in the training stage. The input samples to the CNN can be reconstructed samples before or after DB, or reconstructed samples before or after SAO, or reconstructed samples before or after ALF- ¶0207… the present disclosure provides one or more neural network (NN) filter models trained as part of an in-loop filtering technology or filtering technology used in a post-processing stage for reducing the distortion incurred during compression. In addition, samples with different characteristics may be processed by different NN filter models- ¶0209);
determining a geometric transformation type of the current coding tree unit according to first distortion cost values of the current coding tree unit respectively corresponding to the candidate geometric transformation types (i.e. FIG. 9 shows examples of geometry transformation-based adaptive loop filter (GALF) filter shapes- ¶0039… the filter shape- ¶0147… Three geometric transformations, including diagonal, vertical flip, and rotation- ¶0164);; and
encoding a related syntax element to the geometric transformation type of the current coding tree unit, and writing an encoded bit into a bitstream (i.e. Further, the presence of (e.g., application of) NN filter models may be controlled through syntax elements at various levels. For example, the syntax element(s) that indicate whether to apply a NN filter may be at a first level (e.g., in a sequence parameter set (SPS) and/or a sequence header of a video unit). Syntax element(s) that indicate whether to apply a NN filter may also be at a second level (e.g., a picture header, a picture parameter set (PPS), and/or a slice header of the video unit). Still further, syntax element(s) that indicate whether to apply a NN filter may be at a third level (e.g., the syntax element is indicated for a patch of the video unit, a CTU of the video unit, a CTB of the video unit, a block of the video unit, a subpicture of the video unit, a tile of the video unit, or a region of the video unit- ¶0004.
However, Yue does not teach explicitly:
performing inverse geometric transformation on the filtered reconstructed sample information according to the candidate geometric transformation types, to obtain final reconstructed sample information of the current coding tree unit;
determining a first distortion cost value of the current coding tree unit according to original sample information and the final reconstructed sample information of the current coding tree unit.
In the same field of endeavor, Feng teaches:
performing inverse geometric transformation on the filtered reconstructed sample information according to the candidate geometric transformation types, to obtain final reconstructed sample information of the current coding tree unit (i.e. an inverse interpolation module, configured to input, to a third interpolation filter, the second sub-pixel picture on which a flipping operation is performed, to obtain a first picture, and perform an inverse operation of the flipping operation on the first picture to obtain a second picture, where the second interpolation filter and the third interpolation filter share a filter parameter- ¶0072,… FIG. 6D is an illustrative schematic diagram of another interpolation filter training procedure according to an embodiment of this application- ¶0153);
determining a first distortion cost value of the current coding tree unit according to original sample information and the final reconstructed sample information of the current coding tree unit (i.e. The encoder determines, from the set of candidate interpolation filters according to a rate-distortion cost criterion, the target interpolation filter used for the current encoding picture block- ¶0025… second loss function- fig. 6D… A flipping operation T is performed on a sub-pixel picture Xf that is generated through sub-pixel interpolation, then sub-pixel interpolation is performed by using the third interpolation filter to obtain the first picture, and then an inverse operation T−1 of the flipping operation T is performed on the first picture to obtain a reconstructed picture of the sample picture, that is, the second picture. Both the first picture and the second picture are integer pixel pictures, and the flipping operation includes horizontal flipping, vertical flipping, and diagonal flipping- ¶0304… The second function may be a function that is used to represent the difference between the sample picture and the second picture. The second function may be a loss function, a target function, a cost function, or the like- ¶0306).
It would have been obvious to one with ordinary skill in the art before the effective filing date of the claimed invention, to modify the teachings of Yue teaches with the teachings Feng to improve prediction accuracy of motion information of a picture block, and further improve coding performance (Feng- ¶0007).
Regarding claim 20, apparatus claim 20 is drawn to the apparatus using/performing the same method as claimed in claim 1. Therefore, apparatus claim 20 corresponds to method claim 1, and is rejected for the same reasons of obviousness as used above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CLIFFORD HILAIRE whose telephone number is (571)272-8397. The examiner can normally be reached 5:30-1400.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SATH V PERUNGAVOOR can be reached at (571)272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
CLIFFORD HILAIRE
Primary Examiner
Art Unit 2488
/CLIFFORD HILAIRE/Primary Examiner, Art Unit 2488
Prosecution Insights