DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This communication is responsive to the correspondence filled on 06/16/2025.
Claims 36-55 are presented for examination.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 36-37, 40, 42-45, 48, 50-53 and 54 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang468 (U.S. Pub. No. 20210243468).
Examiner’s Note: Video encoding and decoding are done using same and opposite algorithm.
Regarding to claim 36, 44 and 52:
36. Zhang468 teach a video decoding device, (Zhang468 [0004] Devices, systems and methods related to digital video coding, and specifically, to the video and image coding and decoding in which motion information for affine mode is utilized during video encoding or decoding are described. The described methods may be applied to both the existing video coding standards (e.g., High Efficiency Video Coding (HEVC)) and future video coding standards or video codecs.) comprising: a processor configured to: (Zhang468 [0008] In yet another representative aspect, a device that is configured or operable to perform the above-described method is disclosed. The device may include a processor that is programmed to implement this method.) partition a video block into a plurality of sub-blocks; for a sub-block of the plurality of sub-blocks, (Zhang468 Fig. 9A, Fig. 9B, [0130] In some embodiments, and to reduce the memory bandwidth requirement in affine prediction, each 8×8 block [first level of sub-block inside a video block] within the block is regarded as the basic unit. The MVs of all four 4×4 sub-blocks [second level of sub-block inside a video block which is a partition of level below 8×8 sub-block] inside the 8×8 block are constrained such that the max difference between integer parts of the four 4×4 sub-block MVs is no more than 1 pixel. So that the bandwidth is (8+7+1)*(8+7+1)/(8*8)=4 sample/pixel.) determine whether to split the sub-block for motion compensation; (Zhang468 [0042] The derived CP MVs mv.sub.0.sup.C, mv.sub.1.sup.C and mv.sub.2.sup.C of current block can be used as CP MVs in the affine merge mode. Or they can be used as MVP for affine inter mode in VVC. It should be noted that for the merge mode, if the current block is coded with affine mode, after deriving CP MVs of current block, the current block may be [determine] further split into multiple sub-blocks and each block will derive its motion information based on the derived CP MVs of current block.)
based on a determination to split the sub-block, split the sub-block into a plurality of units that are smaller than the sub-block, (Zhang468 Fig. 9A, Fig. 9B, [0130] In some embodiments, and to reduce the memory bandwidth requirement in affine prediction, each 8×8 block [first level of sub-block inside a video block] within the block is regarded as the basic unit. The MVs of all four 4×4 sub-blocks [second level of sub-block inside a video block which is a partition of level below 8×8 sub-block] inside the 8×8 block are constrained such that the max difference between integer parts of the four 4×4 sub-block MVs is no more than 1 pixel. So that the bandwidth is (8+7+1)* (8+7+1)/ (8*8)=4 sample/pixel. [0131] For example, after the MVs of all sub-blocks inside the current block are calculated with affine model, the MV of the sub-blocks containing the control points are firstly replaced with the corresponding control point MV. This means that, the MV of the top-left, top-right and bottom-left sub-blocks are replaced by the top-left, top-right and bottom-left control points MV, respectively. Then, for each 8×8 block within the current block, the MVs of all four 4×4 sub-blocks are clipped to guarantee the max difference between integer parts of the four MVs no more than 1 pixel. Here it should be noted that the sub-blocks containing the control points (top-left, top-right and bottom-left sub-blocks) use the corresponding control point MV to involve in the MV clipping process. During the clipping process, the MV of the top-right control point is kept un-changed. [0124] In some embodiments, since the current block is divided into 4×4 sub-blocks for luma component and 2×2 sub-blocks for the two chroma components to do the motion compensation, the total bandwidth requirement is much higher than non sub-block inter-prediction. [0125] In some embodiments, a 4×4 block is used as the sub-block size for a uni-directional affine coded CU while 8×4/4×8 block is used as the sub-block size for a bi-directional affine coded CU.) wherein each of the plurality of units comprise a plurality of pixels; (Zhang468 Fig. 9A, Fig. 9B [0140] (2) The MV components of each 4×4 block inside this 8×8 block are clipped as follows:
MVxi=max(MVminx,min(MVmaxx,MVxi))
MVyi=max(MVminy,min(MVmaxy,MVyi))
[0141] Herein, (MVxi, MVyi) is the MV of ith sub-block within one 8×8 block, where i is 0, 1, 2, 3; (MV1x, MV1y) is the MV of the top-right control point; MV_precision is equal to 4 corresponding to 1/16 motion vector fraction accuracy. Since the difference between integer parts of MVminx and MVmaxx (MVminy and MVmaxy) is 1 pixel, the max difference between integer parts of the four 4×4 sub-block MVs is no more than 1 pixel) perform motion compensation for the plurality of units of the sub-block; and (Zhang468 [0143] The memory bandwidth reduction for the affine mode is controlled by restricting the motion vector difference between the affine control points (also named as the control points difference). In general, if the control points differences satisfy the restriction below, the affine motion is using 4×4 sub-blocks (namely 4×4 affine mode). Otherwise, it is using 8×8 sub-blocks (8×8 affine mode). The restrictions for the 6-parameters and 4-parameters model are given as follows.) decode the video block based at least on the plurality of motion compensated units. (Zhang468 [0120] Because the CPMVs are only used for the affine motion data in heritance and the non-adjacent CPMVs can be disposed, in actual implementations the CPMVs do not need to be stored for the entire CTU. In order to minimize the storage requirements, a decoder can store only a top-row and left-column of vector context, rather than a full-CTU, or a full-picture worth of vector context, and as shown in FIGS. 9A and 9B. These figures illustrate the vector context state after decoding CU #10. The CPMV context is shown in FIG. 9A with 8-pixel granularity, and the standard sub-block MV context is shown in FIG. 9B with 4-pixel granularity. As each CU is decoded, the corresponding left and top context are updated, shifting the blue and green lines further down and to the right, working the way across the CTU and eventually across the picture.)
Zhang468 teach limitations of independent claims from different embodiments. It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Zhang468 to teach all limitations of independent claims in a single embodiment with predictable results because ordinary skills know different embodiments can be integrated using obvious software change.
Regarding to claim 37, 45 and 53:
37. Zhang468 teach the device of claim 36, wherein the sub-block is a 4x4 subblock or an 8x8 sub-block. (Zhang468 [0143] The memory bandwidth reduction for the affine mode is controlled by restricting the motion vector difference between the affine control points (also named as the control points difference). In general, if the control points differences satisfy the restriction below, the affine motion is using 4×4 sub-blocks (namely 4×4 affine mode). Otherwise, it is using 8×8 sub-blocks (8×8 affine mode). The restrictions for the 6-parameters and 4-parameters model are given as follows.)
Regarding to claim 40, 48 and 54:
40. Zhang468 teach the device of claim 36, wherein the processor being configured to perform motion compensation for the plurality of units of the sub-block comprises the processor being further configured to: determine a first unit motion vector associated with a first unit of the plurality of units; obtain a first motion compensated unit based on the first unit motion vector; determine a second unit motion vector associated with a second unit of the plurality of units; and obtain a second motion compensated unit based on the second unit motion vector. (Zhang468 Fig. 3 [0031] The motion vector field (MVF) of a block is described by the following equation with the 4-parameter affine model and 6-parameter affine model respectively:
PNG
media_image1.png
248
671
media_image1.png
Greyscale
[0032] Herein, (mv.sup.h.sub.0, mv.sup.h.sub.0) is motion vector of the top-left corner control point (CP), and (mv.sup.h.sub.1, mv.sup.h.sub.1) is motion vector of the top-right corner control point and (mv.sup.h.sub.2, mv.sup.h.sub.2) is motion vector of the bottom-left corner control point, (x, y) represents the coordinate of a representative point relative to the top-left sample within current block. The CP motion vectors may be signaled (like in the affine AMVP mode) or derived on-the-fly (like in the affine merge mode). w and h are the width and height of the current block. In practice, the division is implemented by right-shift with a rounding operation. In VTM, the representative point is defined to be the center position of a sub-block, e.g., when the coordinate of the left-top corner of a sub-block relative to the top-left sample within current block is (xs, ys), the coordinate of the representative point is defined to be (xs+2, ys+2).)
Regarding to claim 42 and 50:
42. Zhang468 teach the device of claim 36, wherein the determination of whether to split the sub-block for motion compensation is based on a maximum difference between a motion vector associated with the sub-block and one or more control point motion vectors (CPMVs) associated with the sub-block. (Zhang468 [0142] In some embodiments, there may be restrictions to the affine mode for the worst-case bandwidth reduction. To ensure that the worst-case bandwidth of the affine block is not worse than an INTER_4×8/INTER_8×4 block or even an INTER_9×9 block, the motion vector differences between affine control points are used to decide whether the subblock size of the affine block is 4×4 or 8×8. [0143] The memory bandwidth reduction for the affine mode is controlled by restricting the motion vector difference between the affine control points (also named as the control points difference). In general, if the control points differences satisfy the restriction below, the affine motion is using 4×4 sub-blocks (namely 4×4 affine mode). Otherwise, it is using 8×8 sub-blocks (8×8 affine mode). The restrictions for the 6-parameters and 4-parameters model are given as follows. [0144] To derive the constraints for different block sizes (w×h), the motion vector differences of the control points are normalized. [0226] (c) In one example, CPMVs of the current block are derived from the motion vector and parameters stored in a neighboring block, and these CPMVs serves as MVPs for the signaled CPMVs of the current block. [0227] (d) In one example, CPMVs of the current block are derived from the motion vector and parameters stored in a neighboring block, and these CPMVs are used to derive the MVs of each sub-block used for motion compensation. CPMVs and motion vector associated with the sub-block are interchangeable because [0131] For example, after the MVs of all sub-blocks inside the current block are calculated with affine model, the MV of the sub-blocks containing the control points are firstly replaced with the corresponding control point MV)
Regarding to claim 43 and 51:
43. Zhang468 teach the device of claim 36, wherein the determination of whether to split the sub-block for the motion compensation is based on a maximum difference between a motion vector associated with the sub- block and respective motion vectors associated with one or more samples of the sub-block. (Zhang468 [0141] Herein, (MVxi, MVyi) is the MV of ith sub-block within one 8×8 block, where i is 0, 1, 2, 3; (MV1x, MV1y) is the MV of the top-right control point; MV_precision is equal to 4 corresponding to 1/16 motion vector fraction accuracy. Since the difference between integer parts of MVminx and MVmaxx (MVminy and MVmaxy) is 1 pixel, the max difference between integer parts of the four 4×4 sub-block MVs is no more than 1 pixel [maximum difference]. [0142] In some embodiments, there may be restrictions to the affine mode for the worst-case bandwidth reduction. To ensure that the worst-case bandwidth of the affine block is not worse than an INTER_4×8/INTER_8×4 block or even an INTER_9×9 block, the motion vector differences between affine control points are used to decide whether the subblock size of the affine block is 4×4 or 8×8. So, each 8×8 blocks are split to 4×4 sub-block to maintain 1 pixel max difference in motion vector)
Claims 38-39 and 46-47 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang468 (U.S. Pub. No. 20210243468), in view of Liu (U.S. Pub. No. 20210235083 A1).
Regarding to claim 38 and 46:
38. Zhang468 teach the device of claim 36, Zhang468 do not explicitly teach wherein the sub-block is split horizontally into the plurality of units.
However Liu teach wherein the sub-block is split horizontally into the plurality of units. (Liu [0439] the video block is horizontally split into the plurality of sub-blocks each having a height not larger than the second threshold if the height of the video block is larger than the second threshold)
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Zhang468, further incorporating Liu in video/camera technology. One would be motivated to do so, to incorporate the sub-block is split horizontally into the plurality of units. This functionality will improve efficiency with predictable results.
Regarding to claim 39 and 47:
39. Zhang468 teach the device of claims 36, Zhang468 do not explicitly teach wherein the sub-block is split vertically into the plurality of units.
However Liu teach wherein the sub-block is split vertically into the plurality of units. (Liu [0438] the video block is vertically split into the plurality of sub-blocks each having a width not larger than the first threshold if the width of the video block is larger than the first threshold; and)
Claims 41, 49 and 55 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang468 (U.S. Pub. No. 20210243468), in view of Zhang897 (U.S. Pub. No. 20220224897 A1).
Regarding to claim 41, 49 and 55:
41. Zhang468 teach the device of claim 40, Zhang468 do not explicitly teach wherein the processor is further configured to perform prediction refinement with optical flow (PROF) for the first unit of the plurality of units, wherein the performance of the PROF for the first unit comprises the processor being further configured to: determine a difference between a motion vector associated with the sub-block and the first unit motion vector associated with the first unit of the plurality of units; and refine the first motion compensated unit of the plurality of motion compensated units based on the determined difference.
However Zhang897 teach wherein the processor is further configured to perform prediction refinement with optical flow (PROF) for the first unit of the plurality of units, wherein the performance of the PROF for the first unit comprises the processor being further configured to: determine a difference between a motion vector associated with the sub-block and the first unit motion vector associated with the first unit of the plurality of units; and refine the first motion compensated unit of the plurality of motion compensated units based on the determined difference. (Zhang897 FIG. 29 teach sub-block MV [0486] To achieve a finer granularity of motion compensation, this contribution proposes a method to refine the sub-block based affine motion compensated prediction with optical flow. After the sub-block based affine motion compensation is performed, luma prediction sample is refined by adding a difference derived by the optical flow equation. The proposed PROF (prediction refinement with optical flow) is described as following four steps. [0487] Step 1) The sub-block-based affine motion compensation is performed to generate sub-block prediction I (i,j). [0488] Step 2) The spatial gradients g.sub.x(i,j) and g.sub.y (i,j) of the sub-block prediction are calculated at each sample location using a 3-tap filter [−1, 0, 1].
g.sub.x(i,j)=I(i+1,j)−1(i−1,j)
g.sub.y(i,j)=I(i,j+1)−I(i,j−1)
[0489] The sub-block prediction is extended by one pixel on each side for the gradient calculation. To reduce the memory bandwidth and complexity, the pixels on the extended borders are copied from the nearest integer pixel position in the reference picture. Therefore, additional interpolation for padding region is avoided. [0490] Step 3) The luma prediction refinement (denoted ΔI) as is calculated by the optical flow equation.
PNG
media_image2.png
46
393
media_image2.png
Greyscale
where the delta MV (denoted as Δv(i,j)) is the difference between pixel MV computed for sample location (i,j), denoted by v(i,j), and the sub-block MV of the sub-block to which pixel (i,j) belongs, as shown in FIG. 29. This is indicated by the arrows in FIG. 29)
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to modify Zhang468, further incorporating Zhang897 in video/camera technology. One would be motivated to do so, to incorporate perform prediction refinement with optical flow (PROF) for the first unit of the plurality of units. This functionality will improve efficiency with predictable results.
Closely related prior art
Examiner notes teaching of Zhang102 (U.S. Pub. No. 20180070102 A1) is/are pertinent to the independent claim(s), however is not used because dependent claims are covered by primary reference.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NASIM N NIRJHAR whose telephone number is (571) 272-3792. The examiner can normally be reached on Monday - Friday, 8 am to 5 pm ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William F Kraig can be reached on (571) 272-8660. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NASIM N NIRJHAR/Primary Examiner, Art Unit 2896
Prosecution Insights