DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The rejection under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ) of claims 2-16 has been withdrawn in view of the amendments.
The prior Office Action indicated allowability of claim 6 as a whole if rewritten to include the base and intervening claim limitations. Applicant incorporated only a portion of claim 6 into the independent claims. That prior indication therefore does not establish that the incorporated portion, standing alone, is patentable over all prior art.
Applicant’s arguments with respect to the previously cited art fails to disclose the amended limitation requiring a filter and a CNN-based down-sampling model to be combined for a down-sampling ratios have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Deshpande et al. (US20220321919A1), hereinafter referred to as Deshpande, in view of Lin et al. (US 20240236327 A1) hereinafter referred to as Lin.
Regarding claim 1, Deshpande discloses method of processing video data, comprising:
determining to apply neural network (NN) based super resolution (See ¶¶[0033], [0325] and [00345] disclosing determining whether to apply NNSR (neural network super resolution)); and
performing a conversion between a current video block of a video and a bitstream of the video based on the determining (See ¶¶[0016] and [0033] disclosing performing a conversion between a current video block of a video and a bitstream based on the NN SR determination).
Deshpande does not explicitly disclose wherein a filter and a down-sampling model that is convolutional neural network (CNN) based are combined for a down-sampling ratio.
However, Lin from the same or similar endeavor of image processing discloses wherein a filter and a down-sampling model that is convolutional neural network (CNN) based are combined for a down-sampling ratio. (See ¶¶[0093]-[0095] disclosing a CNN-based down-sampling method used together with a distinct filtering/resampling method to achieve a specified re-sampling ratio),
It would have been obvious to the person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings disclosed by Deshpande to add the teachings of Lin as above, in order to achieve a better down-sampling quality compared with the traditional down-sampling technology (Lin, [0051]).
Regarding claim 17 , Deshpande and Lin disclose all the limitations of claim 1, and is analyzed as previously discussed with respect to that claim.
Furthermore, Deshpande discloses the method of claim 1, wherein the conversion comprises encoding the current video block into the bitstream (See ¶[0016])
Regarding claim 18 , Deshpande and Lin disclose all the limitations of claim 1, and is analyzed as previously discussed with respect to that claim.
Furthermore, Deshpande discloses the method of claim 1, wherein the conversion comprises decoding the current video block from the bitstream (See ¶[0016])
Regarding claim 19, this claim is rejected based on the same art and evidentiary limitations applied to the method of claim 1, since it claims analogous subject matter in the form of an apparatus for performing the same or equivalent functionality.
Furthermore, Deshpande discloses apparatus for processing video data comprising a processor and a non-transitory memory with instructions (See ¶¶[0360] and [0361])
Regarding claim 20, this claim is rejected based on the same art and evidentiary limitations applied to the method of claim 1, since it claims analogous subject matter for performing the same or equivalent functionality.
Furthermore, Deshpande discloses storing the bitstream in a non-transitory computer-readable recording medium (See ¶¶[0360] and [0361]).
Claims 2, 3, 7-16 and 21are rejected under 35 U.S.C. 103 as being unpatentable over Deshpande and Lin, in view of Kang (US 20230058283 A1) hereinafter referred to as Kang.
Regarding claim 2 , Deshpande and Lin disclose all the limitations of claim 1, and is analyzed as previously discussed with respect to that claim.
Further, Deshpande discloses the method of claim 1, wherein at least one frame in a sequence of the video is performed with a down-sampling process (See ¶ [0033]),
wherein an indication of a down-sampling model is included in a sequence header, a sequence parameter set (SPS), a picture parameter set (PPS), a picture header, a slice header, a coding tree unit (CTU), a coding tree block (CTB), or a rectangle region of the bitstream (See ¶¶[0325] and [0345]),.
Deshpande does not explicitly disclose the wherein the down-sampling model comprises a filter, wherein different down-sampling models are applied for different color components, wherein the down-sampling model is required by a decoder side; and wherein the down-sampling model is NN based, and the NN is convolutional neural network (CNN).
However, Kang from the same or similar endeavor of image processing discloses wherein the down-sampling model comprises a filter (See ¶[0128]),
wherein different down-sampling models are applied for different color components (See ¶¶[0116] and [0128]),
wherein the down-sampling model is required by a decoder side (See ¶[0147])
It would have been obvious to the person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings disclosed by Deshpande and Lin to add the teachings of Kang as above, in order to improve encoding efficiency in video encoding and decoding for frames having luma signals and chroma signals in various sampling formats in one video sequence (Kang, [0008]).
Regarding claim 3, Deshpande, Lin, and Kang disclose all the limitations of claim 2, and is analyzed as previously discussed with respect to that claim.
Deshpande does not explicitly disclose the method of claim 2, wherein a discrete cosine transform interpolation filter (DCTIF) or bilinear interpolation or bicubic interpolation is used for the down-sampling process, and wherein an index indicating a down-sampling filter and/or at least one coefficient of the down-sampling filter are included in the bitstream
However, Kang from the same or similar endeavor of image processing discloses the method of claim 2, wherein a discrete cosine transform interpolation filter (DCTIF) or bilinear interpolation or bicubic interpolation is used for the down-sampling process (See ¶[0128]) , and wherein an index indicating a down-sampling filter and/or at least one coefficient of the down-sampling filter are included in the bitstream (See ¶¶[0107]-[0110] and [0125]).
The motivation for combining Deshpande, Lin, and Kang has been discussed in connection with claim 2, above.
Regarding claim 8, Deshpande, Lin, and Kang disclose all the limitations of claim 2, and is analyzed as previously discussed with respect to that claim.
Furthermore, Deshpande discloses the method of claim 2, wherein the indication of the down-sampling model is signaled to a decoder, wherein for one video unit level, an index of a chosen down-sampling model is signaled to the decoder, and wherein different CTUs within one frame use different down-sampling models, and all indices of corresponding down-sampling models are signaled to the decoder (See ¶¶[0325] and [0345]).
The motivation for combining Deshpande, Lin, and Kang has been discussed in connection with claim 2, above.
Regarding claim 9, Deshpande, Lin, and Kang disclose all the limitations of claim 2, and is analyzed as previously discussed with respect to that claim.
Furthermore, Deshpande discloses the method of claim 2, wherein an input of the down-sampling model is at a video unit level, and the video unit comprises at least one of: a sequence, a picture, a slice, a tile, a brick, a subpicture, a CTU, a CTU row, one or more coding units (CUs), one or more CTUs, one or more CTBs, or wherein the input is a frame level with a size of an original resolution of the frame level, or wherein the input is one CTU level with a size of 128x128, or wherein the input is a block within one frame whose size is not limited, or
wherein the input is a block with a spatial size (M, N), where M=256, N=128. (See ¶[0016], [0022] and [0345]).
Regarding claim 10, Deshpande, Lim, Kang and Chen disclose all the limitations of claim 2, and is analyzed as previously discussed with respect to that claim.
Deshpande does not explicitly disclose the method of claim 2, wherein down-sampling ratios are different for all video unit levels, and a video unit comprises at least one of: a sequence, a picture, a slice, a tile, a brick, a subpicture, a CTU, a CTU row, one or more coding units (CUs), one or more CTUs, one or more CTBs, wherein a down-sample ratio is 2 for all frames of one sequence, or the down-sample ratio is 2 for all CTUs of one frame, or the down-sample ratio is 2 for a first frame and is 4 for a next frame, or the down-sample ratio is 2 for one frame and is 4 for one CTU in a same frame so that the CTU is down-sampled by 4x, and wherein a combination of down-sampling ratios for different video unit levels is used.
However, Kang from the same or similar endeavor of image processing discloses the method of claim 2, wherein down-sampling ratios are different for all video unit levels, and a video unit comprises at least one of: a sequence, a picture, a slice, a tile, a brick, a subpicture, a CTU, a CTU row, one or more coding units (CUs), one or more CTUs, one or more CTBs, wherein a down-sample ratio is 2 for all frames of one sequence, or the down-sample ratio is 2 for all CTUs of one frame, or the down-sample ratio is 2 for a first frame and is 4 for a next frame, or the down-sample ratio is 2 for one frame and is 4 for one CTU in a same frame so that the CTU is down-sampled by 4x, and wherein a combination of down-sampling ratios for different video unit levels is used (See ¶[0116]).
The motivation for combining Deshpande, Lim, Kang and Chen has been discussed in connection with claim 2, above.
Regarding claim 11, Deshpande, Lin, and Kang disclose all the limitations of claim 2, and is analyzed as previously discussed with respect to that claim.
Deshpande does not explicitly disclose the method of claim 2, wherein down-sampling ratios are different for all components of an input video unit level, or wherein a down-sampling ratio is 2 for both luma component and chroma component, or wherein the down-sampling ratio is 2 for the luma component and is 4 for the chroma component.
However, Kang from the same or similar endeavor of image processing discloses the method of claim 2, wherein down-sampling ratios are different for all components of an input video unit level, or wherein a down-sampling ratio is 2 for both luma component and chroma component, or wherein the down-sampling ratio is 2 for the luma component and is 4 for the chroma component (See ¶¶ [0116] and [0128]).
The motivation for combining Deshpande, Lin, and Kang has been discussed in connection with claim 2, above.
Regarding claim 12, Deshpande, Lin, and Kang disclose all the limitations of claim 2, and is analyzed as previously discussed with respect to that claim.
Deshpande does not explicitly disclose the method of claim 2, wherein a down-sampling ratio is 1 which indicates that no down-sampling is performed, and wherein the down-sampling ratio is applied at all video unit levels, and a video unit comprises at least one of: a sequence, a picture, a slice, a tile, a brick, a subpicture, a CTU, a CTU row, one or more coding units (CUs), one or more CTUs, one or more CTBs.
However, Kang from the same or similar endeavor of image processing discloses the method of claim 2, wherein a down-sampling ratio is 1 which indicates that no down-sampling is performed, and wherein the down-sampling ratio is applied at all video unit levels, and a video unit comprises at least one of: a sequence, a picture, a slice, a tile, a brick, a subpicture, a CTU, a CTU row, one or more coding units (CUs), one or more CTUs, one or more CTBs (See ¶[0116]).
The motivation for combining Deshpande, Lin, and Kang has been discussed in connection with claim 2, above.
Regarding claim 13, Deshpande, Lin, and Kang disclose all the limitations of claim 2, and is analyzed as previously discussed with respect to that claim.
Deshpande does not explicitly disclose the method of claim 2, wherein a down-sampling ratio is determined by comparison, wherein in case of there being 2x and 4x down-sampling ratios for one frame that are available, an encoder compresses the frame with 2x down-sampling and then compresses the frame with 4x down-sampling, up-samples low-resolution reconstruction with a same up-sampling model, and calculates a quality metric of each result, so that the down-sampling ratio that achieves a best reconstruction quality is chosen as an implemented down-sampling ratio for compression, wherein the quality metric comprises multi-scale structural similarity index measure (MS-SSIM) or peak signal-to-noise ratio (PSNR); wherein different quality metrics are used as metric for the comparison, and the quality metric comprises PSNR, structural similarity index measure (SSIM), MS-SSIM, or video multi-method assessment fusion (VMAF).
However, Chen from the same or similar endeavor of image processing discloses the method of claim 2, wherein a down-sampling ratio is determined by comparison, wherein in case of there being 2x and 4x down-sampling ratios for one frame that are available, an encoder compresses the frame with 2x down-sampling and then compresses the frame with 4x down-sampling, up-samples low-resolution reconstruction with a same up-sampling model, and calculates a quality metric of each result, so that the down-sampling ratio that achieves a best reconstruction quality is chosen as an implemented down-sampling ratio for compression, wherein the quality metric comprises multi-scale structural similarity index measure (MS-SSIM) or peak signal-to-noise ratio (PSNR) (See ¶[0129]);
wherein different quality metrics are used as metric for the comparison, and the quality metric comprises PSNR, structural similarity index measure (SSIM), MS-SSIM, or video multi-method assessment fusion (VMAF) (See ¶[0129])
The motivation for combining Deshpande, Lin, and Kang has been discussed in connection with claim 2, above.
Furthermore, Deshpande discloses the wherein determination of the down-sampling ratio is performed at an encoder or at a decoder, wherein in case that the determination of the down-sampling ratio is performed at the decoder, distortion is calculated based on samples other than a current picture, a current slice, a current CTU, a current CTB, or a current rectangle region (See ¶[0016]).
wherein the down-sampling ratio is present in a video unit level, and CNN information is included in the SPS, the PPS, the picture header, the slice header, the CTU, or the CTB. (See ¶¶[0325] and [0345])
Regarding claim 14, Deshpande, Lin, and Kang disclose all the limitations of claim 2, and is analyzed as previously discussed with respect to that claim.
Deshpande does not explicitly disclose the method of claim 2, wherein a chroma format of input is changed depending on different down-sampling ratios of color components, and a changed chroma format is used for compression in an encoder,
wherein the chroma format is changed from YUV 4:2:0 to YUV 4:4:4 when a down-sampling ratio is 2 for a luma component and is 1 for chroma components, and YUV 4:4:4 is used as the chroma format for compression,
However, Kang from the same or similar endeavor of image processing discloses the method of claim 2, wherein a chroma format of input is changed depending on different down-sampling ratios of color components, and a changed chroma format is used for compression in an encoder (See ¶¶ [0116] and [0131]),
wherein the chroma format is changed from YUV 4:2:0 to YUV 4:4:4 when a down-sampling ratio is 2 for a luma component and is 1 for chroma components, and YUV 4:4:4 is used as the chroma format for compression (See ¶¶ [0116] and [0131]),
Furthermore, Deshpande discloses the wherein information to recover an original chroma format is present in a video unit level, wherein the down-sampling ratios for all the color components are included in the SPS, the PPS, the picture header, the slice header, the CTU, or the CTB, and wherein the original chroma format is included in the SPS, the PPS, the picture header, the slice header, the CTU, or the CTB (See ¶¶[0325] and [0345])
The motivation for combining Deshpande, Lin, and Kang has been discussed in connection with claim 2, above.
Regarding claim 15, Deshpande, Lin, and Kang disclose all the limitations of claim 2, and is analyzed as previously discussed with respect to that claim.
Furthermore, Deshpande discloses the method of claim 2, wherein a luma component is down-sampled at all video unit levels, and a video unit comprises at least one of: a sequence, a picture, a slice, a tile, a brick, a subpicture, a CTU, a CTU row, one or more coding units (CUs), one or more CTUs, one or more CTBs, and wherein the luma component that is down-sampled is one frame or one CTU. (See ¶¶[0325] and [0345])
Regarding claim 16, Deshpande, Lin, and Kang disclose all the limitations of claim 2, and is analyzed as previously discussed with respect to that claim.
Furthermore, Deshpande discloses the method of claim 2, wherein the down-sampling model that is NN based is used in an encoder and a decoder, and a NN based in-loop filter is used (See ¶¶ [0033] and [0345]); and
wherein the NN based super resolution is applied to luma components of reconstructed YUV as a post filter to recover an original chroma format, or the NN based super resolution is applied before in-loop filters to recover the original chroma format (See ¶[0325]).
Deshpande does not explicitly disclose the wherein two NN based in-loop filters are applied to down-sampled luma reconstruction and chroma reconstruction, respectively; and wherein as one input of the NN based in-loop filter for chroma, the down-sampled luma reconstruction is concatenated with chroma reconstruction
However, Kang from the same or similar endeavor of image processing discloses wherein two NN based in-loop filters are applied to down-sampled luma reconstruction and chroma reconstruction, respectively; and wherein as one input of the NN based in-loop filter for chroma, the down-sampled luma reconstruction is concatenated with chroma reconstruction (See ¶[0116]),
The motivation for combining Deshpande, Lin, and Kang has been discussed in connection with claim 2, above.
Regarding claim 21, this claim is rejected based on the same art and evidentiary limitations applied to the method of claim 2, since it claims analogous subject matter in the form of an apparatus for performing the same or equivalent functionality.
Claims 7 is rejected under 35 U.S.C. 103 as being unpatentable over Deshpande and Lin, in view of Kang (US 20230058283 A1) hereinafter referred to as Kang, and further, in view of Chen (US 20230069953 A1), hereinafter referred to as Chen.
Regarding claim 7, Deshpande, Lin, and Kang disclose all the limitations of claim 2, and is analyzed as previously discussed with respect to that claim.
Furthermore, Deshpande discloses the method of claim 2, wherein when down-sampling an input video unit level, the down-sampling model is chosen by comparing different down-sampling models (See ¶[0345]) wherein in case of there being three down-sampling models that are CNN based, for one input, the three down-sampling models down-sample the input, respectively, down-sampled reconstruction is up-sampled to an original resolution (See ¶¶[0033] and [0345]) wherein indices of the down-sampling models are signaled to an encoder or a decoder (See ¶[0016]).
Deshpande does not explicitly disclose and a quality metric is utilized to measure three up-sampled results.
However, Kang from the same or similar endeavor of image processing discloses a quality metric is utilized to measure three up-sampled results (See ¶[0116])
Furthermore, Chen from the same or similar endeavor of image processing discloses so that the down-sampling model that achieves a best performance is utilized as an implemented down-sampling, wherein the quality metric comprises multi-scale structural similarity index measure (MS-SSIM) or peak signal-to-noise ratio (PSNR)(See ¶0129])
It would have been obvious to the person of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings disclosed by Deshpande, Lin and Kang to add the teachings of Chen as above, in order to provide an efficient way of image modification by employing features of machine learning (Chen, [0008]).
Allowable Subject Matter
Claims 4 and 6 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to FABIO S LIMA whose telephone number is (571)270-0625. The examiner can normally be reached on Monday - Friday 8 am - 4 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jamie Atala can be reached on (571) 272-7384. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/FABIO S LIMA/Primary Examiner, Art Unit 2486
Prosecution Insights