DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Currently, claims 1-15 are pending in the application. Claims 1, 13, and 15 are amended.
Response to Arguments / Amendments
Applicant’s arguments have been fully considered but are rendered moot in view of the new ground of rejection necessitated by amendments initiated by the applicant.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2 and 10-15 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (US 20210195201, hereinafter, Li) in view of Chen et al. (US 20220109860, hereinafter Chen) and PFAFF et al. (US 20240137500, hereinafter PFAFF).
Regarding Claim 1, Li discloses an artificial intelligence (AI)-based video decoding method ([0064], a central processing unit (CPU), and the coprocessor may be a graphics processing unit (GPU) configured to be responsible for rendering and drawing content that a display screen needs to display with an artificial intelligence (AI) processor computing operation related to machine learning) comprising:
obtaining, from a bitstream, a joint chroma residual sample of a current block, Cb component prediction information of the current block, and Cr component prediction information of the current block, the current block comprising a Cb component and a Cr component ([0049] VVC: A published version ( such as VVC draft 7, WET-P2001) of the Specification of VVC standard that details out chroma component prediction; [0050], [0081], Chroma components having symbols Cb and Cr, specifying that a sample array or single sample is representing one of the two color difference signals related to the primary colors with signaling information such as sequence information and parameters for groups of pixels/samples (the groups including pictures, tiles, slices, macroblocks, CTUs, CUs, blocks, transform units, prediction units, etc.), to reconstruct such groups in the video);
determining a prediction sample of the Cb component of the current block based on at least the Cb component prediction information ([0133] VVC supports a mode JCCR (Joint Cb Cr Residual coding) where the chroma residuals are coded jointly and a joint chroma coding mode is indicated by a TU-level flag (also referred as a block-level flag) tu_joint_cbcr_residual_flag and the selected mode is implicitly indicated by the chroma CBFs (coded block flags));
determining a prediction sample of the Cr component of the current block based on at least the Cr component prediction information ([0133] VVC: The flag tu_joint_cbcr_residual_flag is present if either or both chroma CBFs for a TU are equal to 1 and chroma QP offset values are signalled for the joint chroma residual coding mode to differentiate from the chroma QP offset values signalled for regular chroma residual coding mode. These chroma QP offset values are used to derive the chroma QP values for those blocks coded using the joint chroma residual coding mode. When a corresponding joint chroma coding mode (modes 2 in Table 4) is active in a TU, this chroma QP offset is added to the applied luma-derived chroma QP during quantization and decoding of that TU); and
reconstructing the current block by obtaining a reconstructed sample of the Cb component of the current block and a reconstructed sample of the Cr component of the current block ([0092], FIG. 9, VVC with the application of adaptive color transform (ACT): converting the residuals from the input color space to YCgCo space, and then the encoding process moves on to the transform/quantization step based on the residuals coded in YCgCo space; [0108], perform inverse ACT process that coverts residuals at YCgCo domain obtained from inverse transform to residuals at RGB domain after the conversion is the residual information to be combined with the prediction information for reconstructing the current CB).
Li does not explicitly disclose reconstructing the current block as an output of a neural network for reconstructing the Cb component and the Cr Component by inputting data obtained by concatenating the joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component for each sample of the current block to the neural network.
Chen reconstructing the current block as an output of a neural network for reconstructing the Cb component and the Cr Component by inputting data obtained by concatenating the joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component for each sample of the current block to the neural network ([0080], concatenate the downsampled luma component with the filtered downsampled first chroma component and the filtered second chroma component to form concatenated color components and then filter the concatenated color components using a convolutional neural network filter; [0090], FIG. 4A, convolutional NN filter 172 upsamples chroma components (that is, blue hue (Cb) and red hue (Cr) components) to the size of a corresponding luminance (luma (Y) component. And Concatenation unit 174 concatenates the upsampled chrominance samples with luma samples to be used as inputs to the NN-based filter formed by residual blocks 176.).
PNG
media_image1.png
392
554
media_image1.png
Greyscale
Therefore, it would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of neural network by inputting the joint chroma residual sample as taught by Chen ([0080]) into the encoding & decoding system of Li in order to enable reducing or removing redundancy inherent in video sequences and allow a video device to transmit, receive, encode, decode, and store digital video information more efficiently by implementing video coding techniques (Chen, [0135]).
Li and Chen do not explicitly disclose wherein a plurality of models for the neural network are determined based on whether a prediction type of the current block is an intra prediction or inter prediction, and wherein a model of the neural network is determined from among the plurality of models.
PFAFF teaches wherein a plurality of models for the neural network are determined based on whether a prediction type of the current block is an intra prediction or inter prediction, and wherein a model of the neural network is determined from among the plurality of models ([0053], FIG. 5, plurality 66 of intra-prediction modes comprises a set 72 of neural network-based intra-prediction modes as well as a set 74 of further intra-prediction mode of heuristic design. Additionally or alternatively, set 74 may comprise inter-prediction modes which may be called angular inter-prediction modes according to which sample values of the neighboring sample set 60 are copied into block 18 along a certain intra-prediction direction with this intra-prediction direction differing among such angular intra-prediction modes; [0148], FIG. 7a , a flag 70a within the side information 70 for block 18 indicates whether the intra-prediction mode determined to be the best mode for block 18 by the encoder in step 90, is within set 72, i.e., is neural network based intra-prediction mode).
Therefore, it would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of plurality of models for the neural network are determined according to a prediction type of the current block as taught by PFAFF ([0040]) into the encoding & decoding system of Li and Chen in order to provide systems for using DNN to provide improved compression efficiency is achieved by letting block-wise picture codec support the set of intra-prediction modes based on which the intra-prediction signal for the current block of the picture is determined (PFAFF, [0040]).
Regarding Claim 2, Li in view of Chen and PFAFF discloses the AI-based video decoding method of claim 1, Li discloses wherein the reconstructing of the current block further comprises: obtaining at least one of a residual sample of the Cb component of the current block or a residual sample of the Cr component of the current block by inputting the joint chroma residual sample, the prediction sample of the Cb component, and the prediction sample of the Cr component to the neural network; and reconstructing the current block by obtaining the reconstructed sample of the Cb component and the reconstructed sample of the Cr component by using at least one of the residual sample of the Cb component or the residual sample of the Cr component, the prediction sample of the Cb component, and the prediction sample of the Cr component ([0082] The control signal can conditionally restrict the decoder from using a specific scheme to process color components of the current CU in the prediction residual domain. The control signal may include a Joint Cb Cr Residual (JCCR) enabling flag or a (block-based delta pulse code modulation) BDPCM chroma enabling flag that may restrict the decoder from using adaptive color transform (ACT) on the current CU; or the control signal may include an ACT enabling flag that may restrict the decoder from applying inverse JCCR or BDPCM on chroma components of the current CU. The control signal may indicate size threshold that restrict the decoder from using ACT at block level when block size of the current CU is greater than the size threshold. Details of restrictions could be posed by the control signal to the coding process are described below in accordance with embodiments in step S4060
Regarding Claim 10, Li in view of Chen and PFAFF discloses the AI-based video decoding method of claim 1, Li discloses wherein the neural network is trained according to first lossy information corresponding to a difference between an original sample of a Cb component of an original block for training and a reconstructed sample of a Cb component of a reconstructed block for training which is obtained via the neural network, and second lossy information corresponding to a difference between an original sample of a Cr component of the original block for training and a reconstructed sample of a Cr component of the reconstructed block for training which is obtained via the neural network ([0040] FIG. 5, using a decoding process comprising one or a combination of a residual decoding process to generate reconstructed residual from the video bitstream, a prediction process to generate a prediction signal related to each picture, a reconstruction process to generate reconstructed picture from the reconstructed residual and the prediction signal, and at least one filtering process applied to the reconstructed picture in step 520 and using DNN (Deep Neural Network) in step 530, where the target signal provided to DNN input corresponds to the reconstructed residual, output from the prediction process, the reconstruction process or said at least one filtering process, or a combination thereof).
Regarding Claim 11, Li in view of Chen and PFAFF discloses the AI-based video decoding method of claim 10, Li discloses wherein the model of the neural network is determined based on Cb coded block flag (cbf) information indicating whether a transformation coefficient level of the current block with respect to the Cb component comprises a non-zero Cb component, and Cr cbf information indicating whether a transformation coefficient level of the current block with respect to the Cr component comprises a non-zero Cr component, the Cb cbf information and the Cr cbf information being obtained from the bitstream ([0133] VVC: The flag tu_joint_cbcr_residual_flag is present if either or both chroma CBFs for a TU are equal to 1 and chroma QP offset values are signalled for the joint chroma residual coding mode to differentiate from the chroma QP offset values signalled for regular chroma residual coding mode. These chroma QP offset values are used to derive the chroma QP values for those blocks coded using the joint chroma residual coding mode. When a corresponding joint chroma coding mode (modes 2 in Table 4) is active in a TU, this chroma QP offset is added to the applied luma-derived chroma QP during quantization and decoding of that TU);
Regarding Claim 12, Li in view of Chen and PFAFF discloses the AI-based video decoding method of claim 11, Li discloses wherein the model of the neural network is determined according to at least one of the type of a slice comprising the current block, a QP range of the slice, the Cb component and the Cr component of the current block ([0133] VVC: The flag tu_joint_cbcr_residual_flag is present if either or both chroma CBFs for a TU are equal to 1 and chroma QP offset values are signalled for the joint chroma residual coding mode to differentiate from the chroma QP offset values signalled for regular chroma residual coding mode. These chroma QP offset values are used to derive the chroma QP values for those blocks coded using the joint chroma residual coding mode. When a corresponding joint chroma coding mode (modes 2 in Table 4) is active in a TU, this chroma QP offset is added to the applied luma-derived chroma QP during quantization and decoding of that TU);
Regarding Claims 13-14, Apparatus claims 13-14 of using the corresponding method claimed in claims 1 & 2, and the rejections of which are incorporated herein for the same reasons as used above.
Regarding Claim 15, Video encoding claim 15 of using the corresponding decoding method claimed in claim 1, and the rejections of which are incorporated herein for the same reasons as used above.
Claims 3-9 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al. (US 20210195201, hereinafter, Li) in view of Chen et al. (US 20220109860, hereinafter Chen), PFAFF et al. (US 20240137500, hereinafter PFAFF) and IWAMURA et al. (US 20230269383, hereinafter IWAMURA)
Regarding Claim 3, Li in view of Chen and PFAFF discloses the AI-based video decoding method of claim 2, but does not explicitly disclose wherein the obtaining of the at least one of the residual sample of the Cb component or the residual sample of the Cr component further comprises: refining the residual sample of the Cb component by applying a first scale factor to the residual sample of the Cb component obtained via the neural network, and refining the residual sample of the Cr component by applying a second scale factor to the residual sample of the Cr component obtained via the neural network.
IWAMURA teaches refining the residual sample of the Cb component by applying a first scale factor to the residual sample of the Cb component obtained via the neural network, and refining the residual sample of the Cr component by applying a second scale factor to the residual sample of the Cr component obtained via the neural network ([0033], scaler 103 performs chroma residual scaling for the whole encoding-target block of a chrominance component and when the prediction residual of a chrominance component is defined as C.Res, the scaler 103 calculates and outputs the prediction residual of the chrominance component after scaling, C.ResScale, by C.Res*C.Scale, or C.Res/C.ScaleInv; [0099], FIG. 4, when the color space transform application flag indicates that the decoding-target block has been encoded using a color space transform, the scaler 212 determines to perform chroma residual scaling and the scaler 212 and the combiner 213 perform a chrominance block reconstruction process involving chroma residual scaling (step S3); [0092], scaler 212 performs chroma residual scaling when the color space transform application flag (cu_act_enabled_flag) is TRUE (“1”) irrespective of the significant coefficient flags of chrominance components (tu_cb_coded_flag, tu_cr_coded_flag).
Therefore, it would have been obvious to one ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of neural network by inputting the joint chroma residual sample as taught by IWAMURA ([0040]) into the encoding & decoding system of Li in order to provide systems for allowing chroma residual scaling to be properly applied, which can improve the encoding efficiency (IWAMURA, [0092]).
Regarding Claim 4-6, Analogous rejection as the rejection of Claim 3 applies.
Regarding Claim 7, Li in view of Chen, PFAFF and IWAMURA discloses the AI-based video decoding method of claim 6, Chen discloses wherein the neural network is trained to determine a correlation between the Cb component and the Cr component for each sample of a current block for training by receiving input values of a joint chroma residual sample for training, a prediction sample of a Cb component for training, and a prediction sample of a Cr component for training ([0040] FIG. 5, using a decoding process comprising one or a combination of a residual decoding process to generate reconstructed residual from the video bitstream, a prediction process to generate a prediction signal related to each picture, a reconstruction process to generate reconstructed picture from the reconstructed residual and the prediction signal, and at least one filtering process applied to the reconstructed picture in step 520 and using DNN (Deep Neural Network) in step 530, where the target signal provided to DNN input corresponds to the reconstructed residual, output from the prediction process, the reconstruction process or said at least one filtering process, or a combination thereof). The same reason or rational of obviousness motivation applied as used above in claim 1.
Regarding Claim 8, Li in view of Chen, FAFF and IWAMURA discloses the AI-based video decoding method of claim 7, Li discloses wherein, for the correlation between Cb and Cr, one or more weights are respectively determined for the residual sample of the Cb component and the residual sample of the Cr component ([0133] VVC: The flag tu_joint_cbcr_residual_flag is present if either or both chroma CBFs for a TU are equal to 1 and chroma QP offset values are signalled for the joint chroma residual coding mode to differentiate from the chroma QP offset values signalled for regular chroma residual coding mode. These chroma QP offset values are used to derive the chroma QP values for those blocks coded using the joint chroma residual coding mode. When a corresponding joint chroma coding mode (modes 2 in Table 4) is active in a TU, this chroma QP offset is added to the applied luma-derived chroma QP during quantization and decoding of that TU).
Regarding Claim 9, Li in view of Chen, FAFF and IWAMURA discloses the AI-based video decoding method of claim 7, Li discloses wherein the input values of the neural network further comprise at least one of a quantization step size of the current block, a quantization error of the current block, or a block obtained by downsampling a reconstructed luma block based on a chroma format, the reconstructed luma block corresponding to a current chroma block of the current block ([0040] FIG. 5, using a decoding process comprising one or a combination of a residual decoding process to generate reconstructed residual from the video bitstream, a prediction process to generate a prediction signal related to each picture, a reconstruction process to generate reconstructed picture from the reconstructed residual and the prediction signal, and at least one filtering process applied to the reconstructed picture in step 520 and using DNN (Deep Neural Network) in step 530, where the target signal provided to DNN input corresponds to the reconstructed residual, output from the prediction process, the reconstruction process or said at least one filtering process, or a combination thereof). The same reason or rational of obviousness motivation applied as used above in claim 1.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Samuel D Fereja whose telephone number is (469)295-9243. The examiner can normally be reached 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DAVID CZEKAJ can be reached at (571) 272-7327. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAMUEL D FEREJA/Primary Examiner, Art Unit 2487