DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1, 6, 58, 116, and 117 are objected to because of the following informalities: Claims 1, 6, 58, 116 and 117 recite the abbreviation, CNN. The claims, however, do not recite a definition for the abbreviation. Appropriate correction is required. For the purposes of examination, examiner interprets CNN to mean convolutional neural network.
Claims 8, 10, 54, and 55 are objected to because of the following informalities: claims 8, 10, 54, and 55 recite the abbreviation “FIR”. The claims, however, do not recite a definition for the abbreviation. Appropriate correction is required. For the purposes of examination, examiner interprets FIR to mean “finite infinite response”.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 41, 48, 54 – 56, and 57 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 41 and 48 recite the limitation "the predetermined complexity for the predetermined picture area" in line 4 and lines 4-5, respectively. Neither claims 41 and 48, nor claim 1, the claim upon which claims 41 and 48 properly depend, recite the limitation “a predetermined complexity for a predetermine picture area”. There is insufficient antecedent basis for this limitation in the claims.
Regarding claims 54 and 55, the claims recite the limitation “each second pre-filtered version” in lines 18 and 27, respectively. It is unclear whether this recitation is referring to the previously cited “first filtered version” (prior to weighting the first filtered version to obtain the second filtered version) or referring to the previously cited “second filtered version” (prior to a second filtering to obtain the third filtered version). For purposes of examination on the merits, examiner interprets “each second pre-filtered version” to mean the previously cited second filtered version.
Claim 56 properly depends on claim 55 and is therefore likewise rejected.
Regarding claim 57, the claim first recites “perform the switching between performing the soft classification for first pre-reconstructed samples in the first or second manner…” in lines 1 -3. Claim 1, however, the claim upon which claim 57 properly depends, recites “mode switching between one or more first modes of performing the adaptive in-loop filtering…a second mode of bypassing the second in-loop filter.” In other words, claim 1 only recites a mode switching between either applying or not applying adaptive in-loop filtering. The limitation of claim 57, therefore, has insufficient antecedent basis. Claim 57 further recites “perform the switching between performing soft classification…in the first or second manner…by disabling the soft classification…” While claim 57 does not depend on claim 55, claim 55 provides a definition of the first and second manners of soft classification recited in claim 57. Neither the first nor the second manner defined in claim 55 provide for a disabling or a bypassing of the soft classification operation. It is therefore unclear whether the recitation of “disabling the soft classification” refers to one of the two recited soft classification manners of claim 55 or the recitation of “disabling the soft classification” refers to bypassing the second in-loop filter [as recited in the switching operation of claim 1]. For purposes of examination, examiner interprets “perform the switching between performing soft classification…in the first or second manner…by disabling the soft classification…” to mean “mode switching…[to] a second mode of bypassing the second in-loop filter.”
Claim Interpretation
Patentable weight is given to data stored on a computer-readable medium when there exists a functional relationship between the data and its associated substrate. MPEP 2111.05 III. For example, if a claim is drawn to a computer-readable medium containing programming, a functional relationship exists if the programming “performs some function with respect to the computer with which it is associated.” Id. However, if the claim recites that the computer-readable medium merely serves as a support for information or data, no functional relationship exists and the information or data is not given patentable weight. Id.
At present claim 119, is directed to “a non-transitory digital storage medium comprising a bitstream generated by the method of claim 58”, the method of claim 58 comprising a plurality of steps. While the method recited in claim 58 may be performed by an intended computer, the method is not stored on the non-transitory digital storage medium. Rather, only bitstream data is stored on the digital storage medium. It is the bitstream itself, therefore, that must have a functional relationship. Because there are no recitations of the bitstream causing an intended computer to perform some function, Examiner finds that there is no disclosed or claimed functional relationship between the stored bitstream and the medium. Instead, the medium is merely a support or carrier for the bitstream being stored. Therefore, the bitstream stored and the way such bitstream is decoded are not given patentable weight. As such, claim 119 is subject to a prior art rejection based on any non-transitory computer readable storage medium known before the earliest effective filing date of the present application.
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “apparatus for decoding…configured to” in claim 1, and “apparatus for encoding…configured to” in claim 58.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof. The corresponding structure describing in the specification is a “hardware apparatus”, as recited in lines 21 – 22 of page 51 of the originally filed specification.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Examiner Remarks
Claims are interpreted in the alternative only.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 119 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Karczewicz et al. (US 2022/0201292) (hereinafter Karczewicz).
As discussed above, claim 119 has been interpreted as nonfunctional descriptive material under MPEP 2111.05(III) and associated case law cited therein because claim 119 recites “a non-transitory digital storage medium comprising a bitstream generated by the method of claim 58.” As such, claim 119 is subject to a prior art rejection based on any non-transitory digital storage medium known before the earliest effective filing date of the present application. In other words, the proper interpretation of claim 119 is merely a machine-readable media in which the media is merely support or carrier for the bitstream being stored wherein the bitstream stored and the way such bitstream is encoded should not be given patentable weight. Karczewicz teaches a non-transitory digital storage medium storing a bitstream comprising video information (Karczewicz, e.g. Fig. 1, element 110, and pars. 38 and 43: depicting and describing a computer readable storage medium storing encoded video data, wherein encoded video data is the equivalent of the bitstream).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 6, 8, 10, 40 – 49, 52, 54 – 56, 58, 116, 117, and 119 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ye et al. (US 2010/0158103) (hereinafter Ye) in view of Karczewicz et al. (US 2022/0201292)(hereinafter Karczewicz) in view of Ma et al. (WO 2022/072659) (hereinafter Ma).
Regarding claims 1, 58, 116, 117, and 119, Ye teaches an apparatus for decoding a video from a bitstream, the decoding apparatus configured to perform a decoding method, an apparatus for encoding a video into a bitstream, the encoding apparatus configured to perform an encoding method, the method for decoding a video bitstream, the method for encoding bitstream, and a non-transitory digital storage medium storing a bitstream generated by the encoding method, wherein the decoding method and the encoding method comprise:
reconstruct, based on the bitstream, the video using block-based predictive decoding, transform-based residual decoding and a prediction loop into which an in-loop filter tool is serially connected (e.g. Fig. 3 and pars. 65 – 67: depicting and describing that the system reconstructs, from encoded video, the video using predictive decoding [element 54], transform-based residual decoding [elements 56 and 58], and an in-loop filter serially connected to the prediction loop [e.g. Fig. 3, depicting a prediction loop [element 54] being serially connected to in loop filter [element 67]),
wherein the second in-loop filter is configured to subject pre-reconstructed samples of a current picture to an adaptive in-loop filtering, ALF (e.g. par. 73: describing that the second filter is an adaptive loop filter),
wherein the second in-loop filter is configured to perform, based on the bitstream, a mode switching one or more first modes of performing the adaptive in-loop filtering, and optionally, a second mode of bypassing the second in-loop filter (e.g. par. 73: describing that the system determines whether or not to apply adaptive loop filtering to a block of the bitstream).
Ye does not explicitly teach:
wherein the in-loop filter tool comprises a serial connection of a first in-loop filter and a second in-loop filter,
wherein the one or more first modes involve the second in-loop filter assigning a classification to pre-reconstructed samples of the current picture and filtering the pre-reconstructed samples with a filter transfer function which is adapted to the classification,
wherein the classification of the one or more first modes is a soft-classification, and
wherein each of the first modes uses a CNN.
Karczewicz, however, teaches an encoding apparatus, a decoding apparatus, an encoding method, a decoding method, and a non-transitory digital storage medium:
wherein the in-loop filter tool comprises a serial connection of a first in-loop filter and a second in-loop filter (e.g. Fig. 17 and pars. 258 – 259: depicting and describing that the loop filter includes a serial connection of a first loop filter [deblocking filter and SAO filter] and a second loop filter [adaptive loop filtering [alf]),
wherein the one or more first modes involve the second in-loop filter assigning a classification to pre-reconstructed samples of the current picture and filtering the pre-reconstructed samples with a filter transfer function which is adapted to the classification (e.g. Fig. 9, and pars. 113 – 150: depicting and describing that in applying adaptive loop filtering, the system determines a classification for each sample and filters the sample using a filter associated with each classification), and
wherein the classification of the one or more first modes is a soft-classification (e.g. par. 153: describing that the classification step classifies a sample into multiple classes, wherein classifying a sample into multiple classes is the equivalent of the soft-classification).
Ma, however, teaches an encoding apparatus, a decoding apparatus, an encoding method, a decoding method, and a non-transitory digital storage medium:
wherein each of the first modes uses a CNN (e.g. Fig. 18 and par. 143: depicting and describing that the adaptive loop filter uses a CNN).
It therefore would have been obvious to one of ordinary skill in the art to modify the teachings of Ye by adding the teachings of Karczewicz in order for the in-loop filter to include a serial connection between a first in-loop filter and a second in-loop filter, for the one or more first modes to involve the second in-loop filter assigning a classification to pre-reconstructed samples of the current picture and filtering the pre-reconstructed samples with a filter transfer function which is adapted to the classification, and for the classification to be a soft classification, and by adding the teachings of Ma in order for each of the first modes to use a CNN. One of ordinary skill in the art would have been motivated to make such a modification because the modification improves overall quality of decoded video data by accounting for local features of video data better than can be done by a single stage ALF (Karczewicz, e.g. par. 36: describing a desire to improve the overall quality of decoded video data) and because the modification improves coding efficiency by applying neural network technique to video coding (Ma, e.g. par. 8: describing a desire to improve coding efficiency by applying neural network techniques to video coding).
Turning to claim 6, Ye, Karczewicz, and Ma teach all of the limitations of claim 1, as discussed above. Ye does not explicitly teach:
wherein the classification of the one or more first modes is CNN based.
Ma, however, teaches an apparatus for decoding a video from a bitstream:
wherein the classification of the one or more first modes is CNN based (e.g. Fig. 18 and par. 143: depicting and describing that classification is performed using CNN).
It therefore would have been obvious to one of ordinary skill in the art to modify the teachings of Ye by adding the teachings of Ma in order for the classification of the one or more first modes is CNN based. One of ordinary skill in the art would have been motivated to make such a modification because the modification improves coding efficiency by applying neural network technique to video coding (Ma, e.g. par. 8: describing a desire to improve coding efficiency by applying neural network techniques to video coding).
Regarding claim 8, Ye, Karczewicz, and Ma teach all of the limitations of claim 1, as discussed above. Ye does not explicitly teach:
wherein the second in-loop filter is configured to perform the adaptive in-loop filtering by use of FIR filters adapted in a sample- wise manner.
Karczewicz, however, teaches apparatus for decoding a video from a bitstream:
wherein the second in-loop filter is configured to perform the adaptive in-loop filtering by use of FIR filters adapted in a sample- wise manner (e.g. Figs. 9, 11, and 13, and pars. 114 – 152: depicting and describing that the system performs adaptive loop filtering by applying filters adapted in a sample-wise manner based on classification)
It therefore would have been obvious to one of ordinary skill in the art to modify the teachings of Ye by adding the teachings of Karczewicz in order for the second in-loop filter to be configured to perform the adaptive in-loop filtering by use of FIR filters adapted in a sample-wise manner. One of ordinary skill in the art would have been motivated to make such a modification because the modification improves overall quality of decoded video data by accounting for local features of video data better than can be done by a single stage ALF (Karczewicz, e.g. par. 36: describing a desire to improve the overall quality of decoded video data).
Turning to claim 10, Ye, Karczewicz, and Ma teach all of the limitations of claim 1, as discussed above. Ye does not explicitly teach:
wherein the one or more first modes involve the second in-loop filter assigning a classification to pre-reconstructed samples of the current picture and filtering the pre-reconstructed samples with a filter transfer function which is adapted to the classification, the classification of the one or more first modes is a soft-classification,
wherein the second in-loop filter is configured to perform the soft classification for first pre-reconstructed samples by:
assigning, for each first pre-reconstructed sample, a classification value to each of a first set of classes, with each of which an associated FIR filter is associated, and
performing the adaptive in-loop filtering, in case of using the soft classification for the assigning the classification, by, at each first pre- reconstructed sample, applying, for each class of the first set of classes, the associated FIR filter associated with the respective class to the pre-reconstructed samples to obtain a filter result, and forming a weighted sum of the filter results of the first set of classes according to the classification values.
Karczewicz, however, teaches an apparatus for decoding a video from a bitstream:
wherein the one or more first modes involve the second in-loop filter assigning a classification to pre-reconstructed samples of the current picture and filtering the pre-reconstructed samples with a filter transfer function which is adapted to the classification, the classification of the one or more first modes is a soft-classification (Fig. 9, and pars. 113 – 150: depicting and describing that for each sample [R(x,y)], the system classifies the sample into a first set of classes, each of the first set of classes having an associated first filter [First Stage], the system then applying the associated filters for each class to the sample to generate intermediate filtered samples, wherein the associated filters are FIR filters, and wherein classifying a sample into a plurality of classes in a first set of classes is the equivalent of the first classification),
wherein the second in-loop filter is configured to perform the soft classification for first pre-reconstructed samples by:
assigning, for each first pre-reconstructed sample, a classification value to each of a first set of classes, with each of which an associated FIR filter is associated (Fig. 9, and pars. 113 – 150: depicting and describing that for each sample [R(x,y)], the system classifies the sample into a first set of classes, each of the first set of classes having an associated first filter [First Stage], the system then applying the associated filters for each class to the sample to generate intermediate filtered samples, wherein the associated filters are FIR filters, and wherein the intermediate filtered samples are the equivalent of the first filtered version), and
performing the adaptive in-loop filtering, in case of using the soft classification for the assigning the classification, by, at each first pre- reconstructed sample, applying, for each class of the first set of classes, the associated FIR filter associated with the respective class to the pre-reconstructed samples to obtain a filter result, and forming a weighted sum of the filter results of the first set of classes according to the classification values (e.g. pars. 207 – 211: describing that the system determines a weight for each class the sample is classified into and performs a weighted sum of the intermediate filtered samples to obtain a filtered sample of the sample).
It therefore would have been obvious to one of ordinary skill in the art to modify the teachings of Ye by adding the teachings of Karczewicz in order for the one or more first modes assigning a classification using a soft classification, the soft classification performed by assigning the sample to each of a first set of classes, each class with an associated FIR filter, and performing filtering of the sample by applying the FIR filter to the sample based on the class and the associated FIR filter. One of ordinary skill in the art would have been motivated to make such a modification because the modification improves overall quality of decoded video data by accounting for local features of video data better than can be done by a single stage ALF (Karczewicz, e.g. par. 36: describing a desire to improve the overall quality of decoded video data).
Regarding claim 40, Ye, Karczewicz, and Ma teach all of the limitations of claim 1, as discussed above. Ye further teaches:
configured to perform the mode switching by use of a syntax element in the bitstream (e.g. par. 76: describing that the system determines whether to apply or not apply ALF filtering to a block is based on filter flag sent in the bitstream, wherein the filter flag is the equivalent of the syntax element), and
configured to perform the mode switching by:
determining, within a predetermined picture area, a measure for prediction quality or prediction imperfection within the predetermined picture area (e.g. Fig. 5, element 507, and pars. 82 – 83: depicting and describing that the system determines whether there are non-zero residuals in the block, wherein whether there are non-zero residuals is the equivalent of the measure for prediction quality or prediction imperfection, and wherein the block is the equivalent of the predetermined picture area), and
checking whether the measure for prediction or prediction imperfection fulfills a further predetermined criterion, and if so, inferring that the syntax element, if same relates to the predetermined picture area, assumes a predetermined value not corresponding to any first mode, or any first mode exceeding a predetermined complexity, or has a decreased value domain which excludes the one or more first modes, or any first mode exceeding the predetermined complexity, and is decreased relative to a complete value domain the syntax element has outside the predetermined picture area, so that a bit rate for signaling at least one value in the decreased value domain, which does not correspond to any first mode, or any first mode exceeding the predetermined complexity, has a smaller bitrate consumption than compared to a corresponding value in the complete value domain (e.g. Fig. 5, elements 507, 508, and 509, and pars. 82 – 83: depicting and describing that when the system determines that a block does not contain any non-zero residual values, the system determines that ALF flag is 0, indicating that ALF filtering is not applied to the block, wherein whether a block contains any non-zero residual values is the equivalent of the measure for prediction quality or prediction imperfection fulfilling a predetermined criterion, and wherein determining a value of the ALF flag is zero when there are no non-zero residual values in a block is the equivalent of inferring the syntax element for the predetermined picture area is a value not corresponding to any first mode).
Turning to claim 41, Ye, Karczewicz, and Ma teach all of the limitations of claim 1, as discussed above. Ye further teaches:
configured to perform the mode switching based on measure for prediction quality or prediction imperfection within a predetermined picture area by disabling the one or more first modes, or any first mode exceeding the predetermined complexity, for the predetermined picture area if the measure for prediction quality or prediction imperfection fulfills a further predetermined criterion (e.g. Fig. 5, element 507, and pars. 82 – 83: depicting and describing that the system determines whether to apply [ALFflag = 1] or skip [ALFflag = 0] ALF filtering for a block based on whether the block has any non-zero residuals, wherein non-zero residuals is the equivalent of the measure for prediction quality or prediction imperfection, and wherein the block is the equivalent of the predetermined picture area, and wherein skipping ALF filtering is the equivalent of disabling the one or more first modes).
Regarding claim 42, Ye, Karczewicz, and Ma teach all of the limitations of claims 1 and 40, as discussed above. Ye further teaches:
wherein the measure for prediction quality or prediction imperfection includes one or more of the prediction residual being zero within the predetermined picture area, the areal fraction in which the prediction residual is zero, a number of coded non-zero transform coefficients, an energy of coded transform coefficients (e.g. Fig. 5, element 507, and pars. 82 – 83: depicting and describing that the system determines whether there are non-zero prediction residuals in the block).
Turning to claim 43, Ye, Karczewicz, and Ma teach all of the limitations of claims 1 and 40, as discussed above. Ye further teaches:
wherein the predetermined picture area is a coding tree root block, coding block, or slice (e.g. Fig. 5, element 507, and pars. 82 – 83: depicting and describing that the predetermined picture area is a coding block).
Regarding claim 44, Ye, Karczewicz, and Ma teach all of the limitations of claim 1, as discussed above. Ye further teaches:
configured to perform the mode switching by use of a syntax element in the bitstream (e.g. pars. 34 and 76: describing that the system determines whether to apply or not apply ALF filtering to a block is based on filter flag sent in the bitstream, wherein the filter flag is the equivalent of the syntax element), and
configured to perform the mode switching by:
determining a prediction type or inter-prediction hierarchy level of a picture (e.g. Fig. 4 and par. 81: depicting and describing that the system determines whether a block is an intra coded block or an inter-coded block, wherein determining whether a block is intra-coded or inter-coded is the equivalent of determining a prediction type), and
checking whether the prediction type or inter-prediction hierarchy level fulfils an even further predetermined criterion, and if so, inferring that the syntax element, if same relates to the picture, assumes a predetermined value not corresponding to any first mode, or any first mode exceeding a predetermined complexity, or has a decreased value domain which excludes the one or more first modes, or any first mode exceeding a predetermined complexity, and is decreased relative to a complete value domain the syntax element has outside the picture, so that a bit rate for signaling at least one value in the decreased value domain, which does not correspond to any first mode, or any first mode exceeding the predetermined complexity, has a smaller bitrate consumption than compared to a corresponding value in the complete value domain (e.g. Figs. 4 and 9, and pars. 81, and 96 - 97: depicting and describing that when the block is inter-coded, the system determines to not apply the ALF filtering to the block, wherein not applying ALF filtering when a block is inter-coded is the equivalent of inferring the syntax element does not correspond to the first mode) .
Turning to claim 45, Ye, Karczewicz, and Ma teach all of the limitations of claim 1, as discussed above. Ye further teaches:
configured to perform the mode switching based on prediction type or inter-prediction hierarchy level of a picture by disabling the one or more first modes, or any first mode exceeding a predetermined complexity for the picture if the measure for prediction quality or prediction imperfection fulfills a even further predetermined criterion (e.g. Figs. 4 and 9, and pars. 81 and 96 – 97: depicting and describing that the system determines whether to apply ALF filtering based on whether a block is inter-coded or intra-coded).
Regarding claim 46, Ye, Karczewicz, and Ma teach all of the limitations of claims 1 and 44, as discussed above. Ye further teaches:
wherein the prediction type indicates whether the picture is inter-predicted based on reference pictures preceding and succeeding the picture in presentation time order, with the even further predetermined criterion being fulfilled if this is the case, and/or the inter- prediction hierarchy level of a picture indicates a temporal hierarchy level of the picture in a GOP, with the even further predetermined criterion being fulfilled if the hierarchy level exceeds same threshold (e.g. Figs. 4 and 9, and pars. 81 and 96 – 97: depicting and describing that the system determines does not apply ALF filtering when the block is inter-coded, wherein it is known to those of ordinary skill in the art that an inter-coded block is a block that uses reference pictures preceding in time order, succeeding in time order, or both preceding and succeeding in time order).
Turning to claim 47, Ye, Karczewicz, and Ma teach all of the limitations of claim 1, as discussed above. Ye further teaches:
configured to perform the mode switching by use of a syntax element in the bitstream (e.g. pars. 34 and 76: describing that the system determines whether to apply or not apply ALF filtering to a block is based on filter flag sent in the bitstream, wherein the filter flag is the equivalent of the syntax element), and
configured to perform the mode switching by
checking whether a predetermined picture portion has at least one reference picture which succeeds a picture of the predetermined picture portion in presentation time order (e.g. Figs. 4 and 9, and pars. 81, and 96 - 97: depicting and describing that the system determines whether a block is inter-coded, wherein it is known to those of ordinary skill in the art that an inter-coded block uses reference pictures preceding in time order, succeeding in time order, or both preceding and succeeding in time order), and
if so, inferring that the syntax element, if same relates to the predetermined picture portion, assumes a predetermined value not corresponding to any first mode, or any first mode exceeding a predetermined complexity, or has a decreased value domain which excludes the one or more first modes, or any first mode exceeding a predetermined complexity, and is decreased relative to a complete value domain the syntax element has outside the predetermined picture portion, so that a bit rate for signaling at least one value in the decreased value domain, which does not correspond to any first mode, or any first mode exceeding the predetermined complexity, has a smaller bitrate consumption than compared to a corresponding value in the complete value domain (e.g. Figs. 4 and 9, and pars. 81, and 96 - 97: depicting and describing that when the block is inter-coded, the system determines to not apply the ALF filtering to the block, wherein not applying ALF filtering when a block is inter-coded is the equivalent of inferring the syntax element does not correspond to the first mode).
Regarding claim 48, Ye, Karczewicz, and Ma teach all of the limitations of claim 1, as discussed above. Ye further teaches:
configured to perform the mode switching in dependence on whether for a predetermined picture portion at least one reference picture succeeds a picture of the predetermined picture portion in presentation time order by disabling the one or more first modes, or any first mode exceeding the predetermined complexity for the predetermined picture portion if this is the case (e.g. Figs. 4 and 9, and pars. 81 and 96 – 97: depicting and describing that the system determines does not apply ALF filtering when the block is inter-coded, wherein it is known to those of ordinary skill in the art that an inter-coded block is a block that uses reference pictures preceding in time order, succeeding in time order, or both preceding and succeeding in time order).
Turning to claim 49, Ye, Karczewicz, and Ma teach all of the limitations of claims 1 and 47, as discussed above. Ye further teaches:
wherein the predetermined picture portion is a slice or a whole picture (e.g. par. 64: describing that syntax information defines how to apply filtering for blocks within a slice).
Regarding claim 52, Ye, Karczewicz, and Ma teach all of the limitations of claims 1 and 47, as discussed above. Ye further teaches:
wherein the predetermined picture portion is a slice, a whole picture or a CTU or a CU (e.g. par. 64: describing that syntax information defines how to apply filtering for blocks within a slice).
Turning to claim 54, Ye, Karczewicz, and Ma teach all of the limitations of claim 1, as discussed above. Ye does not explicitly teach:
wherein the second in-loop filter is configured to perform the soft classification for first pre-reconstructed samples by:
assigning, for each first pre-reconstructed sample, a classification value to each of a first set of classes, with each of which an associated first FIR filter and a second filter is associated, and
performing the adaptive in-loop filtering, in case of using the soft classification for the assigning the classification, by
applying, for each class of the first set of classes, the associated first FIR filter associated with the respective class onto the pre-reconstructed samples to obtain a first filtered version,
weighting, for each class of the first set of classes, the first filtered version at each sample position with the classification value assigned to the respective class for the first pre- reconstructed sample at the respective sample position to obtain an second filtered version,
applying, for each class of the first set of classes, the associated second FIR filter associated with the respective class onto the second filtered version to obtain a third filtered version,
subjecting, for each second pre-filtered version, the third filtered version obtained for the classes of the first set, to a summation,
wherein for each class of the first set of classes coefficients, and/or a size and/or shape of a kernel of the second FIR filter are conveyed in the bitstream.
Karczewicz, however, teaches an apparatus for decoding a video from a bitstream:
wherein the second in-loop filter is configured to perform the soft classification for first pre-reconstructed samples by:
assigning, for each first pre-reconstructed sample, a classification value to each of a first set of classes, with each of which an associated first FIR filter and a second filter is associated (Karczewicz, Fig. 9 and pars. 113 – 150: depicting and describing that for each sample [R(x,y)], the system classifies the sample into a first set of classes, each of the first set of classes having an associated first filter and second filter [associated filter for each classification in first stage [element 400] and second stage [element 410]], wherein the associated filters are FIR filters), and
performing the adaptive in-loop filtering, in case of using the soft classification for the assigning the classification, by
applying, for each class of the first set of classes, the associated first FIR filter associated with the respective class onto the pre-reconstructed samples to obtain a first filtered version (Fig. 9, and pars. 113 – 150: depicting and describing that for each sample [R(x,y)], the system classifies the sample into a first set of classes, each of the first set of classes having an associated first filter [First Stage], the system then applying the associated filters for each class to the sample to generate intermediate filtered samples, wherein the associated filters are FIR filters, and wherein the intermediate filtered samples are the equivalent of the first filtered version),
weighting, for each class of the first set of classes, the first filtered version at each sample position with the classification value assigned to the respective class for the first pre- reconstructed sample at the respective sample position to obtain an second filtered version (e.g. par. 207 – 211: describing that the system determines a weight for each intermediate filtered sample) ,
applying, for each class of the first set of classes, the associated second FIR filter associated with the respective class onto the second filtered version to obtain a third filtered version (e.g. Fig. 9 and pars. 113 – 150: depicting and describing that the system applies a second filter [second stage] to the intermediate filtered samples, a signaled filter associated with each of the classifications),
subjecting, for each second pre-filtered version, the third filtered version obtained for the classes of the first set, to a summation (e.g. par. 207 – 211: describing that the system performs a weighted sum of all filtered samples to obtain a filtered sample),
wherein for each class of the first set of classes coefficients, and/or a size and/or shape of a kernel of the second FIR filter are conveyed in the bitstream (e.g. par. 149: describing that the second filter set is signaled in the bitstream, wherein the second filter set is a second FIR filter).
It therefore would have been obvious to one of ordinary skill in the art to modify the teachings of Ye by adding the teachings of Karczewicz in order to perform the soft classification by assigning a classification value to each of a first set of classes, each class associated with a first FIR filter and a second filter, and perform the adaptive loop filtering using soft classification by applying, for each class of the first set of classes, the associated FIR filter to obtain a first filtered version, weighting the first filtered version based on the classification value to obtain a second filtered version, applying the second FIR filter to the second filtered version to obtain a third filtered version, and subjecting the third filtered version to a summation. One of ordinary skill in the art would have been motivated to make such a modification because the modification improves overall quality of decoded video data by accounting for local features of video data better than can be done by a single stage ALF (Karczewicz, e.g. par. 36: describing a desire to improve the overall quality of decoded video data).
Regarding claim 55, Ye, Karczewicz, and Ma teach all of the limitations of claim 1, as discussed above. Ye does not explicitly teach:
wherein the second in-loop filter is configured to switch, based on the bitstream, between 1) performing the soft classification for first pre-reconstructed samples in a first manner by assigning, for each first pre-reconstructed sample, a classification value to each of a first set of classes, with each of which an associated FIR filter is associated, and performing the adaptive in-loop filtering, in case of using the soft classification for the assigning the classification, by applying, at each first pre- reconstructed sample, for each class of the first set of classes, the associated FIR filter associated with the respective class to the pre-reconstructed samples to obtain a filter result, and forming a weighted sum of the filter results of the first set of classes according to the classification values; and 2) performing the soft classification for first pre-reconstructed samples in a second manner by assigning, for each first pre-reconstructed sample, a classification value to each of a first set of classes, with each of which an associated first FIR filter and a second filter is associated, and performing the adaptive in-loop filtering, in case of using the soft classification for the assigning the classification, by
applying, for each class of the first set of classes, the associated first FIR filter associated with the respective class onto the pre-reconstructed samples to obtain a first filtered version,
weighting, for each class of the first set of classes, the first filtered version at each sample position with the classification value assigned to the respective class for the first pre- reconstructed sample at the respective sample position to obtain an second filtered version,
applying, for each class of the first set of classes, the associated second FIR filter associated with the respective class onto the second filter version to obtain a third filtered version,
subjecting, for each second pre-filtered version, the third filtered version obtained for the classes of the first set, to a summation,
wherein for each class of the first set of classes coefficients, and/or a size and/or shape of a kernel of the second FIR filter are conveyed in the bitstream.
Karczewicz, however, teaches an apparatus for decoding a video from a bitstream:
wherein the second in-loop filter is configured to switch, based on the bitstream, between 1) performing the soft classification for first pre-reconstructed samples in a first manner by assigning, for each first pre-reconstructed sample, a classification value to each of a first set of classes, with each of which an associated FIR filter is associated, and performing the adaptive in-loop filtering, in case of using the soft classification for the assigning the classification, by applying, at each first pre- reconstructed sample, for each class of the first set of classes, the associated FIR filter associated with the respective class to the pre-reconstructed samples to obtain a filter result, and forming a weighted sum of the filter results of the first set of classes according to the classification values; and 2) performing the soft classification for first pre-reconstructed samples in a second manner by assigning, for each first pre-reconstructed sample, a classification value to each of a first set of classes, with each of which an associated first FIR filter and a second filter is associated, and performing the adaptive in-loop filtering, in case of using the soft classification for the assigning the classification (e.g. Fig. 13 and pars. 207 – 212: depicting and describing that the system switches between 1) classifying, for each sample [R(x,y)] into a plurality of a first set of classes, each of the first set of classes having an associated filter, the system performing filtering of each sample by applying the associated filters based on the class and obtaining a weighted sum to determine a filtered sample, and 2) classifying each sample into a first set of classifications, each classification associated with a first filter and a second filter, the system performing the filtering based on applying the first filter and second filter based on the identified classes, the switching determined by filter usage information, the filter usage information being information in the bitstream [see, e.g. par. 209: describing that the method of adaptive loop filtering performed is determined by filter usage information signaled in the bitstream]), by
applying, for each class of the first set of classes, the associated first FIR filter associated with the respective class onto the pre-reconstructed samples to obtain a first filtered version (Fig. 9, and pars. 113 – 150: depicting and describing that for each sample [R(x,y)], the system classifies the sample into a first set of classes, each of the first set of classes having an associated first filter [First Stage], the system then applying the associated filters for each class to the sample to generate intermediate filtered samples, wherein the associated filters are FIR filters, and wherein the intermediate filtered samples are the equivalent of the first filtered version),
weighting, for each class of the first set of classes, the first filtered version at each sample position with the classification value assigned to the respective class for the first pre- reconstructed sample at the respective sample position to obtain an second filtered version (e.g. par. 207 – 211: describing that the system determines a weight for each intermediate filtered sample),
applying, for each class of the first set of classes, the associated second FIR filter associated with the respective class onto the second filter version to obtain a third filtered version (e.g. Fig. 9 and pars. 113 – 150: depicting and describing that the system applies a second filter [second stage] to the intermediate filtered samples, a signaled filter associated with each of the classifications) ,
subjecting, for each second pre-filtered version, the third filtered version obtained for the classes of the first set, to a summation (e.g. par. 207 – 211: describing that the system performs a weighted sum of all filtered samples to obtain a filtered sample),
wherein for each class of the first set of classes coefficients, and/or a size and/or shape of a kernel of the second FIR filter are conveyed in the bitstream (e.g. par. 149: describing that the second filter set is signaled in the bitstream, wherein the second filter set is a second FIR filter).
It therefore would have been obvious to one of ordinary skill in the art to modify the teachings of Ye by adding the teachings of Karczewicz in order to switch between performing the soft classification in a first manner and a second manner based on the bitstream by applying the first FIR filter associated with each class to obtain a first filtered version, weighting, for each class, the first filtered version to obtain a second filtered version, applying, for each class, the second associated filter to obtain a third filtered version, and subjecting the third filtered version to obtain a summation. One of ordinary skill in the art would have been motivated to make such a modification because the modification improves overall quality of decoded video data by accounting for local features of video data better than can be done by a single stage ALF (Karczewicz, e.g. par. 36: describing a desire to improve the overall quality of decoded video data).
Turning to claim 56, Ye, Karczewicz, and Ma teach all of the limitations of claims 1 and 55, as discussed above. Ye does not explicitly teach:
configured to perform the switching between performing the performing the soft classification for first pre-reconstructed samples in the first or second manner in units of one or more of: coding tree root blocks into which the current picture is pre-subdivided in rows and columns of coding tree root blocks, and from which onwards the picture is subdivided into coding blocks by recursive multi-tree partitioning of the coding tree root blocks, coding blocks into which the current picture is subdivided by pre-subdividing the current picture into coding tree root blocks in rows and columns of coding tree root blocks, and subdividing the picture further from the coding tree root blocks onwards by recursive multi-tree partitioning of the coding tree root blocks, and slices of the current picture; pictures of the video, a sequence of Pictures of the video, the video.
Karczewicz, however, teaches an apparatus for decoding a video from a bitstream:
configured to perform the switching between performing the performing the soft classification for first pre-reconstructed samples in the first or second manner in units of one or more of: coding tree root blocks into which the current picture is pre-subdivided in rows and columns of coding tree root blocks, and from which onwards the picture is subdivided into coding blocks by recursive multi-tree partitioning of the coding tree root blocks, coding blocks into which the current picture is subdivided by pre-subdividing the current picture into coding tree root blocks in rows and columns of coding tree root blocks, and subdividing the picture further from the coding tree root blocks onwards by recursive multi-tree partitioning of the coding tree root blocks, and slices of the current picture; pictures of the video, a sequence of Pictures of the video, the video (e.g. pars. 209: describing that the filter usage information is signaled in units of coding tree block, coding block, slice, picture of the video [PPS] or sequence of pictures of the video [SPS]).
It therefore would have been obvious to one of ordinary skill in the art to modify the teachings of Ye by adding the teachings of Karczewicz in order to perform the switching between the performing of the classification in the first manner and the second manner in units of one or more of coding tree root blocks into which the current picture is pre-subdivided, coding blocks, slices, pictures of the video, sequence of pictures of the video or the video. One of ordinary skill in the art would have been motivated to make such a modification because the modification improves overall quality of decoded video data by accounting for local features of video data better than can be done by a single stage ALF (Karczewicz, e.g. par. 36: describing a desire to improve the overall quality of decoded video data).
Claim(s) 39 and 57 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ye et al. (US 2010/0158103) (hereinafter Ye) in view of Karczewicz et al. (US 2022/0201292)(hereinafter Karczewicz) in view of Ma et al. (WO 2022/072659) (hereinafter Ma) as applied to claim 1 above, and further in view of Xiu et al. (WO 2020/061082) (hereinafter Xiu).
Regarding claim 39, Ye, Karczewicz, and Ma teach all of the limitations of claim 1, as discussed above. Ye does not explicitly teach:
configured to perform the mode switching based on an estimation of a measure of complexity incurred by the second in-loop filter or the one or more first modes of the second in-loop filter within a predetermined video or picture section so far by disabling the one or more first modes, or any first mode exceeding a predetermined complexity for the predetermined video or picture section if the estimation fulfills a predetermined criterion (e.g. exceeds a threshold).
Xiu, however, teaches an apparatus for decoding video from a bitstream:
configured to perform the mode switching based on an estimation of a measure of complexity incurred by the second in-loop filter or the one or more first modes of the second in-loop filter within a predetermined video or picture section so far by disabling the one or more first modes, or any first mode exceeding a predetermined complexity for the predetermined video or picture section if the estimation fulfills a predetermined criterion (e.g. exceeds a threshold) (e.g. pars. 113 and Tables 6: describing that the system disables filtering of blocks above a certain size because the number of multiplications per sample incurred by the filtering blocks above the determined size is greater than the number of multiplications per sample incurred without filtering the block, wherein the number of multiplications per sample is the equivalent of the measure of complexity, and wherein the number of multiplications per sample incurred filtering the block being greater than the number of multiplications incurred without filtering the block is the equivalent of the estimation of a measure for the complexity incurred by the second in-loop filter fulfilling the predetermined criterion).
It therefore would have been obvious to one of ordinary skill in the art to modify the teachings of Ye by adding the teachings of Xie in order to perform the mode switching based on an estimation of a measure of complexity incurred by the second in-loop filter or the one or more first modes of the second in-loop filter within a predetermined video or picture section so far by disabling the one or more first modes, or any first mode exceeding a predetermined complexity for the predetermined video or picture section of the estimation fulfills a predetermined criterion. One of ordinary skill in the art would have been motivated to make such a modification because the modification improves coding efficiency.
Turning to claim 57, Ye, Karczewicz, and Ma teach all of the limitations of claim 1, as discussed above. Ye does not explicitly teach:
configured to perform the switching between performing the soft classification for first pre-reconstructed samples in the first or second manner based on an estimation of a measure for multiplications per sample incurred by the second in-loop filter for the current picture so far by disabling the soft classification if the estimation fulfills a predetermined criterion (e.g. exceeds a threshold).
Xiu, however, teaches an apparatus for decoding a video from a bitstream:
configured to perform the switching between performing the soft classification for first pre-reconstructed samples in the first or second manner based on an estimation of a measure for multiplications per sample incurred by the second in-loop filter for the current picture so far by disabling the soft classification if the estimation fulfills a predetermined criterion (e.g. exceeds a threshold) (e.g. pars. 113 and Tables 6: describing that the system disables filtering of blocks above a certain size because the number of multiplications per sample incurred by the filtering blocks above the determined size is greater than the number of multiplications per sample incurred without filtering the block, wherein the number of multiplications per sample incurred filtering the block being greater than the number of multiplications incurred without filtering the block is the equivalent of the estimation of a measure for multiplications per sample incurred by the second in-loop filter fulfilling the predetermined criterion).
It therefore would have been obvious to one of ordinary skill in the art to modify the teachings of Ye by adding the teachings of Xie in order to perform the mode switching based on an estimation of a measure for multiplications per sample incurred by the second in-loop filter for the current picture by disabling the soft classification if the estimation fulfills a predetermined criterion. One of ordinary skill in the art would have been motivated to make such a modification because the modification improves coding efficiency.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US2024/0015337 – describing an in-loop filtering tool, the in-loop filtering including an adaptive loop filtering tool serial connected to a first loop filtering tool, the adaptive loop filtering tool using multiple classifiers for each sample
WO2023/123398 – describing a system that optionally bypasses the use of a neural network based in-loop filtering during the reconstruction of samples within a video
WO2023/056348 – describing a system that optionally bypasses the use of a neural network based in-loop filtering, determining whether to use or bypass the neural network based loop filtering based on information in the bitstream
WO2021/051369 – providing general knowledge related to use of neural networks for adaptive loop filtering as part of an in-loop filtering process
US2019/0273948 – providing general knowledge of using convolutional neural networks (CNN) for adaptive loop filtering
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHANIKA M BRUMFIELD whose telephone number is (571)270-3700. The examiner can normally be reached M-F 8:30 - 5 PM AWS.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Czekaj can be reached at 571-272-7327. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SHANIKA M. BRUMFIELD
Examiner
Art Unit 2487
/SHANIKA M BRUMFIELD/Examiner, Art Unit 2487
/Dave Czekaj/Supervisory Patent Examiner, Art Unit 2487
Prosecution Insights