DETAILED ACTION
Allowable Subject Matter
Claims 6 and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 7-11, 13, 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Huang (“One-for-All: An efficient variable convolution Neural Network for In-loop filter for VVC,” IEEE 2022) in view of but Fu (“StoHisNet: A hybrid multi-classification model with CNN and Transformer for gastric pathology images,” Computer Methods and Programs in Biomedicine 2021), as evidenced by APXML Courses (“Hybrid CNN-Transformer Models,” https://apxml.com/courses/cnns-for-computer-vision/chapter-5-attention-transformers-vision/hybrid-cnn-transformer-models).
Regarding Claim 1, Huang (“One-for-All: An efficient variable convolution Neural Network for In-loop filter for VVC,” IEEE 2022) discloses a method for enhancing quality of a frame (in loop filter for VVC, title), comprising:
receiving, by a processor (software, page 2348 left column, Section IV.A Experimental settings), the frame and auxiliary information (QP or frame type (FT) value, page 2344 right column, page 2345 left column, Section III.B Attention module) associated with the frame (qp and frame type are associated with the frame, inherent); and applying, by the processor, a neural network-based in-loop filter (VCNN incorporated into VVC, Fig. 2) to the frame (raw frame Y compressed by encoder C using the quantization parameter QP = q, a reconstructed frame X is generated, page 2344 right column, Section III proposed variable CNN) based on the auxiliary information (QP or frame type (FT) value, page 2344 right column, page 2345 left column, Section III.B Attention module) to enhance the quality of the frame (improve the quality of the current frame directly by reducing the compression artifacts, Section I Introduction, page 2342 left column), wherein the NN-based in-loop filter comprises a backbone part (the CNN consists of three parts: the head part, the backbone part, and the reconstruction part, page 2345 right column, Section III.C network architecture) comprising … at least one residual-attention block (residual block with QP attention module and FT attention module, Fig. 5, Section III.B attention module, page 2345 left column), and the at least one RAB comprises at least one attention block (residual block with QP attention module and FT attention module, Fig. 5, Section III.B attention module, page 2345 left column; receives the QP and the FT, Fig. 4) receiving at least part of the auxiliary information (receives the QP and the FT, Fig. 4, Fig. 5, Section III.B attention module, page 2345 left column).
Huang does not disclose, but Fu (“StoHisNet: A hybrid multi-classification model with CNN and Transformer for gastric pathology images,” Computer Methods and Programs in Biomedicine 2021) teaches backbone part (right channel of the network, Fig. 1) comprising at least one transformer block (swin transformer block, Fig. 1) and at least one residual[] block (residual module, Fig. 1).
One of ordinary skill in the art before the application was filed would have been motivated to modify the backbone of the CNN of Huang to include transformer blocks following the residual blocks, as in the right channel of Fu, because Fu teaches, and one persons of ordinary skill in the art knows, that transformer blocks guide the neural network into capturing and preserving long-term, global features of the data, which improves the quality of the CNN output. See summary of “Hybrid CNN-Transformer Models” by APXML Courses for more benefits of combining CNNs with transformers and attention modules.
Regarding Claim 3, Huang (“One-for-All: An efficient variable convolution Neural Network for In-loop filter for VVC,” IEEE 2022) discloses the method of claim 1, wherein the NN-based in-loop filter further comprises a feature extraction part (network architecture consists of three parts: the head part, the backbone part, and the reconstruction part. The head part is responsible for initial feature extraction, Section III.C, page 2345 right column); and
applying the NN-based in-loop filter (in-loop filters integrated with VCNN in VVC, Fig. 2) comprises extracting features from the frame (features F’, refined output, Section III.B attention module, pages 2344-2345) based on the auxiliary information using the feature extraction part (refined based on the attention map M based on QP and FT, Section III.B attention module, pages 2344-2345).
Regarding Claim 7, Huang (“One-for-All: An efficient variable convolution Neural Network for In-loop filter for VVC,” IEEE 2022) discloses the method of claim 3, wherein the RAB further comprises at least one residual block (residual block with QP attention module and FT attention module, Fig. 5, Section III.B attention module); and
applying the NN-based in-loop filter (in-loop filters integrated with VCNN in VVC, Fig. 2) comprises:
processing the features to obtain a local correlation (use of the local refined residual features, Section III.C) between the features (inherent outcome of convolution layers a residual block, See APXML Courses for evidence) using the RAB (residual block with QP attention module and FT attention module, Fig. 5, Section III.B attention module).
Huang does not disclose, but Fu (“StoHisNet: A hybrid multi-classification model with CNN and Transformer for gastric pathology images,” Computer Methods and Programs in Biomedicine 2021) teaches processing the features to obtain a long-range correlation between the features (inherent outcome of a transformer block, APXML Courses) using the transformer block (swin transformer bloc, Fig. 1).
APXML Courses (“Hybrid CNN-Transformer Models,” https://apxml.com/courses/cnns-for-computer-vision/chapter-5-attention-transformers-vision/hybrid-cnn-transformer-models) provides evidence that local correlation between the features using the RAB (local patterns are learned via convolution, page 2, top);
a long-range correlation between the features using the transformer (transformer layers use self-attention to model global dependencies, page 1).
One of ordinary skill in the art before the application was filed would have been motivated to modify the backbone of the CNN of Huang to include transformer blocks following the residual blocks, as in the right channel of Fu, because Fu teaches, and one persons of ordinary skill in the art knows, that transformer blocks guide the neural network into capturing and preserving long-term, global features of the data, which improves the quality of the CNN output. See summary of “Hybrid CNN-Transformer Models” by APXML Courses for more benefits of combining CNNs with transformers and attention modules.
Regarding Claim 8, Huang (“One-for-All: An efficient variable convolution Neural Network for In-loop filter for VVC,” IEEE 2022) discloses the method of claim 1, wherein the frame is a reconstruction frame of luma samples (output of the decoded picture buffer is input to the VCNN, Fig. 2); and
the backbone part comprises three attention blocks (three RRBs in the backbone, Fig. 6).
Huang does not disclose, but Fu (“StoHisNet: A hybrid multi-classification model with CNN and Transformer for gastric pathology images,” Computer Methods and Programs in Biomedicine 2021) renders obvious backbone part comprises six (4 swin transformer blocks, Fig. 1) transformer blocks (swin transformer blocks, Fig. 1).
Where the prior art encompasses the claimed subject matter but not a claimed value, differences in values will not support patentability unless the value is critical, and the value is optimized by more-than-routine experimentation. MPEP 2144.05. Applicant’s specification makes a declaration of the number of blocks without any disclosure of whether, why, or how the value is critical, or whether or how the value has been optimized. Therefore, the claimed value is obvious in view of the prior art.
One of ordinary skill in the art before the application was filed would have been motivated to modify the backbone of the CNN of Huang to include transformer blocks following the residual blocks, as in the right channel of Fu, because Fu teaches, and one persons of ordinary skill in the art knows, that transformer blocks guide the neural network into capturing and preserving long-term, global features of the data, which improves the quality of the CNN output. See summary of “Hybrid CNN-Transformer Models” by APXML Courses for more benefits of combining CNNs with transformers and attention modules.
Regarding Claim 9, Huang (“One-for-All: An efficient variable convolution Neural Network for In-loop filter for VVC,” IEEE 2022) discloses the method of claim 1, wherein the frame is a reconstruction frame of chroma (performance on chroma, Section IV.E, pages 2351-2352) samples (output of the decoded picture buffer is input to the VCNN, Fig. 2).
Huang does not disclose but renders obvious the backbone part comprises one attention block (backbone part makes up of N cascaded recursive reconstruction blocks (RRB). Each RRB comprises N cascaded residual feature aggregation modules, residual block is implemented with the proposed attention module, Fig. 6).
Huang does not disclose but Fu (“StoHisNet: A hybrid multi-classification model with CNN and Transformer for gastric pathology images,” Computer Methods and Programs in Biomedicine 2021) renders obvious and three transformer blocks (4 swin transformer blocks, Fig. 1).
Where the prior art encompasses the claimed subject matter but not a claimed value, differences in values will not support patentability unless the value is critical, and the value is optimized by more-than-routine experimentation. MPEP 2144.05. Applicant’s specification makes a declaration of the number of blocks without any disclosure of whether, why, or how the value is critical, or whether or how the value has been optimized. Therefore, the claimed value is obvious in view of the prior art.
One of ordinary skill in the art before the application was filed would have been motivated to modify the backbone of the CNN of Huang to include transformer blocks following the residual blocks, as in the right channel of Fu, because Fu teaches, and one persons of ordinary skill in the art knows, that transformer blocks guide the neural network into capturing and preserving long-term, global features of the data, which improves the quality of the CNN output. See summary of “Hybrid CNN-Transformer Models” by APXML Courses for more benefits of combining CNNs with transformers and attention modules.
Regarding Claim 10, Huang (“One-for-All: An efficient variable convolution Neural Network for In-loop filter for VVC,” IEEE 2022) discloses the method of claim 3, wherein the NN-based in-loop filter further comprises a reconstruction part (NN consists of three parts: the head part, the backbone part, and the reconstruction part, Section III.C network architecture); and
applying the NN-based in-loop filter (in-loop filters integrated with VCNN in VVC, Fig. 2) comprises:
processing the features (Given a feature map as input, the proposed attention module sequentially infers a 1D attention map M from QP or FT, F’ is the refined output.) based on the at least part of the auxiliary information (residual block with QP attention module and FT attention module receives the QP and FT, Fig. 5, Section III.B attention module, page 2234-2345) to generate global features (backbone part receives the feature F0 as input and sends the extracted global feature F, Section III.C network architecture, pages 2345-2346) of the frame using the backbone part (backbone part makes up of N cascaded recursive reconstruction blocks (RRB). Each RRB comprises N cascaded residual feature aggregation modules RFA with attention module); and
reconstructing the frame based (Y-hat = X + G(F), Y-hat is the output, equation 9, Section III.C network architecture, pages 2345-2346) on the global features to generate (the global feature F is transformed through the reconstruction part, Section III.C network architecture, pages 2345-2346) an enhanced frame using the reconstruction part (effectively reduce compression artifacts, Abstract).
Regarding Claim 11, the claim is rejected on the grounds provided in Claim 1.
Regarding Claim 13, the claim is rejected on the grounds provided in Claim 3.
Regarding Claim 17, the claim is rejected on the grounds provided in Claim 7.
Regarding Claim 18, the claim is rejected on the grounds provided in Claim 8.
Regarding Claim 19, the claim is rejected on the grounds provided in Claim 9.
Regarding Claim 20, Huang (“One-for-All: An efficient variable convolution Neural Network for In-loop filter for VVC,” IEEE 2022) discloses a non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations (software, page 2348 left column, Section IV.A Experimental settings). The remainder of the claim is rejected on the grounds provided in Claim 1.
Claim(s) 2, 4, 5, 12, 14, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Huang (“One-for-All: An efficient variable convolution Neural Network for In-loop filter for VVC,” IEEE 2022) in view of but Fu (“StoHisNet: A hybrid multi-classification model with CNN and Transformer for gastric pathology images,” Computer Methods and Programs in Biomedicine 2021) and Li (“CONVOLUTIONAL NEURAL NETWORK BASED IN-LOOP FILTER FOR VVC INTRA CODING,” IEEE 2021).
Regarding Claim 2, Huang (“One-for-All: An efficient variable convolution Neural Network for In-loop filter for VVC,” IEEE 2022) discloses the method of claim 1, wherein the auxiliary information comprises … a quantization parameter map each associated with the frame (residual block with QP attention module and FT attention module receives the QP and FT, Fig. 5, Section III.B attention module, page 2345 left column).
Huang does not disclose but Li (“CONVOLUTIONAL NEURAL NETWORK BASED IN-LOOP FILTER FOR VVC INTRA CODING,” IEEE 2021) teaches wherein the auxiliary information comprises a prediction map, a partition map (filter takes auxiliary information including partitioning and prediction information as input, Abstract).
One of ordinary skill in the art before the application was filed would have been motivated to further include prediction map and partition map into the auxiliary information of Huang, and use luma samples to filter chroma, because Li teaches that partitioning and prediction are highly related to compression artifacts and reconstruction distortion, Section 2.1, and using luma to predict/filter chroma has showed considerable coding gains in other parts of encoding, Section 2.4, and using these features int the neural network improves the reconstruction quality.
Regarding Claim 4, Huang (“One-for-All: An efficient variable convolution Neural Network for In-loop filter for VVC,” IEEE 2022) discloses the method of claim 3, wherein the frame is a reconstruction frame of chroma samples (performance of CNN on chroma components, Section IV.E generalization ability of VCNN, pages 2351-2352); and
extracting the features from the frame comprises extracting the features from the reconstruction frame of chroma samples (features F’, refined output, Section III.B attention module, pages 2344-2345) based on the auxiliary information (refined based on the attention map M based on QP and FT, Section III.B attention module, pages 2344-2345).
Huang does not disclose but Li (“CONVOLUTIONAL NEURAL NETWORK BASED IN-LOOP FILTER FOR VVC INTRA CODING,” IEEE 2021) teaches extracting the features from the frame comprises extracting the features from the reconstruction frame of chroma samples (M feature maps, Fig. 1, Section 2, page 2015) based on a reconstruction frame of luma samples using the feature extraction part (feed the luma information into the network for filtering chroma components, Section 2.4 dealing with chroma, page 2106; For chroma, auxiliary information further includes luma samples, Abstract, Section 1, page 2105).
One of ordinary skill in the art before the application was filed would have been motivated to further include prediction map and partition map into the auxiliary information of Huang, and use luma samples to filter chroma, because Li teaches that partitioning and prediction are highly related to compression artifacts and reconstruction distortion, Section 2.1, and using luma to predict/filter chroma has showed considerable coding gains in other parts of encoding, Section 2.4, and using these features int the neural network improves the reconstruction quality.
Regarding Claim 5, Huang (“One-for-All: An efficient variable convolution Neural Network for In-loop filter for VVC,” IEEE 2022) discloses the method of claim 3, wherein the at least part of the auxiliary information comprises … a QP map each associated with the frame (residual block with QP attention module and FT attention module, Fig. 5, Section III.B attention module, page 2345 left column; receives the QP and the FT, Fig. 4); and
applying the NN-based in-loop filter (in-loop filters integrated with VCNN in VVC, Fig. 2) comprises processing the features based on the … QP map using the at least one attention block (residual block with QP attention module and FT attention module, Fig. 5, Section III.B attention module, page 2345 left column; receives the QP and the FT, Fig. 4).
Huang does not disclose but Li (“CONVOLUTIONAL NEURAL NETWORK BASED IN-LOOP FILTER FOR VVC INTRA CODING,” IEEE 2021) teaches wherein the at least part of the auxiliary information comprises a partition map (refined based on the attention map M based on QP and FT, Section III.B attention module, pages 2344-2345);
applying the NN-based in-loop filter (in-loop filters integrated with VCNN in VVC, Fig. 2) comprises processing the features based on the partition map (features F’, refined output, Section III.B attention module, pages 2344-2345; refined based on the attention map M based on QP and FT, Section III.B attention module, pages 2344-2345).
One of ordinary skill in the art before the application was filed would have been motivated to further include prediction map and partition map into the auxiliary information of Huang, and use luma samples to filter chroma, because Li teaches that partitioning and prediction are highly related to compression artifacts and reconstruction distortion, Section 2.1, and using luma to predict/filter chroma has showed considerable coding gains in other parts of encoding, Section 2.4, and using these features int the neural network improves the reconstruction quality.
Regarding Claim 12, the claim is rejected on the grounds provided in Claim 2.
Regarding Claim 14, the claim is rejected on the grounds provided in Claim 4.
Regarding Claim 15, the claim is rejected on the grounds provided in Claim 5.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Dosovitskiy (“An image is worth 16x16 words: Transformers for Image recognition at scale,” ICLR 2021)
Woo (“CBAM: Convolutional block attention module,” Springer 2018)
Wang (“Evolving attention with residual convolution,” 38th International conference on Machine Learning 2021)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHADAN E HAGHANI whose telephone number is (571)270-5631. The examiner can normally be reached M-F 9AM - 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jay Patel can be reached at 571-272-2988. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHADAN E HAGHANI/Examiner, Art Unit 2485
Prosecution Insights