Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 6-10, 14 and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (U.S. 2020/0327702), hereinafter Wang in view of LIN, J., et al., "Convolutional Neural Network-Based Block Up-Sampling for HEVC", IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 29, NO. 12, DECEMBER 2019. Lin was cited on the Applicant’s IDS dated 3/18/24 with a copy provided on the same date.
Regarding claims 1 and 19, Wang discloses an apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor (Wang [0018]), cause the processor to:
apply a super resolution (SR) process to a video unit at a level of an SR unit (Wang figs. 2 and 4), wherein the SR unit level includes more than one pixel of the video unit (Wang [0027] and [0045]); and
perform a conversion between a video comprising the video unit and a bitstream of the video based on the SR process as applied (Wang fig. 1),
wherein the SR process comprises a convolutional neural network (CNN) SR process (Wang [0022] and [0001]), and
wherein the CNN SR process is trained on either frame-level data or coding tree unit (CTU)- level data (Wang [0001]), and the CNN SR process is used to up-sample (Wang [0022]-[0023]) an input at a frame-level or at a CTU-level (Wang [0010], [0055] and [0022]-[0023]).
Wang does not explicitly disclose wherein the SR unit changes from one level to another level within a sequence of frames or pictures depending on content of the video data.
However, Lin teaches wherein the SR unit changes from one level to another level within a sequence of frames or pictures depending on content of the video data (Lin p. 3703, section III, first paragraph, p. 3710, right column, last paragraph and p. 3713, section VIII, first paragraph),
wherein the SR process comprises a convolutional neural network (CNN) SR process (Lin p. 3703, section III, first paragraph), and
wherein the CNN SR process is trained on either frame-level data or coding tree unit (CTU)- level data (Lin p. 3706, section V, first paragraph), and the CNN SR process is used to up-sample an input at a frame-level or at a CTU-level (Lin p. 3703, section III, first two paragraphs).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wang’s method with the missing limitations as taught by Lin for efficient compression of natural video (Lin p. 3703, section III, first paragraph).
Regarding claim 6, Wang in view of Lin teaches the method of claim 1, wherein the SR process uses a neural network (NN) with the SR unit, or a region of the video unit that contains the SR unit as well as other pixels of the video unit as one of its inputs (Wang [0022] and [0001]).
Regarding claim 7, Wang in view of Lin teaches the method of claim 1, wherein the SR unit used for the SR process is included in the bitstream or pre-defined prior to the SR process being applied to the video unit (Wang figs. 1, 2 and 4).
Regarding claim 8, Wang in view of Lin teaches the method of claim 1, wherein the SR process applied to a first SR unit is different from the SR process applied to a second SR unit (Wang figs. 2 and 4).
Regarding claim 9, Wang in view of Lin teaches the method of claim 8, wherein the SR process applied to the first SR unit comprises one of a neural network (NN)-based SR process and a non-NN-based SR process, and wherein the SR process applied to the second SR unit comprises the other one of the NN based SR process and the non-NN-based SR process (Wang fig. 2 and [0023]).
Regarding claim 10, Wang in view of Lin teaches the method of claim 1, wherein an input of the SR process is at least one of a plurality of video unit levels, including a sequence of pictures level, a picture level, a slice level, a tile level, a brick level, a subpicture level, one or more CTUs level, a CTU row level, one or more coding units (CUs) level, one or more coding tree blocks (CTBs) level, or a region level, wherein the input of the SR process at the region level covers more than one pixel (Wang [0009] and fig. 4 and Lin p. 3703, section III, first two paragraphs).
The same motivation for claim 1 applies to claim 10.
Regarding claim 14, Wang in view of Lin teaches the method of claim 1, wherein at least one of a down-sampling ratio of the video unit, encoded information of the video unit, and decoded information of the video unit is used as an input of the SR process, and wherein the encoded information, the decoded information, or both comprise one or more of a prediction signal, a partition structure, and an intra prediction mode of the video unit (Wang figs. 1, 2 and 4).
Regarding claim 18, Wang in view of Lin teaches the method of claim 1, wherein the conversion includes decoding the video data from the bitstream (Wang fig. 1).
Regarding claim 20, Wang in view of Lin teaches a non-transitory computer-readable storage medium storing instructions that cause a processor to (Wang [0018] and [0084]): apply a super resolution (SR) process to a video unit at a level of an SR unit, wherein the SR unit level includes more than one pixel of the video unit; and perform a conversion between a video comprising the video unit and a bitstream of the video based on the SR process as applied, wherein the SR unit changes from one level to another level within a sequence of frames or pictures depending on content of the video data, wherein the SR process comprises a convolutional neural network (CNN) SR process, and wherein the CNN SR process is trained on either frame-level data or coding tree unit (CTU)-level data, and the CNN SR process is used to up-sample an input at a frame-level or at a CTU-level (see claim 1).
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Lin as applied to claim 1 above, and further in view of Da Silva Pratas Gabriel et al. (U.S. 2021/0211643), hereinafter Da Silva.
Regarding claim 2, Wang in view of Lin teaches the method of claim 1. Wang does not explicitly disclose wherein the SR unit used for the SR process and a video unit used for down-sampling are at the same level.
However, Da Silva teaches, wherein the SR unit used for the SR process and a video unit used for down-sampling are at the same level (Da Silva [0084] and [0142]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by Wang in view of Lin with the missing limitations as taught by Da Silva to efficiently code high-resolution video frames (Da Silva [0065]).
Claim(s) 3-4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Lin as applied to claim 1 above, and further in view of Machii et al. (U.S. 2021/0327028), hereinafter Machii.
Regarding claim 3, Wang in view of Lin teaches the method of claim 1. Wang does not explicitly disclose wherein the SR unit used for the SR process and a video unit used for down-sampling are at different levels.
However, Machii teaches, wherein the SR unit used for the SR process and a video unit used for down-sampling are at different levels (Machii [0074]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by Wang in view of Lin with the missing limitations as taught by Machii to be able to generate a super resolution imaged having a predetermined resolution based on image features (Machii [0074]).
Regarding claim 4, Wang in view of Lin and Machii teaches the method of claim 3, wherein the SR unit used for the SR process comprises a block or a CTU (Wang fig. 2 and Lin p. 3703, section III, first two paragraphs), and wherein the method further comprises performing down-sampling at a picture level, a slice level, or a tile level (Machii [0025]).
The same motivation for claims 1 and 3 applies to claim 4.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Lin and Machii as applied to claim 3 above, and further in view of Grois et al. (U.S. 2022/0103832), hereinafter Grois.
Regarding claim 5, Wang in view of Lin and Machii teaches the method of claim 3, wherein the SR unit used for the SR process comprises a CTU row, multiple CTUs, or multiple coding tree blocks (CTBs) (Wang [0052] and fig. 4 and Lin p. 3703, section III, first two paragraphs).
The same motivation for claim 1 applies to claim 5.
Wang does not explicitly disclose wherein the method further comprises performing down-sampling at a CTU level or a CTB level.
However, Grois teaches a method, wherein the SR unit used for the SR process comprises a CTU row, multiple CTUs, or multiple coding tree blocks (CTBs), and wherein the method further comprises performing down-sampling at a CTU level or a CTB level (Grois [0079] and Abstract).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by Wang in view of Lin and Machii with the missing limitations as taught by Grois to minimize cost of video coding without reducing visual presentation quality (Grois [0079] and Abstract).
Claim(s) 11 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view in view of Lin as applied to claim 10 above, and further of Joshi et al. (U.S. 2019/0394482), hereinafter Joshi.
Regarding claim 11, Wang in view of Lin teaches the method of claim 10. Wang does not explicitly disclose wherein the input of the SR process is a coding tree block (CTB) that has been down-sampled or a frame that has been down-sampled.
However, Joshi teaches, wherein the input of the SR process is a coding tree block (CTB) that has been down-sampled or a frame that has been down-sampled (Joshi [0049]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by Wang in view of Lin with the missing limitations as taught by Joshi to improve visual quality at low-bitrate settings (Joshi [0020]).
Regarding claim 17, Wang in view of Lin and Joshi teaches the method of claim 1, wherein the conversion includes encoding the video data into the bitstream (Joshi [0061]).
The same motivation for claim 11 applies to claim 17.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Lin as applied to claim 14 above, and further in view of Meardi et al. (U.S. 2024/0040160), hereinafter Meardi.
Regarding claim 15, Wang in view of Lin teaches the method of claim 14. Wang does not explicitly disclose wherein a stride of a convolutional layer of the CNN SR process is dependent on the down-sampling ratio of an input of the CNN SR process.
However, Meardi teaches, wherein a stride of a convolutional layer of the CNN SR process is dependent on the down-sampling ratio of an input of the CNN SR process (Meardi [0150]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by Wang in view of Lin with the missing limitations as taught by Meardi to achieve downsampling with an efficient light-weight neural network architecture (Meardi [0150]).
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Lin as applied to claim 14 above, and further in view of Yea et al. (U.S. 2022/0201307), hereinafter Yea.
Regarding claim 16, Wang in view of Lin teaches the method of claim 14. Wang does not explicitly disclose wherein a horizontal down-sampling ratio and a vertical down-sampling ratio are the same or different.
However, Yea teaches, wherein a horizontal down-sampling ratio and a vertical down-sampling ratio are the same or different (Yea [0150]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by Wang in view of Lin with the missing limitations as taught by Yea to achieve super resolution by applying down sampling in one direction (Yea [0150]).
Response to Arguments
Applicant's arguments filed in regard to the newly amended claims have been fully considered but are moot because the arguments do not apply to the current grounds of rejection being used in the current rejection, i.e. Wang in view of Lin.
The Examiner believes that the amended limitations can be taught by the previously cited art of record, however, in an effort to advance prosecution, the Examiner has added Lin, which was found after further search and consideration of the amended claims.
The Examiner notes that under the broadest reasonable interpretation of the current claim language of claim 1 and the other independent claims, each independent claim only requires either frame-level or CTU-level data, not both. Wang discloses convolutional neural network (CNN) training using training video (i.e. frame-level data) (Wang [0001]) and upsampling a frame of input data (i.e. an input at a frame-level) (Wang [0010], [0055], figs. 4 and 5).
Finally, see additional prior art found below for CTU-level coding after further search and consideration of the amended claim language.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Ratner et al. (U.S. 2017/0337711), hereinafter Ratner discloses training a neural network using CTU-level data (Ratner [0518]). Yin et al. (U.S. 2022/0237741) discloses super-resolution processing at a CTU-level (Yin [0099]).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW KWAN whose telephone number is (571)270-7073. The examiner can normally be reached Monday-Friday 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chris Kelley can be reached at (571)272-7331. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW K KWAN/Primary Examiner, Art Unit 2482
Prosecution Insights