DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Allowable Subject Matter
Claims 5, 10-11 and 14-16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
With regards to claim 5, several of the features of this claim were known in the art as evidenced by Ming Lu et al, “Transformer-based Image Compression” which anticipates the limitations of parent claim 1, In particular, Ming Lu discloses the image compression framework (“[T]he Neural Transformation Unit (NTUs) in both main and hyper encoder-decoders of a VAE framework”) comprises a preprocessing function (“Swin Transformer Block (STB)”) and a compressor (“convolutional layer”; e.g., “Conv(3,2)”) at pp. 5-6, sec. 3.2 and FIG. 2; see, also, pp. 4-5, sec. 3.1 and FIG. 1. Ming Lu further discloses the preprocessing function (“Swin Transformer Block (STB)”) includes “multi-layer perceptron (MLP)” at p. 2, sec. 1.2; pp. 5-6, sec. 3.2 and FIG. 3. A multi-layer perceptron (MLP) is a deep neural network that employs parameters determined through deep learning. Moreover, Ming Lu discloses convolutional layers that employ a kernel size of 3x3 (“k=3”) at p. 5 and FIG. 2. However, Ming Lu does not disclose the convolutional layers include dilated convolution with a dilated parameter set to three, five, or seven.
With regards to claim 10, several of the features of this claim were known in the art as evidenced by Ming Lu et al, “Transformer-based Image Compression” which anticipates the limitations of parent claim 1, In particular, Ming Lu discloses the compressor (“convolutional layer”; e.g., “Conv(3,2)”) comprises a learning based variational auto-encoder, and wherein the preprocessing function (“Swin Transformer Block (STB)”) and the variational auto-encoder are jointly trained from scratch at pp. 4-5, sec. 3.1 (“The optimization objective is to minimize the rate-distortion cost through an end-to-end learning means…”) and FIG. 2 (“The proposed TIC stacks the Neural Transformation Unit (NTUs) in both main and hyper encoder-decoders of a VAE framework.”); see, also: p. 2, sec. 1.2; pp. 5-6, sec. 3.2. However, Ming Lu does not disclose the learning based variational auto-encoder is partially trained prior to training the preprocessing function, and wherein the learning based variational auto-encoder after being partially trained and the preprocessing function are jointly trained
With regards to claim 11, several of the features of this claim were known in the art as evidenced by Ming Lu et al, “Transformer-based Image Compression” which anticipates the limitations of parent claim 1, In particular, Ming Lu discloses the compressor (“convolutional layer”; e.g., “Conv(3,2)”) comprises a learning based variational auto-encoder, and wherein the preprocessing function (“Swin Transformer Block (STB)”) and the variational auto-encoder are jointly trained from scratch at pp. 4-5, sec. 3.1 (“The optimization objective is to minimize the rate-distortion cost through an end-to-end learning means…”) and FIG. 2 (“The proposed TIC stacks the Neural Transformation Unit (NTUs) in both main and hyper encoder-decoders of a VAE framework.”); see, also: p. 2, sec. 1.2; pp. 5-6, sec. 3.2. However, Ming Lu does not disclose the compressor comprises a proxy auto encoder with fixed model parameters for assisting training of the preprocessing function, and wherein the proxy auto encoder includes a prediction function, a transformation function, an inverse transformation function, a quantization function, and a rate estimation function.
With regards to claims 14-15, these claims depend from claim 11 and therefore incorporate the features of that claim that were found allowable.
With regards to claim 16, several of the features of this claim were known in the art as evidenced by Ming Lu et al, “Transformer-based Image Compression” which anticipates the limitations of parent claim 1, In particular, Ming Lu discloses the compressor (“convolutional layer”; e.g., “Conv(3,2)”) comprises a learning based variational auto-encoder, and wherein the preprocessing function (“Swin Transformer Block (STB)”) and the variational auto-encoder are jointly trained from scratch at pp. 4-5, sec. 3.1 (“The optimization objective is to minimize the rate-distortion cost through an end-to-end learning means…”) and FIG. 2 (“The proposed TIC stacks the Neural Transformation Unit (NTUs) in both main and hyper encoder-decoders of a VAE framework.”); see, also: p. 2, sec. 1.2; pp. 5-6, sec. 3.2. However, Ming Lu does not disclose the compressor comprises a hybrid codec that further includes a codec and a learning-based function, and wherein the learning-based function includes at least one of the group consisting of a learning-based in-loop filter, learning-based intra prediction, learning-based inter prediction, and learning-based quantization.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 4, 6, 8-9, 12-13 and 17-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ming Lu et al, “Transformer-based Image Compression”.
With regards to claim 1, Ming Lu discloses performing a conversion between visual media data and a bitstream of the visual media data by an image compression framework (“[T]he Neural Transformation Unit (NTUs) in both main and hyper encoder-decoders of a VAE framework”) at pp. 5-6, sec. 3.2 and FIG. 2; see, also, pp. 4-5, sec. 3.1 and FIG. 1.
Ming Lu discloses the image compression framework (“[T]he Neural Transformation Unit (NTUs) in both main and hyper encoder-decoders of a VAE framework”) comprises a preprocessing function (“Swin Transformer Block (STB)”) and a compressor (“convolutional layer”; e.g., “Conv(3,2)”) at pp. 5-6, sec. 3.2 and FIG. 2; see, also, pp. 4-5, sec. 3.1 and FIG. 1.
Ming Lu discloses the visual media data is processed by the preprocessing function (“Swin Transformer Block (STB)”) and the compressor (“convolutional layer”; e.g., “Conv(3,2)”) sequentially at pp. 5-6, sec. 3.2 and FIGS. 2-3; see, also, pp. 4-5, sec. 3.1 and FIG. 1.
PNG
media_image1.png
296
574
media_image1.png
Greyscale
Ming Lu discloses the preprocessing function (“Swin Transformer Block (STB)”) receives a first image among the visual media data with a size of WOxHOxCO as input and outputs a preprocessed first image with a size of WlxHlxCl, wherein WO is an input width, HO is an input height, and CO is an input channel number, and wherein WI is an output width, HI is an output height, and Cl is an output channel number at pp. 5-6, sec. 3.2 and FIG. 2.
With regards to claim 2, Ming Lu discloses WO is equal to WI, HO is equal to HI, and CO is equal to Cl at pp. 5-6, sec. 3.2 and FIG. 2 (“[F]inally a feature unembedding (FU) layer remaps attention weighted features back to original size of H
×
W
×
C”).
With regards to claim 4, Ming Lu discloses the preprocessed first image output by the preprocessing function comprises features with different spatial dimensions at p. 8, sec. 4 and FIG. 6. As evidenced by FIG. 6, the features generated from a woman’s face have different spatial dimensions than the features generated from an image of a house.
With regards to claim 6, a residual block is a block that adds the input of a layer directly to its output using a shortcut or skip connection as evidenced by Wikipedia, “Residual neural network” at p. 1, “Residual connection.” Ming Lu discloses the preprocessing function (“Swin Transformer Blocks (STB)”) includes residual blocks, i.e., a block that adds the input of a layer directly to its output using a shortcut or skip connection at pp. 5-6, sec. 3.2 and FIG. 3 (“Skip connection is used for better aggregation”). Moreover, supporting this interpretation is Ze Liu et al, “Swin Transformer: Hierarchical Vision Transformer using ShiftedWindows”, which describes Swin Transformer blocks as, “a residual connection is applied after each module” at p. 4, sec. 3.2. Furthemore, Ming Lu inherently discloses the residual blocks comprise an activation layer when it discloses a “multilayer perceptron” at pp. 5-6, 3.2 and FIG. 3; multi-layer perceptrons necessarily comprise activation functions as evidenced by Wikipedia, “Multilayer Perceptron” at p. 1 (“In deep learning, a multilayer perceptron (MLP) is a kind of modern feedforward neural network consisting of fully connected neurons with nonlinear activation functions…”)
With regards to claim 8, Ming Lu discloses the preprocessing function (“Swin Transformer Block (STB)”) is jointly optimized with the compressor (“convolutional layer”; e.g., “Conv(3,2)”) at pp. 4-5, sec. 3.1 (“The optimization objective is to minimize the rate-distortion cost through an end-to-end learning means…”)
With regards to claim 9, Ming Lu discloses the compressor (“convolutional layer”; e.g., “Conv(3,2)”) comprises a learning based variational auto-encoder, and wherein the preprocessing function (“Swin Transformer Block (STB)”) and the variational auto-encoder are jointly trained from scratch at pp. 4-5, sec. 3.1 (“The optimization objective is to minimize the rate-distortion cost through an end-to-end learning means…”) and FIG. 2 (“The proposed TIC stacks the Neural Transformation Unit (NTUs) in both main and hyper encoder-decoders of a VAE framework.”); see, also: p. 2, sec. 1.2; pp. 5-6, sec. 3.2.
With regards to claim 12, Ming Lu discloses the preprocessing function (“Swin Transformer Block (STB)”) includes multiple models directed to different coding bit rates at p. 7 (“In STB, ... We adopt 128 channels for the first 4 models corresponding to low bit rates scenarios, and 192 channels for the rest 4 models to cover high bit rates.”)
With regards to claim 13, Ming Lu discloses the preprocessing function is configured to adapt to different coding bit rates based on a a coding controlling parameter (“λ”) at p. 4, sec. 3.1 (“We adapt λ for rate-distortion trade-off at various bit rates.”)
With regards to claim 17, Ming Lu discloses the conversion includes encoding the visual media data into the bitstream (e.g., “bits”) at pp. 4-5, sec. 3.1 (“The encoder transforms the input x into the latent features y, and then quantize y to discrete symbols
y
^
that are entropy coded into bitstreams using predefined distribution models; The decoding part mirrors the encoding steps by parsing the compressed bitstream to reconstruct pixel blocks to form decoded
x
^
.”) and FIG. 2
With regards to claim 18, Ming Lu discloses the conversion includes decoding the visual media data from the bitstream (e.g., “bits”) at pp. 4-5, sec. 3.1 (“The encoder transforms the input x into the latent features y, and then quantize y to discrete symbols
y
^
that are entropy coded into bitstreams using predefined distribution models; The decoding part mirrors the encoding steps by parsing the compressed bitstream to reconstruct pixel blocks to form decoded
x
^
.”) and FIG. 2
With regards to claim 19, the steps performed by the apparatus of this claim are anticipated by Ming Lu for the same reasons as were provided in the discussion of claim 1, which recites a method performing these same steps.
With regards to claim 20, the steps stored in the computer readable medium of this claim are anticipated by Ming Lu for the same reasons as were provided in the discussion of claim 1, which recites a method performing these same steps.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 3 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Ming Lu et al, “Transformer-based Image Compression” in view of Ze Liu et al, “Swin Transformer: Hierarchical Vision Transformer using ShiftedWindows”.
With regards to claim 3, Ming Lu discloses the preprocessing function (“Swin Transformer Block (STB)”) receives a first image among the visual media data with a size of WOxHOxCO as input and outputs a preprocessed first image with a size of WlxHlxCl, wherein WO is an input width, HO is an input height, and CO is an input channel number, and wherein WI is an output width, HI is an output height, and Cl is an output channel number at pp. 5-6, sec. 3.2 and FIG. 2. But, Ming Lu does not specify WO is equal to WI, HO is equal to HI, and CO is not equal to Cl. However, this limitation was known in the art:
Ze Liu discloses a Swin Transformer Block (STB) wherein WO is equal to WI, HO is equal to HI, and CO is not equal to Cl at pp. 3-4, sec. 3.1 and FIG. 3 (e.g., “2C”, “4C”, “8C”). At the time of the filing of the present application, it would have been obvious to a person of ordinary skill in the art to use a Swin Transformer Block (STB), as taught by Ze Liu, as the compressor for the method taught by Ming Lu because Ming Lu specifically suggests doing so at p. 2, sec. 1.2 and pp. 4-6, secs. 3.1-3.2. Therefore, it would have been obvious to combine Ze Liu with Ming Lu to obtain the invention specified in this claim.
With regards to claim 7, Ming Lu discloses the preprocessing function includes a swin-transformer at p. 2, sec. 1.2 and pp. 4-6, secs. 3.1-3.2. Ze Liu discloses a head number of the swin-transformer six at p. 10, Table 7, and wherein a depth of the swin-transformer is set to two at p. 4, FIG. 3. The motivation for the combination is the same as previously presented.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID F DUNPHY whose telephone number is (571)270-1230. The examiner can normally be reached 9 am - 5 pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID F DUNPHY/Primary Examiner, Art Unit 2673
Prosecution Insights