DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant claims the benefit of US Provisional Application No. 63/467,570, filed May 18, 2023. Claims 1-20 have been afforded the benefit of this filing date.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: “59” in FIG 5, “79” in FIG 7, “89” in FIG 8, “154” and “156” in FIG 15.
Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
Regarding para 5, there is no "upper left of Fig. 2"; this seems to reference the figure in the provisional application, not the non-provisional FIG 2.
In para 6 and 7, "MPL" should read "MLP".
In para 43, "in a scheme" should read "in a scene".
In para 47, "output of hamburger head 12, FH2" should read "output of hamburger head 12, FH1".
Appropriate correction is required.
Claim Objections
Claims 1, 5, 8, 10, and 18-19 are objected to because of the following informalities:
In claims 1 and 19, "(MPL)" should read "(MLP)".
In claims 5 and 18, “first multilayer perceptron” should read “third multilayer perceptron” (consistent with FIG 5, wherein the first and second MLPs of claim 1 are different from the MLP applied to the first intermediate vector in claim 5).
In claims 8 and 18, “second multilayer perceptron” should read “fourth multilayer perceptron” (consistent with FIG 5, wherein the first and second MLPs of claim 1 are different from the MLP applied to the second intermediate vector in claim 8).
In claims 10 and 18, “third multilayer perceptron” should read “fifth multilayer perceptron” (consistent with the changes to claims 5 and 8 above).
In claim 19, "store in the" should read "stored in the".
Appropriate correction is required.
Claim Interpretation
Regarding claim 15 and dependent claims 16-17, the two multilayer perceptrons of the “one hamburger head” are interpreted to be MLPs that utilize no activation or only linear activation to perform a linear transform on the input data (consistent with the hamburger head embodiment of FIG 12 and para 71-74). The MLPs cannot utilize nonlinear activation.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Guo et al. (Guo, M. H., Lu, C. Z., Hou, Q., Liu, Z., Cheng, M. M., & Hu, S. M. (2022). Segnext: Rethinking convolutional attention design for semantic segmentation. Advances in neural information processing systems, 35, 1140-1156.), hereinafter Guo, in view of Lin et al. (CN Patent No. 111340186 A), hereinafter Lin.
Regarding claim 1, Guo teaches a method of segmenting a foreground object from a scene for image editing or augmented reality, wherein the scene is represented in an input image (Guo, pg. 1, abstract: “SegNeXt, a simple convolutional network architecture for semantic segmentation”; trained and validated on image datasets with foreground/background classes, see 1st para on pg. 6), the method comprising: obtaining a plurality of feature vectors using a feature extractor (Guo, convolutional encoder, pg. 4, FIG 2 caption: “We extract multi-scale features”; pg. 4, section 3.1: “multi-scale convolutional attention (MSCA) module”), wherein the feature extractor comprises a plurality of multi-scale convolutional attention blocks (Guo, hierarchical structure to extract features at each resolution stage, pg. 4, section 3.1: “For MSCAN, we adopt a common hierarchical structure, which contains four stages with decreasing spatial resolutions”); and obtaining, based on the plurality of feature vectors and performing an operation using one or more hamburger heads (Guo, see the decoder in FIG 3c, pg. 5 below, emphasis added to underlined text), a prediction image (Guo, performs image segmentation, 2nd to last para on pg. 7: “decoder designs for segmentation”; see image outputs in FIG 4, pg. 8), wherein the prediction image segments the foreground object from the scene (Guo, see pg. 8, FIG 4 and foreground/background classes in 1st para on pg. 6).
PNG
media_image1.png
318
679
media_image1.png
Greyscale
Guo fails to explicitly teach wherein a first hamburger head of the one or more hamburger heads comprises, in sequence, a first multilayer perceptron (MLP), a matrix decomposition, and a second MLP (the hamburger cited in Guo teaches a matrix decomposition in between two linear transform operations, but MLPs are not explicitly disclosed). However, Lin teaches a hamburger head comprising, in sequence, a first multilayer perceptron (MLP), a matrix decomposition, and a second MLP (Lin, para 79: “The “hamburger” module used for illustrative purposes in this invention consists of two linear transformations and a matrix factorization model. As shown in Figure 1, two linear transformations, acting as the “lower bun” and “upper bun,” are placed before and after the “patty” of the tensor decomposition model or matrix decomposition model, respectively, forming a “hamburger module.””; regarding MLPs - see para 66 wherein the lower and upper breads are composed of micro neural network that are linear transforms). Thus, Lin discloses a hamburger head comprising two MLPs, or linear transforms (Lin, para 66), while Guo discloses a hamburger head with two linear transform operations. A person of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized that the hamburger head of Lin could have been substituted for the hamburger head of Guo because both perform a linear transform operation before and after a matrix decomposition. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to substitute the hamburger head of Lin for that of Guo according to known methods to yield the predictable result of implementing a light attention mechanism for a computer vision task (Lin, pg. 12, para 56: “our proposed method, as a highly interpretable representation learning approach, outperforms commonly used attention mechanisms in computer vision with a lighter computational and parameter count”).
Regarding claim 19, Guo teaches an apparatus comprising: one or more memories; and one or more processors, wherein the one or more processors are configured to execute instructions store in the one or more memories to perform the claimed operation steps (Guo, method is a computer task, pg. 9, Table 11 caption: “We test our method with a single RTX-3090 GPU and AMD EPYC 7543 32-core processor CPU”). All further claim limitations are met and rendered obvious by Guo in view of Lin because the method steps of claim 1 are the same as the apparatus steps of claim 19.
Regarding claim 20, Guo teaches a non-transitory computer readable medium storing instructions to be executed by one or more processors, wherein the instructions are configured to cause the one or more processors to perform the claimed operation steps (Guo, method is a computer task, pg. 9, Table 11 caption: “We test our method with a single RTX-3090 GPU and AMD EPYC 7543 32-core processor CPU”). All further claim limitations are met and rendered obvious by Guo in view of Lin because the method steps of claim 1 are the same as the executed steps of claim 20.
Claims 2-5 and 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Lin, in further view of Woo et al. (Woo, S., Park, J., Lee, J. Y., & Kweon, I. S. (2018). Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV) (pp. 3-19).), hereinafter Woo.
Regarding claim 2 (dependent on claim 1), Guo in view of Lin teaches wherein the feature extractor comprises a convolutional layer (Guo, pg. 4, section 3.1: “depth-wise convolution to aggregate local information”), but fails to teach a plurality of multi-scale convolutional attention modules. However, Woo teaches a convolutional model comprising a plurality multi-scale convolutional attention modules (Woo, pg. 1, abstract: “our module sequentially infers attention maps along two separate dimensions, channel and spatial”; see FIG 3 on pg. 7, attached below). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the use of a plurality of attention modules, taught by Woo, in each multi-scale convolutional block in the method of Guo in order to improve the performance of the convolutional model (Woo, pg. 14, section 5: “We apply attention based feature refinement with two distinctive modules, channel and spatial, and achieve considerable performance improvement while keeping the overhead small”).
PNG
media_image2.png
112
490
media_image2.png
Greyscale
Regarding claim 3 (dependent on claim 2), Guo in view of Lin and Woo teaches wherein the obtaining the plurality of feature vectors comprises obtaining the plurality of feature vectors corresponding to the plurality of multi-scale convolutional attention blocks (Guo, features are output at each resolution stage – 4, 8, 26, 32, pg. 4, section 3.1: “For MSCAN, we adopt a common hierarchical structure, which contains four stages with decreasing spatial resolutions”).
Regarding claim 4 (dependent on claim 3), Guo in view of Lin and Woo teaches wherein the prediction image is a final prediction image (Guo, see image outputs in FIG 4, pg. 8 with segmentation results).
Regarding claim 5 (dependent on claim 4), Guo in view of Lin and Woo teaches wherein the plurality of feature vectors comprises a first feature vector, a second feature vector, a third feature vector and a fourth feature vector (Guo, encoder output, pg. 4, last para: “four stages with decreasing spatial resolutions”; see also 4 stages in FIG 3c), the method further comprising: applying the first hamburger head to the third feature vector and the fourth feature vector to obtain a first intermediate vector (Guo, output of head in FIG 3c which is then input to MLP; pg. 5, 1st para in section 3.2: “We aggregate features from the last three stages and use a lightweight Hamburger [22] to further model the global context”, this includes a third and fourth feature vector of the four stages); applying a third multilayer perceptron to the first intermediate vector to obtain a first image (Guo, see MLP sequentially after head in FIG 3c; see image outputs in FIG 4, pg. 8 - first image is the prediction image, in the same way as claim 14 wherein there is one hamburger head).
Regarding claim 14 (dependent on claim 5), Guo in view of Lin and Woo teaches wherein the one or more hamburger heads is only one hamburger head (Guo, see one hamburger head in pg. 5, 1st para of section 3.2 and FIG. 3c) and the first image is the prediction image (Guo, see image outputs in FIG 4, pg. 8).
Regarding claim 15 (dependent on claim 14), Guo in view of Lin and Woo teaches wherein the performing the operation comprises in sequence a first linear transforming, a matrix decomposing, and a second linear transforming (Lin, pg. 5, para 11: “lower-level linear transformation Wl, a matrix decomposition model M, and an upper-level linear transformation Wu connected sequentially”).
Regarding claim 16 (dependent on claim 15), Guo in view of Lin and Woo teaches wherein the matrix decomposing comprises decomposition into a product and a summand (Lin, pg. 21, para 79-80: “transformed representation X is decomposed into the sum of the product of a dictionary matrix D and a reconstruction coefficient matrix C and the residual matrix E. X = DC + E (Equation 3)”).
Regarding claim 17 (dependent on claim 16), Guo in view of Lin and Woo teaches wherein the matrix decomposing further comprises discarding the summand, so that a noise is reduced in the final prediction image (Lin, pg. 21, para 81: “In Equation 3, the residual E is discarded as invalid information”; pg. 16, para 67: “tensor decomposition and matrix decomposition models can be used to recover observations without noise and missing values”).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Guo in view of Lin, Woo, and Fu et al. (CN Patent No. 114419449 A), hereinafter Fu.
Regarding claim 6 (dependent on claim 5), Guo in view of Lin and Woo fails to teach wherein the first image is an initial prediction image; however, Fu teaches a similar segmentation method (Fu, pg. 2, para 1: “semantic segmentation of remote sensing images using self-attention multi-scale feature fusion”) wherein a first image is an initial prediction image (Fu, the prediction results from one scale, pg. 6, para 8: “The prediction results from the four scales are then fused to obtain the final remote sensing image semantic segmentation result”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the initial and final prediction images, taught by Fu, with the method of Guo in order to improve segmentation results by combining the segmentation predictions from different feature scales (Fu, pg. 6, para 8: “This method effectively fuses remote sensing semantic features of different scales, improving segmentation performance”).
Allowable Subject Matter
Claims 7-13 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. The following is a statement of reasons for the indication of allowable subject matter: regarding claim 7, the closest prior art, cited herein, fails to teach “applying a second hamburger head to the second feature vector and the first intermediate vector to obtain a second intermediate vector”. Analogous prior art teaching similar multi-scale decoder models, such as Fu cited above, also fails to teach the limitations of claim 7.
In view of the foregoing, the prior art references alone or in reasonable combination are insufficient to teach the invention as a whole, as claimed in claim 7. Due to their dependence on claim 7, claims 8-13 and 18 are similarly objected to.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Geng et al. (Geng, Z., Guo, M. H., Chen, H., Li, X., Wei, K., & Lin, Z. (2021). Is attention better than matrix decomposition?. arXiv preprint arXiv:2109.04553.) discloses a hamburger head.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EMMA E DRYDEN whose telephone number is (571)272-1179. The examiner can normally be reached M-F 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANDREW BEE can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMMA E DRYDEN/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677