DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on July is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is considered by examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 10-12, 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Gunduc (Tensor-to-Image: Image-to-Image Translation with Vision Transformers, cited in IDS 07/19/2024) in view of Li et al (Hierarchical Opacity Propagation for Image Matting).
Regarding Claim 1, Gunduc teach a system (Tensor-to-Image Architecture; Fig 1 and 3.1 Patches and Patch Encoding) comprising:
a transformer encoder (transformer encoder; Fig 1 and 3.2 Method – Transformer Layers) that generates patch-based encodings from a digital image and a trimap segmentation of the digital image (the transformer encoder intakes and encodes image patches; Fig 1 and 3.1 Method – Patches and Patch Encoding); and
a plurality of neural network layers (transpose convolution, residual block, LeakyReLU, batch normalization layers; Fig 1 and 3.4 Upsampling, 3.5 Architecture) connected to the transformer encoder that generate modified patch-based encodings from the patch-based encodings (the Tensor-to-Image Architecture includes a plurality of neural network layers (Transpose conv Relu, Residual Block) connected to the Transformer Encoder and used to generate the modified patch encodings; Fig 1 and 3.4 Upsampling, 3.5 Architecture).
Gunduc does not teach explicitly teach one or more memory devices comprising: a trimap segmentation of the digital image; a decoder comprising a plurality of upsampling layers that generate an alpha matte for the digital image from the patch-based encodings and the modified patch-based encodings; and one or more processors coupled to the one or more memory devices that cause the system to perform operations comprising generating the alpha matte from the digital image utilizing the transformer encoder, the plurality of neural network layers, and the decoder.
Li et al is analogous art pertinent to the technological problem addressed in the current application and teaches one or more memory devices (11G memory of NVIDIA RTX 2080 Ti to execute HOP algorithm; Table 4) comprising: trimap segmentation of the digital image (the input includes a trimap (segmentation of background, object of interest and outline region of object) of the image; Fig 2 and 2. Related Work ¶ 4, 3.2 Scale-insensitive Position Encoding ¶ 2); and
a decoder (decoder; Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 3-5) comprising a plurality of upsampling layers (a plurality of upsampling layers (5-level decoder and HOP Global and HOP Local layers); Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 3-5) that generate an alpha matte for the digital image from the patch-based encodings and the modified patch-based encodings (an alpha matte is output from the decoder based on the image data; Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 7; Gunduc teach image data is patch-based, encoded, and further processed to generate modified patch based encoded; Fig 1 and 3.4 Upsampling, 3.5 Architecture); and
one or more processors (NVIDIA RTX 2080 Ti to execute HOP algorithm; Table 4) coupled to the one or more memory devices that cause the system to perform operations (NVIDIA RTX 2080 Ti includes the processor and memory to execute HOP algorithm; Table 4) comprising
generating the alpha matte from the digital image utilizing the transformer encoder, the plurality of neural network layers, and the decoder (an alpha matte is generated from the multilayer encoder and decoder; Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 3-5; noted Gunduc teach image data is patch-based, downsampled with a transformer encoder to generate modified patch based encoded data and upsampled with transpose convolutions; Fig 1 and 3.4 Upsampling, 3.5 Architecture).
It would have been obvious to one of ordinary skill in the art to combine the teachings of Gunduc with Li et al including one or more memory devices comprising: a trimap segmentation of the digital image; a decoder comprising a plurality of upsampling layers that generate an alpha matte for the digital image from the patch-based encodings and the modified patch-based encodings; and one or more processors coupled to the one or more memory devices that cause the system to perform operations comprising generating the alpha matte from the digital image utilizing the transformer encoder, the plurality of neural network layers, and the decoder. With using the parallel encoders of the image and trimap to generate the alpha matte, fine details may be captured for an object of interest to generate a matte that estimates the transition between foreground and background at pixel-resolution, while performing such computation efficiently, as recognized by Li et al (1. Introduction ¶ 4-5).
Regarding Claim 2, Gunduc in view of Li et al teach the system of claim 1 (as described above), wherein: the transformer encoder comprises a plurality of transformer neural network layers that generate a plurality of patch-based encodings from the digital image and the trimap segmentation at a plurality of resolutions (Gunduc, the transformer encoder includes a number of layers to produce lower-dimensional (resolution) linear embeddings from the patch data; Fig 1 and 2. Related work ¶ 3, 3.2 Transformer Layers; also noted Li et al teach use of a trimap which is encoded, as described above); and the plurality of neural network layers connected to the transformer encoder comprise a plurality of multilayer perceptrons that generate a plurality of modified patch-based encodings at the plurality of resolutions (Gunduc, the plurality of NN layers connected to the transformer encoder include multilayer perceptions (MLP, a feed-forward network known in the art to have an input layer, hidden layers and output layer), see Fig 1 that shows the MLP following the MHA) that modify the patch-based encodings; Fig 1 and 3.2 Transformer Layers, 3.3 MHA).
Regarding Claim 3, Gunduc in view of Li et al teach the system of claim 2 (as described above), wherein the plurality of multilayer perceptrons provide the plurality of modified patch-based encodings at the plurality of resolutions to the plurality of upsampling layers via a set of skip connections (Gunduc, skip connections are used with the encoded patches in a U-Net-based architecture to concatenate the encoder to the decoder to preserve, directly transfer and upsample (see Fig 1 that show the skip connection in the 3x3 convolutional upsampling), described in generator B network architecture; Fig 1 and 3.4 Upsampling, 4.1 Generator Architectures ¶ 3).
Regarding Claim 4, Gunduc in view of Li et al teach the system of claim 1 (as described above), including wherein the one or more memory devices further comprise an additional encoder in parallel with the transformer encoder that extracts feature sets from the digital image and the trimap segmentation (Li et al, a hierarchical opacity propagation (HOP) matting method is used for analyzing the image and trimap image in generate of the alpha matte that includes the network structured with multiple encoders (opacity encoder and appearance encoder) extracting the image data in parallel; Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 3; Gunduc teach image data is downsampled with a transformer encoder, as described above), wherein the plurality of upsampling layers of the decoder generates the alpha matte further based on the feature sets extracted by the additional encoder (Li et al, an alpha matte is generated from the decoder processing the Opacity Encoder and the Appearance Encoder image data; Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 3).
Regarding Claim 5, Gunduc in view of Li et al teach the system of claim 4 (as described above), wherein: the plurality of neural network layers are connected to a first subset of upsampling layers of the plurality of upsampling layers (Li et al, the Appearance Encoder layers are connected to a plurality of upsampling layers (HOP Global and HOP Local layers); Fig 2 and 3.1 Hierarchical Opacity Propagation Block 3-5); and the additional encoder comprises a plurality of convolutional neural network layers connected to a second subset of upsampling layers of the plurality of upsampling layers (Li et al, the Opacity Encoder layers are connected to a plurality of upsampling layers (Deconvolution and ResBlock layers); Fig 2 and 3.1 Hierarchical Opacity Propagation Block 3-5), the first subset of upsampling layers at least partially overlapping the second subset of upsampling layers (Li et al, the Appearance Encoder and Opacity Encoder layers are overlapping with the blocks of different semantic levels/; Fig 2 and 3.1 HOP Block ¶ 4-7).
Regarding Claim 6, Gunduc in view of Li et al teach the system of claim 4 (as described above), wherein an upsampling layer of the plurality of upsampling layers generates an upsampled feature map from the patch-based encodings based on global context information in the modified patch-based encodings and local context information in the feature sets extracted by the additional encoder (Li et al, the feature maps of the image (appearance encoder) and trimap (opacity encoder) are propagated in parallel with scale-insensitive position encodings for global HOP block and local relative position encodings for local HOP block to the decoder that propagates information together; Fig 2-4 and 3.1 HOP Block ¶ 4-7, 3.2 Positional Encoding; Gunduc teach image data is patch-based, downsampled with a transformer encoder to generate modified patch based encoded data and upsampled with transpose convolutions; Fig 1 and 3.4 Upsampling, 3.5 Architecture).
Regarding Claim 7, Gunduc in view of Li et al teach the system of claim 4 (as described above), wherein generating the alpha matte further comprises generating the alpha matte utilizing the additional encoder (Li et al, the HOP matting output (including concatenated output of the opacity encoder and appearance encoder) is an alpha matte estimation; Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 7).
Regarding Claim 8, Gunduc teach instructions for operations (execution of Tensor-to-Image Architecture; Fig 1 and 3.1 Patches and Patch Encoding) comprising: generating, utilizing a transformer encoder comprising a plurality of transformer neural network layers (transformer encoder with multiple NN layers; Fig 1 and 3.2 Method – Transformer Layers), patch-based encodings from a digital image and a trimap segmentation of the digital image (the transformer encoder intakes and encodes image patches; Fig 1 and 3.1 Method – Patches and Patch Encoding).
Gunduc does not explicitly teach a non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising: trimap segmentation of the digital image; and generating, utilizing a decoder comprising a plurality of upsampling layers, an alpha matte from the digital image from the patch-based encodings by: providing, to the plurality of upsampling layers, global context information from the patch-based encodings via a set of skip connections with a set of neural network layers; and providing, to the plurality of upsampling layers, local context information in feature sets extracted from the digital image and the trimap segmentation utilizing an additional encoder in parallel with the transformer encoder.
Li et al is analogous art pertinent to the technological problem addressed in the current application and teaches a non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations (11G memory of NVIDIA RTX 2080 Ti to execute HOP algorithm; Table 4) comprising: trimap segmentation of the digital image (the input includes a trimap (segmentation) of the image; Fig 2 and 2. Related Work ¶ 4, 3.2 Scale-insensitive Position Encoding ¶ 2); and
generating, utilizing a decoder (decoder; Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 3-5) comprising a plurality of upsampling layers (a plurality of upsampling layers (5-level decoder and HOP Global and HOP Local layers); Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 3-5), an alpha matte from the digital image from the patch-based encodings (an alpha matte is output from the decoder based on the image data; Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 7; Gunduc teach image data is patch-based, encoded, and further processed to generate modified patch based encoded; Fig 1 and 3.4 Upsampling, 3.5 Architecture) by:
providing, to the plurality of upsampling layers, global context information from the patch-based encodings via a set of skip connections with a set of neural network layers (scale-insensitive position encodings for global HOP block encodings with skip connections (self-attention) in the neural network to the decoder; Fig 2-4 and 3.1 HOP Block ¶ 4; Gunduc teach image data is patch-based, transformer encoder encoded, and further processed to generate modified patch based encoded data; Fig 1 and 3.4 Upsampling, 3.5 Architecture); and
providing, to the plurality of upsampling layers, local context information in feature sets extracted from the digital image and the trimap segmentation utilizing an additional encoder in parallel with the transformer encoder (the feature maps of the image (appearance encoder) and trimap (opacity encoder) are propagated in parallel with scale-insensitive position encodings for local relative position encodings for local HOP block; Fig 2-4 and 3.1 HOP Block ¶ 4-7, 3.2 Positional Encoding; Gunduc teach image data is patch-based, and image data is downsampled with a transformer encoder, and further processed to generate modified patch based encoded data; Fig 1 and 3.4 Upsampling, 3.5 Architecture).
It would have been obvious to one of ordinary skill in the art to combine the teachings of Gunduc with Li et al including a non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising: trimap segmentation of the digital image; and generating, utilizing a decoder comprising a plurality of upsampling layers, an alpha matte from the digital image from the patch-based encodings by: providing, to the plurality of upsampling layers, global context information from the patch-based encodings via a set of skip connections with a set of neural network layers; and providing, to the plurality of upsampling layers, local context information in feature sets extracted from the digital image and the trimap segmentation utilizing an additional encoder in parallel with the transformer encoder. With using the parallel encoders of the image and trimap to generate the alpha matte, fine details may be captured for an object of interest to generate a matte that estimates the transition between foreground and background at pixel-resolution, while performing such computation efficiently through use of the skip connections that capture both local and global context, as recognized by Li et al (1. Introduction ¶ 4-5).
Regarding Claim 10, Gunduc in view of Li et al teach the non-transitory computer readable medium of claim 8 (as described above), wherein providing the local context information to the plurality of upsampling layers (Li et al, the Appearance Encoder layers are connected to a plurality of HOP Local upsampling layers; Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 4-7) comprises: extracting the feature sets including the local context information utilizing a plurality of convolutional neural network layers of the additional encoder (Li et al, the Opacity Encoder (additional encoder) layers are connected to a plurality of upsampling layers (Deconvolution and ResBlock layers) and the local context information is obtained; Fig 2-4 and 3.1 Hierarchical Opacity Propagation Block ¶ 4-7); and providing the feature sets from the plurality of convolutional neural network layers to the plurality of upsampling layers via an additional set of skip connections (Li et al, encoder data are propagated in parallel with scale-insensitive position encodings for global HOP block and local relative position encodings for local HOP block using self-attention via skip connections; Fig 2-4 and 3.1 HOP Block ¶ 4-7).
Regarding Claim 11, Gunduc in view of Li et al teach the non-transitory computer readable medium of claim 10 (as described above), wherein providing the local context information to the plurality of upsampling layers comprises: encoding, utilizing a first convolutional neural network layer of the additional encoder, first local features from image patches of the digital image based on the digital image and the trimap segmentation (Li et al, the opacity encoder has a first neural network layer for first local (patch) features based on the image and trimap image; Fig 2-4 and 3.1 HOP Block ¶ 4-7, 3.2 Positional Encoding; Gunduc explicitly states image data is patch-based; Fig 1 and 3.4 Upsampling, 3.5 Architecture); downsampling the first local features encoded from the image patches to a downsampled resolution utilizing the first convolutional neural network layer (Li et al, the local (patch) features are downsampled when encoded (reducing resolution); Fig 2-4 and 3.1 HOP Block ¶ 4-7, 3.2 Positional Encoding; Gunduc explicitly states image data is patch-based; Fig 1 and 3.4 Upsampling, 3.5 Architecture); and encoding, utilizing a second convolutional neural network layer of the additional encoder, second local features from image patches of the digital image at the downsampled resolution based on the first local features (Li et al, the downsampled local (patch) features passed through the first layer of the opacity encoder pass through a second layer for further local feature downsampling ; Fig 2-4 and 3.1 HOP Block ¶ 4-7, 3.2 Positional Encoding; Gunduc explicitly states image data is patch-based; Fig 1 and 3.4 Upsampling, 3.5 Architecture).
Regarding Claim 12, Gunduc teach a computer-implemented method (method of using Tensor-to-Image Architecture; Fig 1 and 3.1 Method – Patches and Patch Encoding, 3.2 Method – Transformer Layers) comprising: steps claimed in parallel to claim 8 (as described above).
Regarding Claim 16, Gunduc in view of Li et al teach the computer-implemented method of claim 12 (as described above), wherein incorporating the local context information at the plurality of upsampling layers (Li et al, the Appearance Encoder layers are connected to a plurality of HOP Local upsampling layers; Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 4-7) comprises: extracting, utilizing the second set of neural network layers, feature sets comprising local features of patches from the digital image and the trimap segmentation at a plurality of resolutions (Li et al, the Opacity Encoder (additional encoder) layers are connected to a plurality of upsampling layers (Deconvolution and ResBlock layers) and the local context information is obtained; Fig 2-4 and 3.1 Hierarchical Opacity Propagation Block ¶ 4-7); and providing the feature sets to a subset of upsampling layers of the plurality of upsampling layers of the decoder (Li et al, encoder data are propagated in parallel with scale-insensitive position encodings for global HOP block and local relative position encodings for local HOP block using self-attention via skip connections; Fig 2-4 and 3.1 HOP Block ¶ 4-7).
Regarding Claim 17, Gunduc in view of Li et al teach the computer-implemented method of claim 12 (as described above), wherein generating the patch-based encodings (Gunduc, the transformer encoder intakes and encodes image patches; Fig 1 and 3.1 Method – Patches and Patch Encoding) comprises: generating a first patch-based encoding at a first resolution utilizing a first transformer neural network layer of the plurality of transformer neural network layers (Gunduc, the transformer encoder includes a number of layers to produce lower-dimensional (resolution) linear embeddings from the patch data; Fig 1 and 2. Related work ¶ 3, 3.2 Transformer Layers; also noted Li et al teach use of a trimap which is encoded, as described above); and generating a second patch-based encoding at a second resolution lower than the first resolution utilizing a second transformer neural network layer of the plurality of transformer neural network layers (Gunduc, the plurality of NN layers connected to the transformer encoder include multilayer perceptions (MLP, see Fig 1 that shows the multiple MLPs in the Transpose, Relu Batch Norm layering) that modify the patch-based encodings; Fig 1 and 3.4 Upsampling, 3.5 Architecture).
Regarding Claim 18, Gunduc in view of Li et al teach the computer-implemented method of claim 17 (as described above), wherein generating the alpha matte (Li et al, the HOP matting output is an alpha matte estimation based on decoding the encoded data; Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 7) comprises: generating, utilizing a first multilayer perceptron of the first set of neural network layers, a first modified patch-based encoding from the first patch-based encoding at the first resolution (Gunduc, the plurality of NN layers connected to the transformer encoder include multilayer perceptions (MLP (feed-forward network known in the art to have an input layer, hidden layers and output layer), see Fig 1 that shows the MLP following the MHA) that modify the patch-based encodings and reduce resolution as downsampled through the first layer; Fig 1 and 3.2 Transformer Layers, 3.3 MHA); and generating, utilizing a second multilayer perceptron of the first set of neural network layers, a second modified patch-based encoding from the second patch-based encoding at the second resolution (Gunduc, the patch-based encodings pass through a second MHA and MLP layer and are further downsampled at a subsequent layer; Fig 1 and 3.2 Transformer Layers, 3.3 MHA).
Regarding Claim 19, Gunduc in view of Li et al teach the computer-implemented method of claim 18 (as described above), wherein generating the alpha matte (Li et al, the HOP matting output is an alpha matte estimation based on decoding the encoded data; Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 7) comprises generating, utilizing an upsampling layer of the plurality of upsampling layers, an upsampled feature map based on a modified patch-based encoding from the first multilayer perceptron, an upsampled feature map from a previous upsampling layer of the plurality of upsampling layers, and a feature set extracted by a convolutional neural network layer of the second set of neural network layers (Li et al, the feature maps of the image (appearance encoder) and trimap (opacity encoder) are propagated in parallel in the decoder with decoding upsampling layers together; Fig 2-4 and 3.1 HOP Block ¶ 4-7, 3.2 Positional Encoding; Gunduc teach image data is patch-based, downsampled with a transformer encoder to generate modified patch based encoded data and upsampled with transpose convolutions; Fig 1 and 3.4 Upsampling, 3.5 Architecture).
Regarding Claim 20, Gunduc in view of Li et al teach the computer-implemented method of claim 19 (as described above), wherein generating the alpha matte (Li et al, the HOP matting output is an alpha matte estimation based on decoding the encoded data; Fig 2 and 3.1 Hierarchical Opacity Propagation Block ¶ 7) comprises generating, utilizing an activation function of the decoder (Gunduc, the encoded image data is passed to the transpose convolutions for upsampling with a ReLU activation and an output activation function tanh; Fig 1 and 3.5 Architecture, 6. Appendix Network architecture), the alpha matte based on an initial feature set extracted by an initial convolutional neural network layer of the second set of neural network layers and the upsampled feature map (Li et al, the feature maps of the image (appearance encoder) and trimap (opacity encoder) are propagated in parallel with scale-insensitive position encodings for local relative position encodings for local HOP block; Fig 2-4 and 3.1 HOP Block ¶ 4-7, 3.2 Positional Encoding.
Claims 9, 13 are rejected under 35 U.S.C. 103 as being unpatentable over Gunduc (Tensor-to-Image: Image-to-Image Translation with Vision Transformers, cited in IDS 07/19/2024) in view of Li et al (Hierarchical Opacity Propagation for Image Matting) and Ren et al (Unifying Global-Local Representations in Salient Object Detection with Transformer).
Regarding Claim 9, Gunduc in view of Li et al teach the non-transitory computer readable medium of claim 8 (as described above), including providing the global context information to the plurality of upsampling layers (Li et al, the Appearance Encoder layers are connected to a plurality of HOP Global upsampling layers; Fig 2 and 3.1 Hierarchical Opacity Propagation Block 3-5).
Gunduc in view of Li et al does not explicitly teach generating modified patch-based encodings including the global context information utilizing a plurality of multilayer perceptrons connected to the plurality of transformer neural network layers; and providing the modified patch-based encodings from the plurality of multilayer perceptrons to the plurality of upsampling layers via the set of skip connections.
Ren et al is analogous art pertinent to the technological problem addressed in the current application and teaches generating modified patch-based encodings including the global context information utilizing a plurality of multilayer perceptrons connected to the plurality of transformer neural network layers (patch embeddings are generated, including for global feature extraction, and passed through the transformer encoder, with the transformer layers including a multi-head attention and a multilayer perceptron; Fig 3 and 3 Method, 3.2.2 Transformer Encoder, Transformer layer); and providing the modified patch-based encodings from the plurality of multilayer perceptrons to the plurality of upsampling layers via the set of skip connections (the MLP creates modified patch-based encodings, which are then passed with skip connections to the Deeply-transformed Decoder for upsampling (corresponding transformer layer with previous layer) in a plurality of stages; Fig 3, 4 and 3.3 Deeply-transformed Decoder).
It would have been obvious to one of ordinary skill in the art to combine the teachings of Gunduc in view of Li et al with Ren et al including generating modified patch-based encodings including the global context information utilizing a plurality of multilayer perceptrons connected to the plurality of transformer neural network layers; and providing the modified patch-based encodings from the plurality of multilayer perceptrons to the plurality of upsampling layers via the set of skip connections. By jointly learning global and local features and using a transformer encoder, global representation is learned in addition to the local features and by using a MHA and MLP, the image representations are better preserved for decoding, thereby leading to improved mean absolute error (more effective representations after decoding), as recognized by Ren et al (1. Introduction ¶ 5-6).
Regarding Claim 13, Gunduc in view of Li et al teach the computer-implemented method of claim 12 (as described above), including generating the patch-based encodings (Gunduc, the transformer encoder includes a number of layers to produce lower-dimensional (resolution) linear embeddings from the patch data; Fig 1 and 2. Related work ¶ 3, 3.2 Transformer Layers).
Gunduc in view of Li et al does not explicitly teach generating the patch-based encodings to include the global context information by comparing different areas of the digital image; and providing the patch-based encodings at a plurality of resolutions to the first set of neural network layers.
Ren et al is analogous art pertinent to the technological problem addressed in the current application and teaches generating the patch-based encodings to include the global context information by comparing different areas of the digital image (global features are passed through the transformer encoder, with the transformer layers considering the global features while analyzing the patch local (different areas) data; Fig 3 and 3 Method, 3.2.2 Transformer Encoder, Transformer layer); and providing the patch-based encodings at a plurality of resolutions to the first set of neural network layers (the MLP creates modified patch-based encodings, which are then processed at different resolutions to generate reduced resolution at each stage during encoding before decoding in a plurality of stages to reach the same spatial resolution of the input images; Fig 3, 4 and 3.2.1 Image Serialization, 3.3 Deeply-transformed Decoder).
It would have been obvious to one of ordinary skill in the art to combine the teachings of Gunduc in view of Li et al with Ren et al including generating the patch-based encodings to include the global context information by comparing different areas of the digital image; and providing the patch-based encodings at a plurality of resolutions to the first set of neural network layers. By jointly learning global and local features and using a transformer encoder, global representation is learned by using a MHA and MLP, the image representations are better preserved for decoding to upsample to the resolution of the input image, thereby leading to improved mean absolute error (more effective representations after decoding), as recognized by Ren et al (1. Introduction ¶ 5-6).
Claims 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Gunduc (Tensor-to-Image: Image-to-Image Translation with Vision Transformers, cited in IDS 07/19/2024) in view of Li et al (Hierarchical Opacity Propagation for Image Matting), Ren et al (Unifying Global-Local Representations in Salient Object Detection with Transformer) and Gao et al (sTransFuse: Fusing Swin Transformer and CNN for Remote Sensing Image Semantic Segmentation).
Regarding Claim 14, Gunduc in view of Li et al and Ren et al teach the computer-implemented method of claim 13 (as described above), including incorporating the global context information at the plurality of upsampling layers (Ren et al, (patch embeddings are generated, including for global feature extraction, and passed through the transformer encoder, with the transformer layers including a multi-head attention and a multilayer perceptron; Fig 3 and 3 Method, 3.2.2 Transformer Encoder, Transformer layer).
Gunduc in view of Li et al and Ren et al do not explicitly teach generating, from the patch-based encodings, modified patch-based encodings utilizing the first set of neural network layers by unifying channel dimensions of feature sets of the patch-based encodings.
Gao et al is analogous art pertinent to the technological problem addressed in the current application and teaches generating, from the patch-based encodings, modified patch-based encodings utilizing the first set of neural network layers by unifying channel dimensions of feature sets of the patch-based encodings (the SwinTransformer builds hierarchical feature maps by merging image patches in deeper layers, including projecting to the Channel C dimension through a linear embedding layer; Fig 1, 2 and III.B. Method STransFuse Architecture ¶ 4-5).
It would have been obvious to one of ordinary skill in the art to combine the teachings of Gunduc in view of Li et al and Ren et al with Gao et al including generating, from the patch-based encodings, modified patch-based encodings utilizing the first set of neural network layers by unifying channel dimensions of feature sets of the patch-based encodings. By applying linear embedding layers to change the output dimension of the channel, the feature maps are encoded with the global and local data without gradient or feature map data loss while maintaining efficient performance and computational speed, as recognized by Gao et al (I. Introduction ¶ 6-7).
Regarding Claim 15, Gunduc in view of Li et al, Ren et al and Gao et al teach the computer-implemented method of claim 14 (as described above), wherein incorporating the global context information at the plurality of upsampling layers comprises providing the modified patch-based encodings from the first set of neural network layers to a subset of upsampling layers of the plurality of upsampling layers of the decoder via the set of skip connections (Li et al, scale-insensitive position encodings for global HOP block encodings with skip connections (self-attention) in the neural network to the decoder; Fig 2-4 and 3.1 HOP Block ¶ 4; Gunduc teach image data is patch-based, transformer encoder encoded, and further processed to generate modified patch based encoded data; Fig 1 and 3.4 Upsampling, 3.5 Architecture).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zheng et al (Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers) teach the use of a transformer encoder for global and local image analysis towards identifying and segmenting objects.
Zeng et al (Learning Joint Spatial-Temporal Transformations for Video Inpainting) teach a system to learn spatial temporal data for video and fill missing regions using self-attention processes and a Spatial-Temporal transformer with multiple-layers to encode.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHLEEN M BROUGHTON whose telephone number is (571)270-7380. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATHLEEN M BROUGHTON/Primary Examiner, Art Unit 2661
Prosecution Insights