DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d).
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 06/28/2024 was filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Preliminary Amendments
Claims 1-17, 19-20 was amended, claim 18 was canceled, claim 21 newly added claim. Therefore, 1-17 and 19-21 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 8, 11, 16-17, and 19-21 are rejected under 35 U.S.C. 103 as being unpatentable over Wang (US 20200151503 A1, Filed Date: Nov. 8, 2018) in view of CHEN (CN114241279B, Filed Date: Dec. 30, 2021).
Regarding independent claim 1, Wang teaches: A text error correction method, comprising:
performing image encoding on an acquired image to be analyzed, so as to obtain image features; (Wang – [0019] all text label information is extracted by the feature encoder from an image. [0044-0045] image encoding using feature encoder on acquired image data to generate image feature maps (features) for subsequent text recognition.)
performing text encoding on acquired noisy text, so as to obtain text features; (Wang – Fig. 3, noisy text in image; [0040] first feature map (e.g., a feature map extracted from a noisy image) [0045-0046] Text recognition application 112 includes noisy image module 148,)
and predicting an initial text label according to the error correction signal by using a trained decoder, so as to obtain error-corrected text information. (Wang – [0020] a text recognition system includes a post-processing step that receives a text prediction generated by the text recognition system, and detects, corrects, or detects and corrects errors based on a confidence score generated from an image adversarial loss term. [0063-0065] [0065] For instance, text system 118 may receive a pre-trained text decoder, T, and a pre-trained feature encoder, E, from server 108 as part of a text recognition system to recognize text in images, such as to generate text prediction 122 from image 120.)
Wang does not explicitly teach: to obtain an error correction signal
However, CHEN teaches: performing feature comparison on the image features and the text features according to a set attention mechanism, so as to obtain an error correction signal; (CHEN − [pdf pages 1, 6] determine error correction results for target text (text features) and the scene pictures (image features) based on multimodal vector representations (attention mechanism). The degree of match between (comparison) text information in the image target to determine the image targe contains any errors (error correction signal).)
PNG
media_image1.png
186
878
media_image1.png
Greyscale
PNG
media_image2.png
300
874
media_image2.png
Greyscale
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate CHEN’s multimodal attention-based features interaction into Wang’s recognition framework in order to improve identification and correction of recognition errors by leveraging contextual information from both image and text modalities, thereby increasing the robustness and accuracy of text correction.
Regarding dependent claim 2, depends on claim 1, Wang teaches: wherein a number of the text features is the same as a number of characters comprised in the noisy text. (Wang – Fig. 3, noisy text in image; [0040] first feature map (e.g., a feature map extracted from a noisy image) [0045-0046] Text recognition application 112 includes noisy image module 148,)
Regarding dependent claim 3, depends on claim 1, Wang does not explicitly teach: cross-attention mechanism;
CHEN teaches: wherein the attention mechanism comprises a self-attention mechanism and a cross-attention mechanism; (CHEN − [pdf page 2] processing the text vector representation and the picture vector based on a self-attention model to obtain global interaction information between the text information of the target text and the image information of the scene picture; [pdf page 11] sufficient cross-modal data can be provided to perform model learning)
and the performing feature comparison on the image features and the text features according to a set attention mechanism, so as to obtain an error correction signal comprises: (CHEN − [pdf pages 1, 6] determine error correction results for target text (text features) and the scene pictures (image features) based on multimodal vector representations (attention mechanism). The degree of match between (comparison) text information in the image target to determine the image targe contains any errors (error correction signal).)
performing association analysis on the image features and the text features according to the self-attention mechanism, so as to obtain alignment features; and analyzing the alignment features and the text features according to the self-attention mechanism and the cross-attention mechanism, so as to obtain the error correction signal. (CHEN − [pdf pages 1, 6] The degree of match between (comparison) text information in the image target to determine the image targe contains any errors (error correction signal).) CHEN teaches global interaction information between text and image information and multimodal representation
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate CHEN’s multimodal attention-based features interaction into Wang’s recognition framework in order to improve identification and correction of recognition errors by leveraging contextual information from both image and text modalities, thereby increasing the robustness and accuracy of text correction.
Regarding dependent claim 4, depends on claim 3, Wang teaches: wherein the alignment features comprise correspondence relationships between the image features and the text features. (Wang − [0019] A text recognition system is trained to be feature complete by requiring that all text label information is extracted by the feature encoder from an image. This requirement is equivalent to requiring the existence of an image generator that can generate a clean image from features extracted from a noisy image. Hence, an image generator receives features extracted from a training image using a feature encoder, and generates a reconstructed clean image. At the pixel level, an image discriminator is adversarially trained against the feature encoder and the image generator using the clean image and the reconstructed clean image.)
Regarding dependent claim 5, depends on claim 3, Wang does not explicitly teach: wherein the self-attention mechanism comprises a self-attention layer, a layer normalization module, and an adding module.
CHEN teaches: wherein the self-attention mechanism comprises a self-attention layer, a layer normalization module, and an adding module. (CHEN − [pdf page 2] determining a multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information includes: adding the global interaction information and the first normalization information to obtain first sum information; inputting the first sum information into a fully connected layer for processing, and then normalizing the output result of the fully connected layer to obtain second normalized information; adding [pdf page 12] when determining the multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information,)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate CHEN’s multimodal attention-based features interaction into Wang’s recognition framework in order to improve identification and correction of recognition errors by leveraging contextual information from both image and text modalities, thereby increasing the robustness and accuracy of text correction.
Regarding dependent claim 8, depends on claim 7, Wang does not explicitly teach: wherein the self-attention vectors comprise associated features between each dimension of feature of the image features and each dimension of feature of the text features.
CHEN teaches: wherein the self-attention vectors comprise associated features between each dimension of feature of the image features and each dimension of feature of the text features. (CHEN – [pdf page 8] The length dimension of the global interaction information is the same as that of the embedding vector representation. [pdf page 2] determining a multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information includes: adding the global interaction information and the first normalization information to obtain first sum information; inputting the first sum information into a fully connected layer for processing, and then normalizing the output result of the fully connected layer to obtain second normalized information; adding [pdf page 12] when determining the multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information,)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate CHEN’s multimodal attention-based features interaction into Wang’s recognition framework in order to improve identification and correction of recognition errors by leveraging contextual information from both image and text modalities, thereby increasing the robustness and accuracy of text correction.
Regarding dependent claim 11, depends on claim 10, Wang does not explicitly teach: wherein the error correction processing is achieved based on superimposition of a plurality of error correction layers.
CHEN teaches: wherein the error correction processing is achieved based on superimposition of a plurality of error correction layers. (CHEN − [pdf pages 1, 6] determine error correction results for target text (text features) and the scene pictures (image features) based on multimodal vector representations (attention mechanism). The degree of match between (comparison) text information in the image target to determine the image targe contains any errors (error correction signal).)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate CHEN’s multimodal attention-based features interaction into Wang’s recognition framework in order to improve identification and correction of recognition errors by leveraging contextual information from both image and text modalities, thereby increasing the robustness and accuracy of text correction.
Regarding dependent claim 16, depends on claim 1, Wang teaches: further comprising: training a decoder to obtain the trained decoder. (Wang − [0041] Storage 128 also includes text data 140, including data associated with text of images, such as a ground truth text string, a text prediction generated by a text decoder of text recognition system 110, [0045] Text recognition system 110 also includes text recognition application 112. Text recognition application 112 includes noisy image module 148, clean image module 150, feature extraction module 152, text prediction module 154, image generator module 156, feature discriminator module 158, image discriminator module 160, and training module 162. These modules work in conjunction with each other to train text recognition systems to recognize text in images, such as a feature encoder of feature extraction module 152 and a text decoder of text prediction module 154.)
Regarding dependent claim 17, depends on claim 16, Wang teaches: wherein the training the decoder comprises: (Wang − [0041] Storage 128 also includes text data 140, including data associated with text of images, such as a ground truth text string, a text prediction generated by a text decoder of text recognition system 110, [0045] Text recognition system 110 also includes text recognition application 112. Text recognition application 112 includes noisy image module 148, clean image module 150, feature extraction module 152, text prediction module 154, image generator module 156, feature discriminator module 158, image discriminator module 160, and training module 162. These modules work in conjunction with each other to train text recognition systems to recognize text in images, such as a feature encoder of feature extraction module 152 and a text decoder of text prediction module 154.)
Wang does not explicitly teach: error correction signal
CHEN teaches: acquiring a historical error correction signal and correct text corresponding to the historical error correction signal; and training the decoder by using the historical error correction signal and the correct text, so as to obtain the trained decoder. (CHEN − [pdf pages 1, 6] determine error correction results for target text (text features) and the scene pictures (image features) based on multimodal vector representations (attention mechanism). The degree of match between (comparison) text information in the image target to determine the image targe contains any errors (error correction signal).)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate CHEN’s multimodal attention-based features interaction into Wang’s recognition framework in order to improve identification and correction of recognition errors by leveraging contextual information from both image and text modalities, thereby increasing the robustness and accuracy of text correction.
Regarding independent claim 19, is directed to an electronic device. Claim 19 have similar/same technical features/limitations as claim 1 and claim 19 is rejected under the same rationale.
Regarding independent claim 20, is directed to a non-transitory computer-readable medium. Claim 20 have similar/same technical features/limitations as claim 1 and claim 20 is rejected under the same rationale.
Regarding dependent claim 21, Wang teaches: further comprising: updating text information in the acquired noisy text in input samples of a Multi Modal learning model with text information corresponding to the initial text label with is predicted. (Wang – [0020] a text recognition system includes a post-processing step that receives a text prediction generated by the text recognition system, and detects, corrects, or detects and corrects errors based on a confidence score generated from an image adversarial loss term. [0063-0065] [0065] For instance, text system 118 may receive a pre-trained text decoder, T, and a pre-trained feature encoder, E, from server 108 as part of a text recognition system to recognize text in images, such as to generate text prediction 122 from image 120.)
Claim(s) 6-7, 10, 13, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Wang, CHEN as applied to claims 1 and 3 above, and further in view of Li (US 20230154146 A1, Filed Date: Dec. 30, 2021).
Regarding dependent claim 6, depends on claim 3, Wang does not explicitly teach: wherein the performing association analysis on the image features and the text features according to the self-attention mechanism,
CHEN teaches: wherein the performing association analysis on the image features and the text features according to the self-attention mechanism, (CHEN − [pdf page 2] determining a multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information includes: adding the global interaction information and the first normalization information to obtain first sum information; inputting the first sum information into a fully connected layer for processing, and then normalizing the output result of the fully connected layer to obtain second normalized information; adding [pdf page 12] when determining the multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information,)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate CHEN’s multimodal attention-based features interaction into Wang’s recognition framework in order to improve identification and correction of recognition errors by leveraging contextual information from both image and text modalities, thereby increasing the robustness and accuracy of text correction.
Wang does not explicitly teach: so as to obtain alignment features comprises: splicing the image features with the text features, inputting spliced image features and text features to the self-attention mechanism for encoding, so as to obtain the alignment features output by the self-attention mechanism.
Li teaches: so as to obtain alignment features comprises: splicing the image features with the text features, inputting spliced image features and text features to the self-attention mechanism for encoding, so as to obtain the alignment features output by the self-attention mechanism. (Li − [0051] The video-and-language alignment module 630 may generate an output 650, such as an alignment prediction between the video and text inputs. [0056] At step 704, a video encoder (e.g., 220 in FIG. 3) may encode the plurality of video frames into video feature representations. For example, each video frame may be partitioned into a number of non-overlapping patches. The number of non-overlapping patches may be fed to a linear projection layer to produce a sequence of patch tokens. A video start token may be appended to the sequence of patch tokens. Self-attention may then be applied along a temporal dimension and a spatial dimension to an input sequence of tokens to result in per-frame features, and temporal fusion may then be applied to the per-frame features along the temporal dimension to aggregate per-frame features into video features.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate Li’s feature fusion and self-attention processing into Wang and CHEN framework In order to jointly process image and text features within a common attention framework, thereby improving utilization of contextual information from both modalities during text error correction.
Regarding dependent claim 7, depends on claim 6, Wang does not explicitly teach: wherein the performing association analysis on the image features and the text features according to the self-attention mechanism, so as to obtain alignment features comprises: determining self-attention vectors of the image features and the text features; and performing layer normalization and adding processing on the self-attention vectors, so as to obtain the alignment features.
Li teaches: wherein the performing association analysis on the image features and the text features according to the self-attention mechanism, so as to obtain alignment features comprises: determining self-attention vectors of the image features and the text features; and performing layer normalization and adding processing on the self-attention vectors, so as to obtain the alignment features. (Li − [0051] The video-and-language alignment module 630 may generate an output 650, such as an alignment prediction between the video and text inputs. [0056] At step 704, a video encoder (e.g., 220 in FIG. 3) may encode the plurality of video frames into video feature representations. For example, each video frame may be partitioned into a number of non-overlapping patches. The number of non-overlapping patches may be fed to a linear projection layer to produce a sequence of patch tokens. A video start token may be appended to the sequence of patch tokens. Self-attention may then be applied along a temporal dimension and a spatial dimension to an input sequence of tokens to result in per-frame features, and temporal fusion may then be applied to the per-frame features along the temporal dimension to aggregate per-frame features into video features.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate Li’s feature fusion and self-attention processing into Wang and CHEN framework In order to jointly process image and text features within a common attention framework, thereby improving utilization of contextual information from both modalities during text error correction.
Regarding dependent claim 10, depends on claim 3, Wang does not explicitly teach: performing attention analysis on the text features according to the self-attention mechanism, so as to obtain self-attention features of the text features;
CHEN teaches: performing attention analysis on the text features according to the self-attention mechanism, so as to obtain self-attention features of the text features; (CHEN – [pdf page 8] The length dimension of the global interaction information is the same as that of the embedding vector representation. [pdf page 2] determining a multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information includes: adding the global interaction information and the first normalization information to obtain first sum information; inputting the first sum information into a fully connected layer for processing, and then normalizing the output result of the fully connected layer to obtain second normalized information; adding [pdf page 12] when determining the multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information,)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate CHEN’s multimodal attention-based features interaction into Wang’s recognition framework in order to improve identification and correction of recognition errors by leveraging contextual information from both image and text modalities, thereby increasing the robustness and accuracy of text correction.
Wang does not explicitly teach: wherein the analyzing the alignment features and the text features according to the self-attention mechanism and the cross-attention mechanism, so as to obtain the error correction signal comprises:
Li teaches: wherein the analyzing the alignment features and the text features according to the self-attention mechanism and the cross-attention mechanism, so as to obtain the error correction signal comprises: (Li − [0051] The video-and-language alignment module 630 may generate an output 650, such as an alignment prediction between the video and text inputs. [0056] At step 704, a video encoder (e.g., 220 in FIG. 3) may encode the plurality of video frames into video feature representations. For example, each video frame may be partitioned into a number of non-overlapping patches. The number of non-overlapping patches may be fed to a linear projection layer to produce a sequence of patch tokens. A video start token may be appended to the sequence of patch tokens. Self-attention may then be applied along a temporal dimension and a spatial dimension to an input sequence of tokens to result in per-frame features, and temporal fusion may then be applied to the per-frame features along the temporal dimension to aggregate per-frame features into video features.)
performing attention analysis on the alignment features according to the self-attention mechanism, so as to obtain self-attention features of the alignment features; (Li − [0051] The video-and-language alignment module 630 may generate an output 650, such as an alignment prediction between the video and text inputs. [0056] At step 704, a video encoder (e.g., 220 in FIG. 3) may encode the plurality of video frames into video feature representations. For example, each video frame may be partitioned into a number of non-overlapping patches. The number of non-overlapping patches may be fed to a linear projection layer to produce a sequence of patch tokens. A video start token may be appended to the sequence of patch tokens. Self-attention may then be applied along a temporal dimension and a spatial dimension to an input sequence of tokens to result in per-frame features, and temporal fusion may then be applied to the per-frame features along the temporal dimension to aggregate per-frame features into video features.)
determining cross-attention vectors between the self-attention features of the alignment features and the self-attention features of the text features; (Li − [0032] Existing sparse video-language pre-training models use either dot-product or rely entirely on a transformer encoder to model cross-modal interactions. However, since video and text features reside in different embedding spaces, such methods lead to less satisfactory alignment. Instead, the video representation 315 and text representation 316 are fed to a video-text contrastive (VTC) loss module 330 to align features from the unimodal encoders 220 and 222 before sending them into the multimodal encoder 230. Specifically, given the embeddings of video [CLS] token 315 and the embedding 316 of text [CLS] tokens, a similarity score is computed between video V and text T: )
and performing layer normalization, adding, and error correction processing on the cross-attention vectors, so as to obtain the error correction signal. (Li − [0051] The video-and-language alignment module 630 may generate an output 650, such as an alignment prediction between the video and text inputs. [0056] At step 704, a video encoder (e.g., 220 in FIG. 3) may encode the plurality of video frames into video feature representations. For example, each video frame may be partitioned into a number of non-overlapping patches. The number of non-overlapping patches may be fed to a linear projection layer to produce a sequence of patch tokens. A video start token may be appended to the sequence of patch tokens. Self-attention may then be applied along a temporal dimension and a spatial dimension to an input sequence of tokens to result in per-frame features, and temporal fusion may then be applied to the per-frame features along the temporal dimension to aggregate per-frame features into video features.)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate Li’s feature fusion and self-attention processing into Wang and CHEN framework In order to jointly process image and text features within a common attention framework, thereby improving utilization of contextual information from both modalities during text error correction.
Regarding dependent claim 13, depends on claim 10, Wang does not explicitly teach: wherein the determining cross-attention vectors between the self-attention features of the alignment features and the self-attention features of the text features comprises: setting a threshold attention mechanism, and determining the cross-attention vectors between the self-attention features of the alignment features and the self-attention features of the text features through the threshold attention mechanism.
Li teaches: wherein the determining cross-attention vectors between the self-attention features of the alignment features and the self-attention features of the text features comprises: setting a threshold attention mechanism, and determining the cross-attention vectors between the self-attention features of the alignment features and the self-attention features of the text features through the threshold attention mechanism. (Li − [0032] Existing sparse video-language pre-training models use either dot-product or rely entirely on a transformer encoder to model cross-modal interactions. However, since video and text features reside in different embedding spaces, such methods lead to less satisfactory alignment. Instead, the video representation 315 and text representation 316 are fed to a video-text contrastive (VTC) loss module 330 to align features from the unimodal encoders 220 and 222 before sending them into the multimodal encoder 230. Specifically, given the embeddings of video [CLS] token 315 and the embedding 316 of text [CLS] tokens, a similarity score is computed between video V and text T: )
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate Li’s feature fusion and self-attention processing into Wang and CHEN framework In order to jointly process image and text features within a common attention framework, thereby improving utilization of contextual information from both modalities during text error correction.
Regarding dependent claim 15, depends on claim 1, Wang teaches: so as to obtain error-corrected text information comprises: performing self-attention analysis on the error correction signal and the initial text label, (Wang – [0020] a text recognition system includes a post-processing step that receives a text prediction generated by the text recognition system, and detects, corrects, or detects and corrects errors based on a confidence score generated from an image adversarial loss term. [0063-0065] [0065] For instance, text system 118 may receive a pre-trained text decoder, T, and a pre-trained feature encoder, E, from server 108 as part of a text recognition system to recognize text in images, such as to generate text prediction 122 from image 120.)
Wang does not explicitly teach: wherein the initial text label comprises a starting symbol;
Li teaches: wherein the initial text label comprises a starting symbol; and the predicting an initial text label according to the error correction signal by using a trained decoder, and determining a next character adjacent to the initial text label; and adding the next character into the initial text label, returning to a step of performing self-attention analysis on the error correction signal and the initial text label, and determining a next character adjacent to the initial text label until the next character is a termination character, and using a current initial text label as error-corrected text information to update the acquired noisy text. (Li − [0031] In one embodiment, the text encoder 222 may be a 6-layer transformer model to represent text tokens in the text input 304. Given an input text description 304 of N.sub.t tokens, the text encoder 222 outputs an embedding sequence and the embedding 316 of the text [CLS] token. Similar to video encoder 220, positional embeddings are added to the text tokens. sequence generation with a special beginning token that serves as initial decoder input)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate Li’s feature fusion and self-attention processing into Wang and CHEN framework In order to jointly process image and text features within a common attention framework, thereby improving utilization of contextual information from both modalities during text error correction.
Claim(s) 9 is rejected under 35 U.S.C. 103 as being unpatentable over Wang, CHEN applied to claim 8 above, and further in view of Gu (US 20220382975 A1, Filed Date: May 28, 2021).
Regarding dependent claim 9, depends on claim 8, Wang does not explicitly teach: wherein the determining self-attention vectors of the image features and the text features comprises: determining the self-attention vectors of the image features and the text features according to the following formulas:
wherein the determining self-attention vectors of the image features and the text features comprises: determining the self-attention vectors of the image features and the text features according to the following formulas: the spliced image features and text features; (CHEN – [pdf page 8] The length dimension of the global interaction information is the same as that of the embedding vector representation. [pdf page 2] determining a multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information includes: adding the global interaction information and the first normalization information to obtain first sum information; inputting the first sum information into a fully connected layer for processing, and then normalizing the output result of the fully connected layer to obtain second normalized information; adding [pdf page 12] when determining the multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information,)
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate CHEN’s multimodal attention-based features interaction into Wang’s recognition framework in order to improve identification and correction of recognition errors by leveraging contextual information from both image and text modalities, thereby increasing the robustness and accuracy of text correction.
Wang does not explicitly teach:
PNG
media_image3.png
184
641
media_image3.png
Greyscale
Gu teaches:
PNG
media_image3.png
184
641
media_image3.png
Greyscale
(Gu – [0078] , the self-attention function is defined using the expression below
PNG
media_image4.png
221
547
media_image4.png
Greyscale
Claim(s) 12 is rejected under 35 U.S.C. 103 as being unpatentable over Wang, CHEN, and Li applied to claim 10 above, and further in view of Gu (US 20220382975 A1, Filed Date: May 28, 2021).
Regarding dependent claim 12, depends on claim 10, Wang does not explicitly teach: wherein the determining cross-attention vectors between the self-attention features of the alignment features and the self-attention features of the text features comprises: determining the cross-attention vectors between the self-attention features of the alignment features and the self-attention features of the text features according to the following formula wherein f represents the self-attention vectors of the alignment features; g represents the self- attention vectors of the text features; and Wq, Wk, and Wv are al model parameters obtained by model training.
Li teaches: wherein the determining cross-attention vectors between the self-attention features of the alignment features and the self-attention features of the text features comprises: (Li − [0051] The video-and-language alignment module 630 may generate an output 650, such as an alignment prediction between the video and text inputs. [0056] At step 704, a video encoder (e.g., 220 in FIG. 3) may encode the plurality of video frames into video feature representations. For example, each video frame may be partitioned into a number of non-overlapping patches. The number of non-overlapping patches may be fed to a linear projection layer to produce a sequence of patch tokens. A video start token may be appended to the sequence of patch tokens. Self-attention may then be applied along a temporal dimension and a spatial dimension to an input sequence of tokens to result in per-frame features, and temporal fusion may then be applied to the per-frame features along the temporal dimension to aggregate per-frame features into video features.)
determining the cross-attention vectors between the self-attention features of the alignment features and the self-attention features of the text features according to the following formula (Li − [0051] The video-and-language alignment module 630 may generate an output 650, such as an alignment prediction between the video and text inputs. [0056] At step 704, a video encoder (e.g., 220 in FIG. 3) may encode the plurality of video frames into video feature representations. For example, each video frame may be partitioned into a number of non-overlapping patches. The number of non-overlapping patches may be fed to a linear projection layer to produce a sequence of patch tokens. A video start token may be appended to the sequence of patch tokens. Self-attention may then be applied along a temporal dimension and a spatial dimension to an input sequence of tokens to result in per-frame features, and temporal fusion may then be applied to the per-frame features along the temporal dimension to aggregate per-frame features into video features.)
wherein f represents the self-attention vectors of the alignment features; g represents the self- attention vectors of the text features; and Wq, Wk, and Wv are al model parameters obtained by model training. (Li − [0032] Existing sparse video-language pre-training models use either dot-product or rely entirely on a transformer encoder to model cross-modal interactions. However, since video and text features reside in different embedding spaces, such methods lead to less satisfactory alignment. Instead, the video representation 315 and text representation 316 are fed to a video-text contrastive (VTC) loss module 330 to align features from the unimodal encoders 220 and 222 before sending them into the multimodal encoder 230. Specifically, given the embeddings of video [CLS] token 315 and the embedding 316 of text [CLS] tokens, a similarity score is computed between video V and text T: )
Accordingly, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claim invention, to incorporate Li’s feature fusion and self-attention processing into Wang and CHEN framework In order to jointly process image and text features within a common attention framework, thereby improving utilization of contextual information from both modalities during text error correction.
Wang does not explicitly teach:
PNG
media_image5.png
207
650
media_image5.png
Greyscale
Gu teaches:
PNG
media_image5.png
207
650
media_image5.png
Greyscale
(Gu – [0078] , the self-attention function is defined using the expression below
PNG
media_image4.png
221
547
media_image4.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claim invention to incorporate Gu’s cross-attention into Wang, CHEN, and Li frame work in order to improve interaction between image-derived features and text-derived features, thereby enabling the model to selectively focus on relevant multimodal information and improving the accuracy of error detection and correction.
Allowable Subject Matter
Claim 14 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Schulze, US 20160150235 A1, Text correction for noisy textual information.
Hashimoto, US 10963652 B2, Structured Text Translation via Self-Attention.
Srinivasan, US 20210264109 A1, Text Rewriting for a Target Author.
Zhang, US 20230162490 A1, Vision-language Distribution Alignment.
Li, US 20230237773 A1, Unified Vision-language Understanding.
XUE, US 20240119743 A1. Pre-training for Scene Text Detection.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CARL E BARNES JR whose telephone number is (571)270-3395. The examiner can normally be reached Monday-Friday 9am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Hong can be reached at (571) 272-4124. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CARL E BARNES JR/Examiner, Art Unit 2178
/STEPHEN S HONG/Supervisory Patent Examiner, Art Unit 2178
Prosecution Insights