DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/27/2026 has been entered.
The status of the claims is as follows.
Claims 1, 3, 10 and 12 are amended. Claims 1-4, 9, 10-13, 18 and 19 are currently pending.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 9; 10, 18; 19 are rejected under 35 U.S.C. 103 as being unpatentable over Tsai et al. (“Multimodal Transformer for Unaligned Multimodal Language Sequences” [2019], hereinafter “Tsai”) in view of Zhao et al. (US20200356802A1, hereinafter “Zhao”).
Regarding Claim 1,
Tsai discloses An electronic apparatus for performing a preset task by using a deep neural network (DNN) (Tsai [Page 1 Section 1 Paragraph 2]; “To address the above issues, in this paper we propose the Multimodal Transformer (MulT), an end-to-end model that extends the standard Transformer network (Vaswani et al., 2017) to learn representations directly from unaligned multimodal streams.” Wherein the multimodal transformer being an end-to-end model extension of the standard Transformer network thus reads on the preset task being performed through a deep neural network)
the electronic apparatus comprising: an input interface configured to receive input data of a first type and input data of a second type (Tsai [Figure 3];
PNG
media_image1.png
277
622
media_image1.png
Greyscale
)
a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory (Tsai [Page 1 Section 1 Paragraph 2]; “To address the above issues, in this paper we propose the Multimodal Transformer (MulT), an end-to-end model that extends the standard Transformer network (Vaswani et al., 2017) to learn representations directly from unaligned multimodal streams.” Wherein the multimodal transformer being an end-to-end model extension of the standard Transformer network thus reads on the preset task being performed through a deep neural network
Tsai [Acknowledgements]; “This work was supported in part by DARPA HR00111990016, AFRL FA8750-18-C-0014, NSF IIS1763562 #1750439 #1722822, Apple, Google focused award, and Samsung. We would also like to acknowledge NVIDIAs GPU support” wherein NVIDIA’s usage as GPU support reads on a processor configured to execute instructions stored in the memory associated with MuIT architecture)
obtain first sub-feature information corresponding to the input data of the first type and second sub-feature information corresponding to the input data of the second type (Tsai [Section 3.1]; “We consider two modalities α and β, with two (potentially non-aligned) sequences from each of them denoted Xα ∈ RTα× dα and Xβ ∈ RTβ× dβ, respectively. For the rest of the paper, T(·) and d(·) are used to represent sequence length and feature dimension, respectively. Inspired by the decoder transformer in NMT (Vaswani et al., 2017) that translates one language to another, we hypothesize a good way to fuse crossmodal information is pro viding a latent adaptation across modalities; i.e., β to α. Note that the modalities consider in our pa per may span very different domains such as facial attributes and spoken words.”
obtain feature information from each of a plurality of layers of the DNN by inputting the first sub-feature information and the second sub-feature information into the DNN (Tsai [Section 3.2];
PNG
media_image2.png
320
311
media_image2.png
Greyscale
)
obtain first query information corresponding to each of the plurality of layers, based on the first sub-feature information and a pre-trained query matrix corresponding to each of the plurality of layers, wherein the first query information indicates a weight of the first sub-feature information (Tsai [Section 3.1];
PNG
media_image3.png
445
308
media_image3.png
Greyscale
Wherein Qa = XaWQa thus reads on first query information associated with the first sub-feature modality information based in part on a pre-trained query matrix WQa)
obtain second query information corresponding to each of the plurality of layers, based on the second sub-feature information and the pre-trained query matrix, wherein the second query information indicates a weight of the second sub-feature information (Tsai [Section 3.1];
PNG
media_image3.png
445
308
media_image3.png
Greyscale
Wherein Qb = XbWQb thus reads on second query information associated with the second sub-feature modality information based in part on a pre-trained query matrix WQb)
obtain key information corresponding to each of the plurality of layers, based on the feature information extracted from each of the plurality of layers and a pre-trained key matrix corresponding to each of the plurality of layers (Tsai [Section 3.1];
PNG
media_image3.png
445
308
media_image3.png
Greyscale
Wherein the keys obtained through the feature information X across the plurality of modalities thus reads on key information; wherein the key information obtained through WKB thus reads on such key information being based in part on a pre-trained key matrix)
obtain first context information corresponding to each of the plurality of layers, the first context information indicating a correlation between the first query information and the key information (Tsai [Figure 3];
PNG
media_image4.png
297
615
media_image4.png
Greyscale
Wherein the bidirectional cross modal transformation involving the softmax computation to determine cross modal attention between elements of the two modalities thus reads on some first context information indicative of a correlation between the first query and second-modality key information (QaKBT element representative of aforementioned relationship))
obtain second context information corresponding to each of the plurality of layers, the second context information indicating a correlation between the second query information and the key information (Tsai [Figure 3];
PNG
media_image4.png
297
615
media_image4.png
Greyscale
Wherein the bidirectional cross modal transformation involving the softmax computation to determine cross modal attention between elements of the two modalities thus reads on some second context information indicative of an opposite direction correlation between the first query and second-modality key information (QbKAT element representative of aforementioned relationship))
calculate a weight for each type corresponding to each of the plurality of layers based on the first context information and the second context information corresponding to each of the plurality of layers (Tsai [Figure 3];
PNG
media_image4.png
297
615
media_image4.png
Greyscale
Wherein the bidirectional cross modal transformation involving the softmax computation involving the conversion of such context scores QaKbT/sqrt(dk) and QbKaT/sqrt(dK) into weights through the softmax activation function thus reads on obtained weights associated with the plurality of modality types)
obtain a final output value corresponding to the preset task by applying the weight for each type, in each of the plurality of layers (Tsai [Section 3.1];
PNG
media_image5.png
353
329
media_image5.png
Greyscale
Tsai [Figure 6];
PNG
media_image6.png
279
646
media_image6.png
Greyscale
)
Tsai does not disclose but Zhao discloses wherein the pre-trained query matrix comprises parameters related to the first sub-feature information and the second sub-feature information (Zhao [0050]; “feature extraction is performed on a to-be-processed image to generate a feature map of the image, a feature weight corresponding to each of multiple feature points included in the feature map is determined” wherein each of multiple feature points included in the feature map reads on the pre-trained query matrix comprising parameters of the extracted sub-features’ information.
Zhao [0148]; “The feature extraction network involved in the embodiments can be pre-trained or untrained” wherein the feature map understood to be pre-trained reads on a pre-trained query matrix)
It would have been obvious to use Zhao’s method of utilizing a single query matrix comprising parameters related to the first and second sub-feature information in place of Tsai’s separate two modality query matrices each comprising first and second sub-feature information respectively. One would have been motivated to do so “in order to obtain comprehensive weight information corresponding to each feature point in the feature map” (Zhao [0093]).
Regarding Claim 9,
Tsai/Zhao teaches the method of Claim 1 (and thus the rejection of Claim 1 is incorporated). Tsai/Zhao further discloses wherein the input data of the first type and the input data of the second type comprise at least one of image data, text data, sound data, or video data (Tsai [Figure 1];
PNG
media_image7.png
356
324
media_image7.png
Greyscale
Wherein video, text and sound data are all disclosed modalities)
Claims 10, 18 recite the same method performed by the electronic apparatus of Claims 1, 9. Thus, Claims 10, 18 are rejected for reasons set forth in the rejection of Claims 1, 9, respectively.
Claim 19 recites a non-transitory medium storing instructions to perform the same method of Claim 10. Thus, Claim 19 is rejected for reasons set forth in the rejection of Claim 10.
Claims 2, 11 are rejected under 35 U.S.C. 103 as being unpatentable over Tsai et al. (“Multimodal Transformer for Unaligned Multimodal Language Sequences” [2019], hereinafter “Tsai”) in view of Zhao et al. (US20200356802A1, hereinafter “Zhao”) in view of Cai et al. (US20200279156A1, hereinafter “Cai”).
Regarding Claim 2,
Tsai/Zhao teaches the method of Claim 1 (and thus the rejection of Claim 1 is incorporated). Tsai/Zhao fails to explicitly disclose but Cai discloses to obtain the first sub-feature information by inputting the input data of the first type into a pre-trained first sub-network; and obtain the second sub-feature information by inputting the input data of the second type into a pre-trained second sub-network (Cai [0016]; “System 100 generally receives video input 110 and audio input 132, trains itself based on sample data, extracts features from the inputs using both deep learning models (e.g., deep learning based video models 114 and deep learning based audio models)
It would have been obvious to perform Tsai/Zhao’s sub-feature information extraction through inputting input data of first and second modalities into a single network model by utilizing Cai’s first and second sub-networks with models specifically designed for feature extraction of a certain modality. One would have been motivated to do so because “Deep learning based models may differ depending upon which data modality they are used for. For example, the deep learning based video models 114 may include convolutional neural networks (CNNs), etc., while the deep learning based audio models 134 may include recurrent neural networks (RNNs)” (Cai [0019]).
Claim 11 recites the same method performed by the electronic apparatus of Claim 2. Thus, Claim 11 is rejected for reasons set forth in the rejection of Claim 2.
Claims 3-4; 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Tsai et al. (“Multimodal Transformer for Unaligned Multimodal Language Sequences” [2019], hereinafter “Tsai”) in view of Zhao et al. (US20200356802A1, hereinafter “Zhao”) in view of Chan et al. (US20210406266A1, hereinafter “Chan”).
Regarding Claim 3,
Tsai/Zhao teaches the method of Claim 1 (and thus the rejection of Claim 1 is incorporated). Tsai/Zhao further discloses to input the … first sub-feature information and the … second sub- feature information to the DNN (Tsai [Figure 3];
PNG
media_image1.png
277
622
media_image1.png
Greyscale
)
Tsai/Zhao fails to explicitly disclose but Chan teaches to encode, based on type identification information that distinguishes between the first type and the second type, the first sub-feature information and the second sub-feature information (Chan [0027]; “Particular embodiments derive (e.g., receive or generate) a feature vector that represents the set of features of the first cell based on the extracting of the set of features. A “feature vector” (also referred to as a “vector”) as described herein includes one or more real numbers, such as a series of floating values or integers (e.g., [0, 1, 0, 0]) that represent one or more other real numbers, a natural language (e.g., English) word and/or other character sequence (e.g., a symbol (e.g., @, !, #), a phrase, and/or sentence, etc.). Such natural language words and/or character sequences correspond to the set of features and are encoded or converted into corresponding feature vectors so that computers can process the corresponding extracted features.” wherein the feature vectors of an integer type as well as language type read on first and second sub-feature information
Chan [0092]; “The feature the feature vector 404 represents a shape vector. A “shape vector” is a feature vector that indicates the type or class of one or more characters of a particular element. For instance, the shape vector can indicate whether a character is a letter, number (e.g., an integer or other real number), and/or a symbol or type of symbol (e.g., picture, exclamation point, question mark, etc.).” wherein the shape vector indicating the type of the input data reads on type identification information
Chan [0028]; “Particular embodiments of the present disclosure model or process these data by sequentially encoding particular feature vectors based on using one or more machine learning models. For example, some embodiments convert each feature vector of a row in a table from left to right in an ordered fashion into another concatenated feature vector. In some embodiments, such sequential encoding includes using a 1-dimensional and/or 2-dimensional bi-directional Long Short Term Memory (LSTM) model to encode sequential data into a concatenated or aggregated feature vector of multiple values representing multiple cells in a table.” wherein encoding is performed through concatenation of the feature vector information)
Tsai/Zhao reads on input[ting] the … first sub-feature information and the … second sub- feature information to the DNN. Although Tsai/Zhao doesn’t read on input[ting] the encoded first sub-feature information and the encoded second sub- feature information to the DNN, Chan reads on the encoded first and second sub-feature. Therefore, by using Chan’s encoded sub-features in the method of Tsai/Zhao, the combination reads on input[ting] the encoded first sub-feature information and the encoded second sub- feature information to the DNN
It would have been obvious to use Chan’s encoded sub-features in Tsai/Zhao’s method of inputting features into the DNN. One would have been motivated to encode the inputted features because: “Such natural language words and/or character sequences correspond to the set of features and are encoded or converted into corresponding feature vectors so that computers can process the corresponding extracted features” (Chan [0027]).
Regarding Claim 4,
The combination of Tsai/Zhao/Chan teaches the method of Claim 3 (and thus the rejection of Claim 3 is incorporated). The combination already discloses to encode the first sub-feature information and the second sub-feature information by concatenating the first sub-feature information and the second sub- feature information (Chan [0028]; “Particular embodiments of the present disclosure model or process these data by sequentially encoding particular feature vectors based on using one or more machine learning models. For example, some embodiments convert each feature vector of a row in a table from left to right in an ordered fashion into another concatenated feature vector. In some embodiments, such sequential encoding includes using a 1-dimensional and/or 2-dimensional bi-directional Long Short Term Memory (LSTM) model to encode sequential data into a concatenated or aggregated feature vector of multiple values representing multiple cells in a table.” wherein encoding is performed through concatenation of sequential feature vectors thus reading on encoding the first and second sub-feature information through concatenation)
Claims 12-13 recite the same method performed by the electronic apparatus of Claims 3-4. Thus, Claims 12-13 are rejected for reasons set forth in the rejection of Claims 3-4, respectively.
Response to Arguments
The Examiner acknowledges the Applicant’s amendments to Claims 1, 3, 10 and 12.
Applicant’s arguments regarding the 35 U.S.C. § 103 rejection of Claims 1-4, 9, 10-13, 18 and 19 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure:
“FusAtNet: Dual Attention based SpectroSpatial Multimodal Fusion Network for Hyperspectral and LiDAR Classification” (as disclosed in the IDS) [2020] which discloses a multimodal fusion network comprising attention correlation computations between modalities
“Neural Network Apparatus, Neural Network Processor, and Method of Operating Neural Network Processor” (US 20180253636 A1) which discloses determination of input feature information and associated weight information of the input features
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN J KIM whose telephone number is (571) 272-0523. The examiner can normally be reached 10-7.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached on (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000
/JONATHAN J KIM/Examiner, Art Unit 2141
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
