DETAILED ACTION
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
1. Regarding the objection to the drawings, Applicant has amended the drawings to address the issues. Accordingly, the objection to the drawings is withdrawn.
2. Regarding the rejection of claims 13-24 under 35 U.S.C. § 101, Applicant's arguments filed 10/16/2025 have been fully considered but they are not persuasive.
Applicant argues on pgs. 9-12 of the Remarks that claims 13-24 are patent eligible. Applicant first argues on pgs. 10-11 that the claims are patent eligible because the claims do not receive any of the abstract idea categories under Step 2A Prong 1 analysis (see pg. 10, 3rd para. of Remarks). Specifically, that while Equations (1)-(12) in amended claim 13 “are formally mathematical calculations (since formulas can express the schemes more clearly than words), but in essence they are practical applications for overcoming unbalanced contribution of each modality to a final solution space and effectively exploring a more complicated multimodal emotion context, thereby improving the ability of the model to distinguish between different emotions” (pg. 11, 4th para. of Remarks). The Examiner respectfully disagrees. A claim that recites a numerical formula or equation will be considered as falling within the mathematical concepts grouping of abstract ideas (see MPEP 2106.04(a)(2)(I.)(B) Mathematical Formulas or Equations). Claim 13 recites several mathematical formulas. Equations (1)-(4) are recited in claim 13 with regards to calculating a weight map based on a multi-head attention score of a modality to obtain a new vector Zm. Equations (5) and (6) are recited in claim 13 with regards to an orthogonality constraint and a guidance vector Z, respectively. Furthermore, Equations (7) and (8) are recited in claim 13 with regards to a post-guidance matrix using the Transformer module. Finally, Equations (10)-(12) are recited in claim 13 with regards to forming an augmented representation, loss function of supervised contrastive learning, and a function for calculating a similarity between samples. Determination as to whether the claims recite practical applications falls under Step 2A Prong 2 analysis. Therefore, the claims recite mathematical concepts which fall under the category of abstract idea under Step 2A Prong 1 (Step 2A Prong 1: YES).
Furthermore, Applicant argues on pgs. 11-12 that the claims integrate the judicial exception into a practical application under Step 2A Prong 2 analysis. Specifically, that recitation of a multimodal emotion classification device, “comprising a processor and a memory having a TokenLearner module and a fully connected layer stored therein” is “a particular machine being able to distinguish between different emotion, thereby improving the performance of an emotion analysis task, rather than a conventional computer,” thus representing “a technical improvement and are integrated into a practical application” (see pg. 12, 1st para. of Remarks). The Examiner respectfully disagrees. The “multimodal emotion classification device” recited in amended claim 13 is recited at a high level of generality, amounting to mere instructions to implement the judicial exception using a generic computer, which cannot integrate the judicial exception into a practical application. Thus, the claims do not integrate the judicial exception into a practical application and are directed to an abstract idea under Step 2A Prong 2 (Step 2A Prong 2: NO, Step 2A: YES). However, the Examiner notes that incorporating language from para. 0082 of the Specification into claim 13, specifically with regards to the following section:
“[0082] …To enable the model to distinguish between various emotions more easily, we introduced supervised contrastive learning to constrain the multimodal representation Hfinal. This strategy introduces label information. In the case of fully utilizing the label information, samples of a same emotion are pushed closer, and samples of different emotions mutually repel. Finally, fused information is input to a linear classification layer, and output information is compared with an emotion category label to obtain a final classification result.”
would help to further prosecution with regards to integrating of the judicial exception into a practical application.
Hence, Applicant’s arguments are not persuasive.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
3. Claims 13-24 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claims contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Independent claim 13 was amended to recite establishing a multimodal emotion classification device comprising a processor and a memory having a TokenLearner module and a fully connected layer stored therein: wherein the multimodal emotion classification device is obtained by steps (1)-(5), acquiring target emotion data of the plurality of modalities, and inputting the target emotion data into the multimodal emotion classification device, to obtain emotion category of the target emotion data. These features were not described in the specification in such a way as to reasonably convey to a person of ordinary skill in the art that the inventor(s) has possession of the claimed invention at the time the application was filed. The specification does not describe establishing a multimodal emotion classification device comprising a processor and a memory having a TokenLearner module and a fully connected layer stored therein: wherein the multimodal emotion classification device is obtained by steps (1)-(5). The specification further does not describe acquiring target emotion data of the plurality of modalities. The specification further does not describe inputting the target emotion data into the multimodal emotion classification device, to obtain emotion category of the target emotion data. These limitations are therefore new matter and are rejected for inadequate written description.
Dependent claims 14-16 inherit the rejection from independent claim 13. Independent claim 17 recites an electronic device with machine-executable instructions to “implement the method according to claim 13”, and thus is also rejected. Dependent claims 18-20 inherit the rejection from independent claim 17. Independent claim 21 recites a machine-readable storage medium which stores machine-executable instructions which “implement the method according to claim 13”, and thus is also rejected. Dependent claims 22-24 inherit the rejection from independent claim 21.
Claim Rejections - 35 USC § 101
4. Claims 13-24 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 13, “A method for multimodal emotion classification” is recited, which is directed to one of the four statutory categories of invention (process). However, the claims limitations, under their broadest reasonable interpretation, recite mathematical concepts which fall into the category of abstract idea.
The following limitations, under their broadest reasonable interpretation, recite mathematical concepts:
step (1), receiving data of a plurality of modalities: preprocessing feature information of the plurality of modalities and extracting primary representations Ht, Ha, and Hv of an audio modality, a video modality, and a text modality: extracting primary representations of each modality amounts to a mathematical calculation which falls under the abstract idea grouping of mathematical concept.
step (2), using the TokenLearner module to obtain a guidance vector: using the TokenLearner module for each modality m E {t, a, v}, wherein t, a, and v represent the text, audio, and video modalities, respectively; the TokenLearner module is used repeated in each guidance; the TokenLearner module is configured to calculate a weight map based on a multi-head attention score of a modality and then obtain a new vector Zm according to the weight map:
PNG
media_image1.png
128
352
media_image1.png
Greyscale
wherein wherein am represents a layer of one-dimensional convolution with a softmax function being added after the convolution; WQ and WlK represent weights of Q and K, respectively; dk represents dimensions of Hm; n represents a number of a plurality of heads; and MultiHead(Q, K) represents the multi-head attention score; head represents an attention score of the ith head; and Attention(Q, K) represents a function for calculating an attention score: the step of obtaining a vector Zm based off of Equations 1-4 amounts to reciting of a mathematical equation, which falls under the abstract idea grouping of mathematical concepts.
to guarantee that information in Zm represents complementary information of a corresponding modality, adding an orthogonality constraint to train the TokenLearner module for each modality, reducing redundant potential representations, and encouraging the TokenLearner modules to encode the plurality of modalities in different aspects; wherein the orthogonality
PNG
media_image2.png
58
320
media_image2.png
Greyscale
constraint is defined as:
wherein ||•||F2 represents square Frobenius norm: the limitation of using an orthogonality constraint defined by Equation 5 amounts to reciting of a mathematical equation, which falls under the abstract idea grouping of mathematical concepts.
calculating a weighted average of Zm to obtain the guidance vector Z by the following formula:
PNG
media_image3.png
34
344
media_image3.png
Greyscale
wherein wm represents a weight: calculating a weighted average to obtain a guidance vector Z amounts to reciting of a mathematical equation, which falls under the abstract idea grouping of mathematical concepts.
step (3), guiding a modality to approach a solution space: concurrently guiding spaces where the three modalities are located to approach the solution space according to the guidance vector Z obtained in step (2), wherein during each guidance, the guidance vector Z is updated in real time based on current states of the spaces where the three modalities are located; and more specifically, for the lth guidance, a post-guidance matrix for each modality is expressed as follows:
PNG
media_image4.png
26
392
media_image4.png
Greyscale
wherein θm represents a model parameter of the Transformer module; [H,, Zi] represents splicing of Hi and Zi; and the guidance of the guidance vector Z for each modality is completed by a Transformer: calculating a post-guidance matrix for each modality via a Transformer amounts to reciting of a mathematical equation, which falls under the abstract idea grouping of mathematical concepts.
expanding the formula (7) to derive:
PNG
media_image5.png
28
506
media_image5.png
Greyscale
wherein MSA represents a multi-head self-attention module; LN represents a layer normalization module; and MLP represents a multilayer perceptron: expanding Equation 7 to derive and use Equation 8 amounts to reciting of a mathematical equation, which falls under the abstract idea grouping of mathematical concepts.
extracting last rows of data in the post-guidance matrices for the three modalities obtained after L rounds of guidance and splicing the last rows of data into a multimodal representation vector Hfinal, wherein L represents a maximum number of rounds of guidance: extracting last rows and splicing the last rows into a multimodal representation vector amounts to reciting a mathematical calculation which falls under the abstract idea grouping of mathematical concept.
step (4), constraining the multimodal representation vector Hfinal to form an augmented representation Ĥfinal, and removing a gradient thereof; and based on a mechanism described above, expanding N samples to obtain 2N samples, expressed as follows:
PNG
media_image6.png
102
364
media_image6.png
Greyscale
Wherein Lscl represents a loss function of supervised contrastive learning; X ∈ ℝ2Nx3d, i ∈ {1,2,…2N} represents an index of any sample in a multi-view batch; τ ∈ R+ represents an adjustable coefficient for control separation of categories; P(i) is a sample set which is different from but has a same category with i, and A(i) represents all indexes other than i; and SIM() represents a function for calculating a similarity between samples: forming an augmented representation amounts to a mathematical calculation, and expanding N samples to obtain 2N samples using Equations 11-12 amounts to reciting a mathematical equation, both of which fall under the abstract idea grouping of mathematical concept.
Step (5), acquiring, by the fully connected layer, a classification result: obtaining a final prediction ŷ for the multimodal representation vector Hfinal by a fully connected layer to realize multimodal emotion classification: obtaining a final prediction via a fully connected layer amounts to a mathematical calculation which falls under the abstract idea grouping of mathematical concept.
Claim 13 does not contain any additional elements which integrate the judicial exception into a practical application. The only additional limitations are “establishing a multimodal emotion classification device comprising a processor and a memory having a TokenLearner module and a fully connected layer stored therein; wherein the multimodal emotion classification device is obtained by steps (1)-(5)”, and “acquiring target emotion data of the plurality of modalities; and inputting the target emotion data into the multimodal emotion classification device, to obtain emotion category of the target emotion data”. These limitations are recited at a high level of generality and amount to mere instructions to implement the judicial exception using a generic computer. Even when viewed in combination, mere instructions to implement the judicial exception using a generic computer do not integrate the judicial exception into a practical application as they do not impose any meaningful limits on practicing the abstract idea. Therefore, claim 13 is directed to an abstract idea.
Claim 13 does not contain any additional elements which amount to significantly more than the judicial exception. As discussed above, the additional limitations amount to mere instructions to implement the judicial exception using a generic computer. Even when viewed in combination, mere instructions to implement the judicial exception using a generic computer do not amount to significantly more than the judicial exception as they cannot provide an inventive concept. Therefore, claim 13 is not patent eligible.
Regarding dependent claims 14-16, “The method” is recited, which is directed to one of the four statutory categories of invention (process). However, the claims limitations, under their broadest reasonable interpretation, recite mathematical concepts which fall into the category of abstract idea.
The following limitations, under their broadest reasonable interpretation, recite mathematical concepts:
Claim 14:
wherein during training, prediction quality during training is estimated using a mean square error loss:
PNG
media_image7.png
30
314
media_image7.png
Greyscale
wherein y represents a true label; and an overall loss Loverall is weighted by a composed of Ltask, Ldiff, and Lscl, expressed as follows:
PNG
media_image8.png
32
380
media_image8.png
Greyscale
wherein Ltask, Ldiff, and Lscl represent a loss function for an emotion classification task, a loss function for an orthogonality constraint, and a loss function for supervised contrastive learning respectively; and α, β, and γ represents weights of Ltask, Ldiff, and Lscl, respectively: training using the task loss defined in Equation 13 and utilizing the overall loss defined in Equation 14 amounts to reciting a mathematical equation which falls under the abstract idea grouping of mathematical concept.
Claim 15:
wherein a Bidirectional Encoder Representations from Transformers (BERT) model is employed for preprocessing the text modality in step (1): using a BERT model for preprocessing falls under the abstract idea grouping of mathematical concept.
Claim 16:
wherein a Transformer model is employed for preprocessing the audio modality and the video modality in step (1): using a Transformer model preprocessing falls under the abstract idea grouping of mathematical concept.
Claims 14-16 do not contain any additional elements which integrate the judicial exception into a practical application. There are no additional limitations in claims 14-16 which do not fall under the category of mathematical concept. Therefore, claims 14-16 are directed to an abstract idea.
Claims 14-16 do not contain any additional elements which amount to significantly more than the judicial exception. As discussed above, there are no additional limitations in claims 14-16 which do not fall under the category of mathematical concept. Therefore, claims 14-16 are not patent eligible.
Regarding claim 17, “An electronic device” is recited, which is directed to one of the four statutory categories of invention (machine). However, the claims limitations, under their broadest reasonable interpretation, recite mathematical concepts which fall into the category of abstract idea.
The following limitations, under their broadest reasonable interpretation, recite mathematical concepts:
to implement the method according to claim 13: see above explanation with regards to claim 13 and mathematical concepts.
Claim 17 does not contain any additional limitation which integrate the judicial exception into a practical application. The only additional limitations are “An electronic device, comprising a processor and a memory, wherein the memory stores machine-executable instructions capable of being executed by the processor, and the processor is configured to execute the machine-executable instructions”. These limitations are recited broadly, and amount to mere instructions to implement the judicial exception using a generic computer. Mere instructions to implement the judicial exception using a generic computer do not integrate the judicial exception into a practical application. Therefore, claim 17 is directed to an abstract idea.
Claim 17 does not contain any additional limitations which amount to significantly more than the judicial exception. As discussed above, the additional limitations amount to mere instructions to implement the judicial exception using a generic computer. mere instructions to implement the judicial exception using a generic computer do not amount to significantly more than the judicial exception. Therefore, claim 17 is not patent eligible.
Regarding dependent claims 18-20, “The electronic device” is recited, which is directed to one of the four statutory categories of invention (machine). However, the claims limitations, under their broadest reasonable interpretation, recite mathematical concepts which fall into the category of abstract idea.
The following limitations, under their broadest reasonable interpretation, recite mathematical concepts:
Claim 18:
wherein during training, prediction quality during training is estimated using a mean square error loss:
PNG
media_image7.png
30
314
media_image7.png
Greyscale
wherein y represents a true label; and an overall loss Loverall is weighted by a composed of Ltask, Ldiff, and Lscl, expressed as follows:
PNG
media_image8.png
32
380
media_image8.png
Greyscale
wherein Ltask, Ldiff, and Lscl represent a loss function for an emotion classification task, a loss function for an orthogonality constraint, and a loss function for supervised contrastive learning respectively; and α, β, and γ represents weights of Ltask, Ldiff, and Lscl, respectively: training using the task loss defined in Equation 13 and utilizing the overall loss defined in Equation 14 amounts to reciting a mathematical equation which falls under the abstract idea grouping of mathematical concept.
Claim 19:
wherein a Bidirectional Encoder Representations from Transformers (BERT) model is employed for preprocessing the text modality in step (1): using a BERT model for preprocessing falls under the abstract idea grouping of mathematical concept.
Claim 20:
wherein a Transformer model is employed for preprocessing the audio modality and the video modality in step (1): using a Transformer model preprocessing falls under the abstract idea grouping of mathematical concept.
Claims 18-20 do not contain any additional elements which integrate the judicial exception into a practical application. There are no additional limitations in claims 18-20 which do not fall under the category of mathematical concept. Therefore, claims 18-20 are directed to an abstract idea.
Claims 18-20 do not contain any additional elements which amount to significantly more than the judicial exception. As discussed above, there are no additional limitations in claims 18-20 which do not fall under the category of mathematical concept. Therefore, claims 18-20 are not patent eligible.
Regarding claim 21, the claims limitations, under their broadest reasonable interpretation, recite mathematical concepts which fall into the category of abstract idea.
The following limitations, under their broadest reasonable interpretation, recite mathematical concepts:
to implement the method according to claim 13: see above explanation with regards to claim 13 and mathematical concepts.
Claim 21 does not contain any additional limitation which integrate the judicial exception into a practical application. The only additional limitations are “A machine-readable storage medium, storing machine-executable instructions which, when called and executed by a processor, cause the processor to”. These limitations are recited broadly, and amount to mere instructions to implement the judicial exception using a generic computer. Mere instructions to implement the judicial exception using a generic computer do not integrate the judicial exception into a practical application. Therefore, claim 21 is directed to an abstract idea.
Claim 21 does not contain any additional limitations which amount to significantly more than the judicial exception. As discussed above, the additional limitations amount to mere instructions to implement the judicial exception using a generic computer. mere instructions to implement the judicial exception using a generic computer do not amount to significantly more than the judicial exception. Therefore, claim 21 is not patent eligible.
Regarding dependent claims 22-24, the claims limitations, under their broadest reasonable interpretation, recite mathematical concepts which fall into the category of abstract idea.
The following limitations, under their broadest reasonable interpretation, recite mathematical concepts:
Claim 22:
wherein during training, prediction quality during training is estimated using a mean square error loss:
PNG
media_image7.png
30
314
media_image7.png
Greyscale
wherein y represents a true label; and an overall loss Loverall is weighted by a composed of Ltask, Ldiff, and Lscl, expressed as follows:
PNG
media_image8.png
32
380
media_image8.png
Greyscale
wherein Ltask, Ldiff, and Lscl represent a loss function for an emotion classification task, a loss function for an orthogonality constraint, and a loss function for supervised contrastive learning respectively; and α, β, and γ represents weights of Ltask, Ldiff, and Lscl, respectively: training using the task loss defined in Equation 13 and utilizing the overall loss defined in Equation 14 amounts to reciting a mathematical equation which falls under the abstract idea grouping of mathematical concept.
Claim 23:
wherein a Bidirectional Encoder Representations from Transformers (BERT) model is employed for preprocessing the text modality in step (1): using a BERT model for preprocessing falls under the abstract idea grouping of mathematical concept.
Claim 24:
wherein a Transformer model is employed for preprocessing the audio modality and the video modality in step (1): using a Transformer model preprocessing falls under the abstract idea grouping of mathematical concept.
Claims 22-24 do not contain any additional elements which integrate the judicial exception into a practical application. There are no additional limitations in claims 22-24 which do not fall under the category of mathematical concept. Therefore, claims 22-24 are directed to an abstract idea.
Claims 22-24 do not contain any additional elements which amount to significantly more than the judicial exception. As discussed above, there are no additional limitations in claims 22-24 which do not fall under the category of mathematical concept. Therefore, claims 22-24 are not patent eligible.
5. Claims 21-24 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claims do not fall within at least one of the four categories of patent eligible subject matter because it recites a “machine-readable storage medium”. The applicant’s specification does not provide a special definition for “machine-readable storage medium”; thus, using its plain meaning, the term includes data signals per se as one potential form of the media. Data signals per se do not fall into one of the four statutory categories of invention. As such, they are non-statutory subject matter. In contrast, a claimed “non-transitory machine-readable storage medium” excludes data signals from its scope and does fall into one of the four statutory categories of invention.
Allowable Subject Matter
6. Claims 13-24 would be allowable if rewritten or amended to overcome the rejections under 35 U.S.C. § 101 and 35 U.S.C. § 112(a).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Kim et al. (US 2025/0201267 A1): multimodal emotion recognition (Fig. 3)
Kollada & Banerjee (US 2022/0392637 A1): multimodal mental health diagnosis (Fig. 5C)
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CODY DOUGLAS HUTCHESON whose telephone number is (703)756-1601. The examiner can normally be reached M-F 8:00AM-5:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at (571)-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CODY DOUGLAS HUTCHESON/Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659