DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on February 26, 2026, has been entered.
Response to Arguments
Applicant’s arguments, filed February 26, 2026, with respect to claims 1, 3 – 9, 11, 13 – 19 and 21 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 21 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being incomplete for omitting essential steps, such omission amounting to a gap between the steps. See MPEP § 2172.01.
The omitted steps are: a step to train the speech emotion recognition (SER) model.
Claim 21 recites, “a method for training a speech emotion recognition (SER) model for use with a specific application or product”. However, claim 21 does not include any limitations that contain a reference to a speech emotion recognition model.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 7 – 9, 11, 13 and 17 – 19 are rejected under 35 U.S.C. 103 as being unpatentable over Plaza-del-Arco et al. ("Natural Language Inference Prompts for Zero-shot Emotion Classification in Text across Corpora"), hereinafter Plaza-del-Arco, in view of Compton et al. (US Patent Application Publication No. 2024/0028838), hereinafter Compton, Mohammad et al. ("SemEval-2018 Task 1: Affect in Tweets"), hereinafter Mohammad, and Deng et al. ("Semisupervised Autoencoders for Speech Emotion Recognition"), hereinafter Deng.
Regarding claim 1, Plaza-del-Arco discloses a method for training a speech emotion recognition (SER) model for use with a specific application or product, comprising:
applying, by the one or more processors, the text transcript to a pre-trained language model (Section 3, lines 1-4, "In this section, we explain how we apply NLI for ZSL emotion classification and propose a collection of prompts to contextualize and represent the emotion concept in different corpora."; Section 3.3, line 6, "The ensemble model takes as input a text x"; Section 4.1.2, lines 10-17, "For our ZSL experiments, we explore three state-of-the-art pretrained NLI models publicly available within the Hugging Face Transformers Python library (Wolf et al., 2020), and fine-tuned on the MultiNLI dataset (Williams et al., 2018). Concretely, we choose RoBERTa, BART and DeBERTa as they cover different architectures and represent competitive approaches across a set of NLP tasks."; Section 4.1.2, lines 44-47, "All experiments are performed on a node equipped with two Intel Xeon Silver 4208 CPU at 2.10GHz, 192GB RAM, as main processors, and six GPUs NVIDIA GeForce RTX 2080Ti (with 11GB each)."; Input text reads on a text transcript, and pretrained natural language inference (NLI) models read on a pre-trained language model.);
generating, using the pre-trained language model according to an engineered prompt and a particular predetermined taxonomy, a textual entailment from the text transcript (Section 3.1, lines 1-14, "The NLI task is commonly defined as a sentence-pair classification in which two sentences are given: a premise s1 and a hypothesis s2. The task is to learn a function
f
N
L
I
(s1, s2) → {E,C,N}, in which E expresses the entailment of s1 and s2, C denotes a contradiction and N is a neutral output. We treat ZSL emotion classification as a textual entailment problem, but represent each label under consideration with multiple prompts, in contrast to Yin et al. (2019). Given a sentence to be classified x (premise) and an emotion e, we have a function g(e) that generates a set of prompts (hypothesis) out of the class e ∈ E (with E being the set of emotions under consideration).”; Section 3.2.1, lines 1-8, "We generate a set of prompts with the function g(e) = c + r(e), in which c represents what we call the context and r(e) represents a set of emotion representations.1 As c, we use either an empty string ϵ, the text “This text expresses”, “This person feels”, or “This person expresses”, motivated by our choice of the language register presented in the datasets used in our experiments (see § 4)."; Section 3.2.2, lines 1-9, "Each prompt in this paper consists of context and the emotion representation. There are three prompts which have in common the emotion name representation, namely Emo-Name, Expr-Emo, and Feels-Emo. Variations of these prompts are Emo-S, Expr-S, and Feels-S, where the emotion name representation is replaced by multiple emotion synonyms and EmoLex where the emotion name is replaced by entries from an emotion word lexicon."; Section 3.3, line 6, "The ensemble model takes as input a text x"; Section 4.1.2, lines 10-17, "For our ZSL experiments, we explore three state-of-the-art pretrained NLI models publicly available within the Hugging Face Transformers Python library (Wolf et al., 2020), and fine-tuned on the MultiNLI dataset (Williams et al., 2018). Concretely, we choose RoBERTa, BART and DeBERTa as they cover different architectures and represent competitive approaches across a set of NLP tasks."; Pretrained natural language inference (NLI) models read on a pre-trained language model, the entailment of premise s1 and hypothesis s2 reads on a textual entailment from the text transcript, generating a set of prompts reads on an engineered prompt, and an emotion word lexicon reads on a particular predetermined taxonomy.);
and generating, by the one or more processors using the textual entailment, a predicted emotion corresponding to the input speech (Section 3.1, lines 10-25, "Given a sentence to be classified x (premise) and an emotion e, we have a function g(e) that generates a set of prompts (hypothesis) out of the class e ∈ E (with E being the set of emotions under consideration). Under the assumption of an NLI model m, which calculates the entailment probability
p
m
(γ, x) for some emotion representation γ ∈ g(e), we assign the average entailment probability across all emotion representations as
p
m
-
g
(
e
,
x
)
=
1
|
g
e
|
∑
γ
∈
g
(
e
)
p
m
(
γ
,
x
)
for a particular prompt generation method g. The classification decision
e
x
g
(
e
,
x
)
=
a
r
g
m
a
x
e
∈
E
p
m
-
g
(
e
,
x
)
returns the emotion corresponding to the maximum entailment probability."; Returning the emotion corresponding to the maximum entailment probability reads on generating a predicted emotion corresponding to the input speech using the textual entailment.).
Plaza-del-Arco does not specifically disclose: generating, by one or more processors, a text transcript for a snippet of input speech.
Compton teaches:
generating, by one or more processors, a text transcript for a snippet of input speech (Paragraph 0175, lines 1-5, "FIG. 9 is a flowchart illustrating a routine 900 that may be executed by the system 500. In block 902, at least one processor receives a digital speech signal. In block 904, the at least one processor converts the digital speech signal to text.").
Compton is considered to be analogous to the claimed invention because it is in the same field of predicting emotion of text. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Plaza-del-Arco to incorporate the teachings of Compton to convert a speech signal to text. Doing so would allow an automated dialogue system to react appropriately to patient provided information during medical history taking (Compton; Paragraph 0031, lines 1-15).
Plaza-del-Arco in view of Compton does not specifically disclose: wherein the predetermined taxonomy (i) is chosen based on the specific application or product with which the SER model is used and (ii) includes a set of words or phrases corresponding to the specific application or product.
Mohammad teaches:
wherein the predetermined taxonomy (i) is chosen based on the specific application or product with which the SER model is used and (ii) includes a set of words or phrases corresponding to the specific application or product (Section 1, lines 24-39, "Natural language applications in commerce, public health, disaster management, and public policy can benefit from knowing the affectual states of people—both the categories and the intensities of the emotions they feel. We thus present the SemEval-2018 Task 1: Affect in Tweets, which includes an array of subtasks where automatic systems have to infer the affectual state of a person from their tweet. We will refer to the author of a tweet as the tweeter. Some of the tasks are on the intensities of four basic emotions common to many proposals of basic emotions: anger, fear, joy, and sadness. Some of the tasks are on valence or sentiment intensity. Finally, we include an emotion classification task over eleven emotions commonly expressed in tweets."; Section 3.1.1, lines 15-17, " We polled the Twitter API, over the span of two months (June and July, 2017), for tweets that included the query terms."; Performing an emotion classification task over eleven emotions commonly expressed in tweets reads on the predetermined taxonomy being chosen based on a specific application for which a speech emotion recognition model is used and the predetermined taxonomy including a set of words or phrases corresponding to the specific application.).
Mohammad is considered to be analogous to the claimed invention because it is in the same field of predicting emotion of text. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Plaza-del-Arco in view of Compton to incorporate the teachings of Mohammad to perform an emotion classification task over eleven emotions commonly expressed in tweets. Doing so would allow for automatically inferring the affectual state of a person from their tweet (Mohammad; Section 1, lines 24-39).
Plaza-del-Arco in view of Compton and Mohammad does not specifically disclose: wherein the predicted emotion is applied as a weak label to train the SER model on the input speech for weakly-supervised learning of the SER model.
Deng teaches:
wherein the predicted emotion is applied as a weak label to train the SER model on the input speech for weakly-supervised learning of the SER model (Page 32, right column, lines 3-5, "Encouraged by the recent success of deep semi-supervised learning, we propose semi-supervised autoencoders for speech emotion recognition."; Page 33, left column, lines 30-34, "In this paper, we exhibit, for the first time ever, our proposed semi-supervised learning method for speech emotion recognition, which can reach state-of-the-art accuracy with only a few labelled examples."; Semi-supervised learning of autoencoders for speech emotion recognition using labelled examples reads on the predicted emotion being applied as a weak label to train a speech emotion recognition (SER) model for weakly-supervised learning of the SER model.).
Deng is considered to be analogous to the claimed invention because it is in the same field of predicting emotion of text. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Plaza-del-Arco in view of Compton and Mohammad to incorporate the teachings of Deng to perform semi-supervised learning of autoencoders for speech emotion recognition using labelled examples. Doing so would allow for reducing the dependence on a great quantity of labelled training examples (Deng; Page 32, right column, lines 3-14).
Regarding claim 3, Plaza-del-Arco in view of Compton, Mohammad, and Deng discloses the method as claimed in claim 1.
Deng further teaches:
wherein training the SER model includes: generating an SER predicted emotion using the SER model (Page 32, right column, lines 3-5, "Encouraged by the recent success of deep semi-supervised learning, we propose semi-supervised autoencoders for speech emotion recognition."; Page 33, left column, lines 30-34, "In this paper, we exhibit, for the first time ever, our proposed semi-supervised learning method for speech emotion recognition, which can reach state-of-the-art accuracy with only a few labelled examples."; Autoencoders for speech emotion recognition reads on generating an SER predicted emotion using the SER model.);
and comparing the weak label against the SER predicted emotion (Section III, lines 13-16, "In this end, the objective function is a weighted sum of the supervised cross entropy loss and the unsupervised mean square error loss for the labelled and unlabelled data."; The objective function including a loss for the labeled data reads on comparing the weak label against the SER predicted emotion.).
Deng is considered to be analogous to the claimed invention because it is in the same field of predicting emotion of text. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Plaza-del-Arco in view of Compton, Mohammad, and Deng to further incorporate the teachings of Deng to perform semi-supervised learning of autoencoders for speech emotion recognition using labelled examples, where the objective function including a loss for the labeled data. Doing so would allow for reducing the dependence on a great quantity of labelled training examples (Deng; Page 32, right column, lines 3-14).
Regarding claim 7, Plaza-del-Arco in view of Compton, Mohammad, and Deng discloses the method as claimed in claim 1.
Plaza-del-Arco further discloses:
wherein at least one of the pre-trained language model or the SER model has a transformer architecture (Section 4.1.2, lines 10-17, "For our ZSL experiments, we explore three state-of-the-art pretrained NLI models publicly available within the Hugging Face Transformers Python library (Wolf et al., 2020), and fine-tuned on the MultiNLI dataset (Williams et al., 2018). Concretely, we choose RoBERTa, BART and DeBERTa as they cover different architectures and represent competitive approaches across a set of NLP tasks."; Section 4.1.2, lines 18-27, "RoBERTa. The Robustly Optimized BERT Pretraining Approach (Liu et al., 2020) is a modified version of BERT which includes some changes such as the removal of the next-sentence prediction task, the replacement of the WordPiece tokenization with a variation of the byte-pair encoding, and the replacement of the static masking (the same input masks are fed to the model on each epoch) with dynamic masking (the masking is generated every time the sequence is fed to the model)."; A pre-trained language model based on a Bidirectional Encoder Representations from Transformers (BERT) model reads on the pre-trained language model having a transformer architecture.).
Regarding claim 8, Plaza-del-Arco in view of Compton, Mohammad, and Deng discloses the method as claimed in claim 1.
Plaza-del-Arco further discloses:
wherein the pre-trained language model is trained via token masking (Section 4.1.2, lines 10-17, "For our ZSL experiments, we explore three state-of-the-art pretrained NLI models publicly available within the Hugging Face Transformers Python library (Wolf et al., 2020), and fine-tuned on the MultiNLI dataset (Williams et al., 2018). Concretely, we choose RoBERTa, BART and DeBERTa as they cover different architectures and represent competitive approaches across a set of NLP tasks."; Section 4.1.2, lines 18-27, "RoBERTa. The Robustly Optimized BERT Pretraining Approach (Liu et al., 2020) is a modified version of BERT which includes some changes such as the removal of the next-sentence prediction task, the replacement of the WordPiece tokenization with a variation of the byte-pair encoding, and the replacement of the static masking (the same input masks are fed to the model on each epoch) with dynamic masking (the masking is generated every time the sequence is fed to the model).").
Regarding claim 9, Plaza-del-Arco in view of Compton, Mohammad, and Deng discloses the method as claimed in claim 1.
Plaza-del-Arco further discloses:
wherein the pre-trained language model is constrained to output a set of words that correspond to emotion perception (Section 3.1, lines 10-25, "Given a sentence to be classified x (premise) and an emotion e, we have a function g(e) that generates a set of prompts (hypothesis) out of the class e ∈ E (with E being the set of emotions under consideration). Under the assumption of an NLI model m, which calculates the entailment probability
p
m
(γ, x) for some emotion representation γ ∈ g(e), we assign the average entailment probability across all emotion representations as
p
m
-
g
(
e
,
x
)
=
1
|
g
e
|
∑
γ
∈
g
(
e
)
p
m
(
γ
,
x
)
for a particular prompt generation method g. The classification decision
e
x
g
(
e
,
x
)
=
a
r
g
m
a
x
e
∈
E
p
m
-
g
(
e
,
x
)
returns the emotion corresponding to the maximum entailment probability."; Section 3.2.2, lines 1-9, "Each prompt in this paper consists of context and the emotion representation. There are three prompts which have in common the emotion name representation, namely Emo-Name, Expr-Emo, and Feels-Emo. Variations of these prompts are Emo-S, Expr-S, and Feels-S, where the emotion name representation is replaced by multiple emotion synonyms and EmoLex where the emotion name is replaced by entries from an emotion word lexicon."; Returning the emotion corresponding to the maximum entailment probability, where an emotion is in a set of emotions under consideration, reads on constraining the pre-trained language model to output a set of words that correspond to emotion perception.).
Regarding claim 11, Plaza-del-Arco discloses a system for training a speech emotion recognition (SER) model for use with a specific application or product, comprising:
memory configured to store one or more language models; and one or more processors operatively coupled to the memory (Section 4.1.2, lines 44-47, "All experiments are performed on a node equipped with two Intel Xeon Silver 4208 CPU at 2.10GHz, 192GB RAM, as main processors, and six GPUs NVIDIA GeForce RTX 2080Ti (with 11GB each)."; Performing experiments on processors demonstrates a processor executing instructions from memory.), the one or more processors being configure to:
apply the text transcript to a pre-trained language model (Section 3, lines 1-4, "In this section, we explain how we apply NLI for ZSL emotion classification and propose a collection of prompts to contextualize and represent the emotion concept in different corpora."; Section 3.3, line 6, "The ensemble model takes as input a text x"; Section 4.1.2, lines 10-17, "For our ZSL experiments, we explore three state-of-the-art pretrained NLI models publicly available within the Hugging Face Transformers Python library (Wolf et al., 2020), and fine-tuned on the MultiNLI dataset (Williams et al., 2018). Concretely, we choose RoBERTa, BART and DeBERTa as they cover different architectures and represent competitive approaches across a set of NLP tasks."; Section 4.1.2, lines 44-47, "All experiments are performed on a node equipped with two Intel Xeon Silver 4208 CPU at 2.10GHz, 192GB RAM, as main processors, and six GPUs NVIDIA GeForce RTX 2080Ti (with 11GB each)."; Input text reads on a text transcript, and pretrained natural language inference (NLI) models read on a pre-trained language model.);
generate, using the pre-trained language model according to an engineered prompt and a particular predetermined taxonomy, a textual entailment from the text transcript (Section 3.1, lines 1-14, "The NLI task is commonly defined as a sentence-pair classification in which two sentences are given: a premise s1 and a hypothesis s2. The task is to learn a function
f
N
L
I
(s1, s2) → {E,C,N}, in which E expresses the entailment of s1 and s2, C denotes a contradiction and N is a neutral output. We treat ZSL emotion classification as a textual entailment problem, but represent each label under consideration with multiple prompts, in contrast to Yin et al. (2019). Given a sentence to be classified x (premise) and an emotion e, we have a function g(e) that generates a set of prompts (hypothesis) out of the class e ∈ E (with E being the set of emotions under consideration).”; Section 3.2.1, lines 1-8, "We generate a set of prompts with the function g(e) = c + r(e), in which c represents what we call the context and r(e) represents a set of emotion representations.1 As c, we use either an empty string ϵ, the text “This text expresses”, “This person feels”, or “This person expresses”, motivated by our choice of the language register presented in the datasets used in our experiments (see § 4)."; Section 3.2.2, lines 1-9, "Each prompt in this paper consists of context and the emotion representation. There are three prompts which have in common the emotion name representation, namely Emo-Name, Expr-Emo, and Feels-Emo. Variations of these prompts are Emo-S, Expr-S, and Feels-S, where the emotion name representation is replaced by multiple emotion synonyms and EmoLex where the emotion name is replaced by entries from an emotion word lexicon."; Section 3.3, line 6, "The ensemble model takes as input a text x"; Section 4.1.2, lines 10-17, "For our ZSL experiments, we explore three state-of-the-art pretrained NLI models publicly available within the Hugging Face Transformers Python library (Wolf et al., 2020), and fine-tuned on the MultiNLI dataset (Williams et al., 2018). Concretely, we choose RoBERTa, BART and DeBERTa as they cover different architectures and represent competitive approaches across a set of NLP tasks."; Pretrained natural language inference (NLI) models read on a pre-trained language model, the entailment of premise s1 and hypothesis s2 reads on a textual entailment from the text transcript, generating a set of prompts reads on an engineered prompt, and an emotion word lexicon reads on a particular predetermined taxonomy.);
generate, using the textual entailment, a predicted emotion corresponding to the input speech (Section 3.1, lines 10-25, "Given a sentence to be classified x (premise) and an emotion e, we have a function g(e) that generates a set of prompts (hypothesis) out of the class e ∈ E (with E being the set of emotions under consideration). Under the assumption of an NLI model m, which calculates the entailment probability
p
m
(γ, x) for some emotion representation γ ∈ g(e), we assign the average entailment probability across all emotion representations as
p
m
-
g
(
e
,
x
)
=
1
|
g
e
|
∑
γ
∈
g
(
e
)
p
m
(
γ
,
x
)
for a particular prompt generation method g. The classification decision
e
x
g
(
e
,
x
)
=
a
r
g
m
a
x
e
∈
E
p
m
-
g
(
e
,
x
)
returns the emotion corresponding to the maximum entailment probability."; Returning the emotion corresponding to the maximum entailment probability reads on generating a predicted emotion corresponding to the input speech using the textual entailment.);
and store the SER model in the memory (Section 4.1.2, lines 44-47, "All experiments are performed on a node equipped with two Intel Xeon Silver 4208 CPU at 2.10GHz, 192GB RAM, as main processors, and six GPUs NVIDIA GeForce RTX 2080Ti (with 11GB each)."; Performing experiments on processors demonstrates a processor executing instructions from memory and storing results in memory.).
Plaza-del-Arco does not specifically disclose: generate a text transcript for a snippet of input speech.
Compton teaches:
generate a text transcript for a snippet of input speech (Paragraph 0175, lines 1-5, "FIG. 9 is a flowchart illustrating a routine 900 that may be executed by the system 500. In block 902, at least one processor receives a digital speech signal. In block 904, the at least one processor converts the digital speech signal to text.").
Compton is considered to be analogous to the claimed invention because it is in the same field of predicting emotion of text. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Plaza-del-Arco to incorporate the teachings of Compton to convert a speech signal to text. Doing so would allow an automated dialogue system to react appropriately to patient provided information during medical history taking (Compton; Paragraph 0031, lines 1-15).
Plaza-del-Arco in view of Compton does not specifically disclose: wherein the particular predetermined taxonomy (i) is chosen based on the specific application or product with which the SER model is used and (ii) includes a set of words or phrases corresponding to the specific application or product.
Mohammad teaches:
wherein the particular predetermined taxonomy (i) is chosen based on the specific application or product with which the SER model is used and (ii) includes a set of words or phrases corresponding to the specific application or product (Section 1, lines 24-39, "Natural language applications in commerce, public health, disaster management, and public policy can benefit from knowing the affectual states of people—both the categories and the intensities of the emotions they feel. We thus present the SemEval-2018 Task 1: Affect in Tweets, which includes an array of subtasks where automatic systems have to infer the affectual state of a person from their tweet. We will refer to the author of a tweet as the tweeter. Some of the tasks are on the intensities of four basic emotions common to many proposals of basic emotions: anger, fear, joy, and sadness. Some of the tasks are on valence or sentiment intensity. Finally, we include an emotion classification task over eleven emotions commonly expressed in tweets."; Section 3.1.1, lines 15-17, " We polled the Twitter API, over the span of two months (June and July, 2017), for tweets that included the query terms."; Performing an emotion classification task over eleven emotions commonly expressed in tweets reads on the predetermined taxonomy being chosen based on a specific application for which a speech emotion recognition model is used and the predetermined taxonomy including a set of words or phrases corresponding to the specific application.).
Mohammad is considered to be analogous to the claimed invention because it is in the same field of predicting emotion of text. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Plaza-del-Arco in view of Compton to incorporate the teachings of Mohammad to perform an emotion classification task over eleven emotions commonly expressed in tweets. Doing so would allow for automatically inferring the affectual state of a person from their tweet (Mohammad; Section 1, lines 24-39).
Plaza-del-Arco in view of Compton and Mohammad does not specifically disclose: apply the predicted emotion as a weak label to train the SER model on the input speech for weakly-supervised learning of the SER model.
Deng teaches:
apply the predicted emotion as a weak label to train the SER model on the input speech for weakly-supervised learning of the SER model (Page 32, right column, lines 3-5, "Encouraged by the recent success of deep semi-supervised learning, we propose semi-supervised autoencoders for speech emotion recognition."; Page 33, left column, lines 30-34, "In this paper, we exhibit, for the first time ever, our proposed semi-supervised learning method for speech emotion recognition, which can reach state-of-the-art accuracy with only a few labelled examples."; Semi-supervised learning of autoencoders for speech emotion recognition using labelled examples reads on the predicted emotion being applied as a weak label to train a speech emotion recognition (SER) model for weakly-supervised learning of the SER model.).
Deng is considered to be analogous to the claimed invention because it is in the same field of predicting emotion of text. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Plaza-del-Arco in view of Compton and Mohammad to incorporate the teachings of Deng to perform semi-supervised learning of autoencoders for speech emotion recognition using labelled examples. Doing so would allow for reducing the dependence on a great quantity of labelled training examples (Deng; Page 32, right column, lines 3-14).
Regarding claim 13, arguments analogous to claim 3 are applicable.
Regarding claim 17, arguments analogous to claim 7 are applicable.
Regarding claim 18, arguments analogous to claim 8 are applicable.
Regarding claim 19, arguments analogous to claim 9 are applicable.
Claims 4 – 6 and 14 – 16 are rejected under 35 U.S.C. 103 as being unpatentable over Plaza-del-Arco in view of Compton, Mohammad, and Deng, and further in view of Bhaumik et al. ("Adapting Emotion Detection to Analyze Influence Campaigns on Social Media"), hereinafter Bhaumik.
Regarding claim 4, Plaza-del-Arco in view of Compton, Mohammad, and Deng discloses the method as claimed in claim 3, but does not specifically disclose: wherein the comparing includes evaluating a probability for each emotion in the particular predetermined taxonomy.
Bhaumik teaches:
wherein the comparing includes evaluating a probability for each emotion in the predetermined taxonomy (Section 3.3, lines 11-13, "We use this dataset to train a model to predict scores over the six Ekman emotions"; Section 3.4, lines 12-22, "The purpose of the linear layer is to convert the final hidden state vector into a vector related to the distinct emotion labels in the corresponding dataset. Subsequently, this vector can be converted into probabilities via the Softmax function. The labels of each model are listed in Table 2. In the first step, two models are fine-tuned to output normalized scores on the six Ekman emotions using the CBET Twitter data and GoEmotions Reddit data."; Training a model to predict scores over the six Ekman emotions reads on comparing labels to predicted emotions, and converting a final hidden state vector into a vector related to distinct emotion labels from a corresponding dataset, where the vector is converted into probabilities via the Softmax function and the models output normalized scores on the six Ekman emotions, reads on evaluating a probability for each emotion in the predetermined taxonomy, where emotion labels from a corresponding dataset read on a particular predetermined taxonomy.).
Bhaumik is considered to be analogous to the claimed invention because it is in the same field of predicting emotion of text. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Plaza-del-Arco in view of Compton, Mohammad, and Deng to incorporate the teachings of Bhaumik to convert a final hidden state vector into a vector related to distinct emotion labels from a corresponding dataset, where the vector is converted into probabilities via the Softmax function and the models output normalized scores on the six Ekman emotions. Doing so would allow for implementing emotion detection models that can be adapted to specialized emotion labels (Bhaumik; Section 6, lines 1-8).
Regarding claim 5, Plaza-del-Arco in view of Compton, Mohammad, and Deng, and further in view of Bhaumik, discloses the method as claimed in claim 4.
Bhaumik further teaches:
wherein the evaluating is performed according to a cross-entropy loss function (Appendix A, lines 1-9, "To fine-tune the pretrained twitter-RoBERTa-base-emotion models on each of the six training and validation datasets, we use the following settings, chosen in order to stay close to the pretrained weights and also alleviate overfitting to the target domains. We use a binary cross-entropy loss for the task of multi-label classification, an Adam optimizer, an initial learning rate of 1e-6, and a batch size of 16.").
Bhaumik is considered to be analogous to the claimed invention because it is in the same field of predicting emotion of text. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Plaza-del-Arco in view of Compton, Mohammad, and Deng and further in view of Bhaumik to further incorporate the teachings of Bhaumik to use a cross-entropy loss for multi-label classification training. Doing so would allow for implementing emotion detection models that can be adapted to specialized emotion labels (Bhaumik; Section 6, lines 1-8).
Regarding claim 6, Plaza-del-Arco in view of Compton, Mohammad, and Deng discloses the method as claimed in claim 3, but does not specifically disclose: further comprising fine-tuning the SER model by evaluating the SER predicted emotion to one or more ground truth labels.
Bhaumik teaches:
further comprising fine-tuning the SER model by evaluating the SER predicted emotion to one or more ground truth labels (Section 3.3, lines 11-13, "We use this dataset to train a model to predict scores over the six Ekman emotions"; Section 3.6, lines 14-18, "We use a subset of the manually annotated French election dataset to fine-tune both the mapping weights and the classification thresholds by first optimizing the weights, and subsequently choosing the thresholds for each label."; Section 3.7, lines 1-6, "Our annotation team utilized the emotion label set E, as detailed in Section 3, to annotate a subset of the 2017 French Presidential Election dataset. Three raters independently assigned one or more emotions to each tweet, with a label considered ground truth if confirmed by at least two annotators.").
Bhaumik is considered to be analogous to the claimed invention because it is in the same field of predicting emotion of text. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Plaza-del-Arco in view of Compton, Mohammad, and Deng to incorporate the teachings of Bhaumik to use a dataset annotated with ground truth labels to fine-tune an emotion prediction model. Doing so would allow for implementing emotion detection models that can be adapted to specialized emotion labels (Bhaumik; Section 6, lines 1-8).
Regarding claim 14, arguments analogous to claim 4 are applicable.
Regarding claim 15, arguments analogous to claim 5 are applicable.
Regarding claim 16, arguments analogous to claim 6 are applicable.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Plaza-del-Arco in view of Compton and Mohammad.
Regarding claim 21, Plaza-del-Arco discloses a method for training a speech emotion recognition (SER) model for use with a specific application or product, comprising:
applying, by the one or more processors, the text transcript to a pre-trained language model that is trained via token masking (Section 3, lines 1-4, "In this section, we explain how we apply NLI for ZSL emotion classification and propose a collection of prompts to contextualize and represent the emotion concept in different corpora."; Section 3.3, line 6, "The ensemble model takes as input a text x"; Section 4.1.2, lines 10-17, "For our ZSL experiments, we explore three state-of-the-art pretrained NLI models publicly available within the Hugging Face Transformers Python library (Wolf et al., 2020), and fine-tuned on the MultiNLI dataset (Williams et al., 2018). Concretely, we choose RoBERTa, BART and DeBERTa as they cover different architectures and represent competitive approaches across a set of NLP tasks."; Section 4.1.2, lines 18-27, "RoBERTa. The Robustly Optimized BERT Pretraining Approach (Liu et al., 2020) is a modified version of BERT which includes some changes such as the removal of the next-sentence prediction task, the replacement of the WordPiece tokenization with a variation of the byte-pair encoding, and the replacement of the static masking (the same input masks are fed to the model on each epoch) with dynamic masking (the masking is generated every time the sequence is fed to the model)."; Section 4.1.2, lines 44-47, "All experiments are performed on a node equipped with two Intel Xeon Silver 4208 CPU at 2.10GHz, 192GB RAM, as main processors, and six GPUs NVIDIA GeForce RTX 2080Ti (with 11GB each)."; Input text reads on a text transcript, pretrained natural language inference (NLI) models read on a pre-trained language model, and dynamic masking reads on training via token masking.);
generating, using the pre-trained language model according to an engineered prompt and a predetermined taxonomy, a textual entailment from the text transcript (Section 3.1, lines 1-14, "The NLI task is commonly defined as a sentence-pair classification in which two sentences are given: a premise s1 and a hypothesis s2. The task is to learn a function
f
N
L
I
(s1, s2) → {E,C,N}, in which E expresses the entailment of s1 and s2, C denotes a contradiction and N is a neutral output. We treat ZSL emotion classification as a textual entailment problem, but represent each label under consideration with multiple prompts, in contrast to Yin et al. (2019). Given a sentence to be classified x (premise) and an emotion e, we have a function g(e) that generates a set of prompts (hypothesis) out of the class e ∈ E (with E being the set of emotions under consideration).”; Section 3.2.1, lines 1-8, "We generate a set of prompts with the function g(e) = c + r(e), in which c represents what we call the context and r(e) represents a set of emotion representations.1 As c, we use either an empty string ϵ, the text “This text expresses”, “This person feels”, or “This person expresses”, motivated by our choice of the language register presented in the datasets used in our experiments (see § 4)."; Section 3.2.2, lines 1-9, "Each prompt in this paper consists of context and the emotion representation. There are three prompts which have in common the emotion name representation, namely Emo-Name, Expr-Emo, and Feels-Emo. Variations of these prompts are Emo-S, Expr-S, and Feels-S, where the emotion name representation is replaced by multiple emotion synonyms and EmoLex where the emotion name is replaced by entries from an emotion word lexicon."; Section 3.3, line 6, "The ensemble model takes as input a text x"; Section 4.1.2, lines 10-17, "For our ZSL experiments, we explore three state-of-the-art pretrained NLI models publicly available within the Hugging Face Transformers Python library (Wolf et al., 2020), and fine-tuned on the MultiNLI dataset (Williams et al., 2018). Concretely, we choose RoBERTa, BART and DeBERTa as they cover different architectures and represent competitive approaches across a set of NLP tasks."; Pretrained natural language inference (NLI) models read on a pre-trained language model, the entailment of premise s1 and hypothesis s2 reads on a textual entailment from the text transcript, generating a set of prompts reads on an engineered prompt, and an emotion word lexicon reads on a predetermined taxonomy.);
and generating, by the one or more processors using the textual entailment, a predicted emotion corresponding to the input speech (Section 3.1, lines 10-25, "Given a sentence to be classified x (premise) and an emotion e, we have a function g(e) that generates a set of prompts (hypothesis) out of the class e ∈ E (with E being the set of emotions under consideration). Under the assumption of an NLI model m, which calculates the entailment probability
p
m
(γ, x) for some emotion representation γ ∈ g(e), we assign the average entailment probability across all emotion representations as
p
m
-
g
(
e
,
x
)
=
1
|
g
e
|
∑
γ
∈
g
(
e
)
p
m
(
γ
,
x
)
for a particular prompt generation method g. The classification decision
e
x
g
(
e
,
x
)
=
a
r
g
m
a
x
e
∈
E
p
m
-
g
(
e
,
x
)
returns the emotion corresponding to the maximum entailment probability."; Returning the emotion corresponding to the maximum entailment probability reads on generating a predicted emotion corresponding to the input speech using the textual entailment.).
Plaza-del-Arco does not specifically disclose: generating, by one or more processors, a text transcript for a snippet of input speech.
Compton teaches:
generating, by one or more processors, a text transcript for a snippet of input speech (Paragraph 0175, lines 1-5, "FIG. 9 is a flowchart illustrating a routine 900 that may be executed by the system 500. In block 902, at least one processor receives a digital speech signal. In block 904, the at least one processor converts the digital speech signal to text.").
Compton is considered to be analogous to the claimed invention because it is in the same field of predicting emotion of text. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Plaza-del-Arco to incorporate the teachings of Compton to convert a speech signal to text. Doing so would allow an automated dialogue system to react appropriately to patient provided information during medical history taking (Compton; Paragraph 0031, lines 1-15).
Plaza-del-Arco in view of Compton does not specifically disclose: wherein the predetermined taxonomy (i) is chosen based on the specific application or product with which the SER model is used and (ii) includes a set of words or phrases corresponding to the specific application or product.
Mohammad teaches:
wherein the predetermined taxonomy (i) is chosen based on the specific application or product with which the SER model is used and (ii) includes a set of words or phrases corresponding to the specific application or product (Section 1, lines 24-39, "Natural language applications in commerce, public health, disaster management, and public policy can benefit from knowing the affectual states of people—both the categories and the intensities of the emotions they feel. We thus present the SemEval-2018 Task 1: Affect in Tweets, which includes an array of subtasks where automatic systems have to infer the affectual state of a person from their tweet. We will refer to the author of a tweet as the tweeter. Some of the tasks are on the intensities of four basic emotions common to many proposals of basic emotions: anger, fear, joy, and sadness. Some of the tasks are on valence or sentiment intensity. Finally, we include an emotion classification task over eleven emotions commonly expressed in tweets."; Section 3.1.1, lines 15-17, " We polled the Twitter API, over the span of two months (June and July, 2017), for tweets that included the query terms."; Performing an emotion classification task over eleven emotions commonly expressed in tweets reads on the predetermined taxonomy being chosen based on a specific application for which a speech emotion recognition model is used and the predetermined taxonomy including a set of words or phrases corresponding to the specific application.).
Mohammad is considered to be analogous to the claimed invention because it is in the same field of predicting emotion of text. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Plaza-del-Arco in view of Compton to incorporate the teachings of Mohammad to perform an emotion classification task over eleven emotions commonly expressed in tweets. Doing so would allow for automatically inferring the affectual state of a person from their tweet (Mohammad; Section 1, lines 24-39).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Basile et al. ("Probabilistic Ensembles of Zero-and Few-Shot Learning Models for Emotion Classification")
Demszky et al. ("GoEmotions: A Dataset of Fine-Grained Emotions")
Yin et al. ("Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach")
Any inquiry concerning this communication or earlier communications from the examiner should be directed to James Boggs whose telephone number is (571)272-2968. The examiner can normally be reached M-F 8:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at (571)272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAMES BOGGS/Examiner, Art Unit 2657
Prosecution Insights