DETAILED ACTION
This communication is in response to the Amendments and Arguments filed on 7/5/2025.
Claims 1,3-5,12,16-17,21-22, and 24 are pending and have been examined.
All previous objections / rejections not mentioned in this Office Action have been withdrawn by the examiner.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 6/24/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Response to Arguments
Regarding the Applicant’s arguments for the rejections under 35 U.S.C. § 103, Applicant has amended independent claims 1, 21, and 24. The specific written limitations of claim 1, 21, and 24 are different, but contain limitations that are in alignment. Applicant asserts that prior art reference Fernandes or Bocklet does not specifically teach “taking the mean value for each time band as well as running mean normalization” and “reduce the multi-dimensional matrix into a single dimension”. Examiner respectfully disagrees. During patent examination, pending claims must be “given their broadest reasonable interpretation consistent with the specification.” MPEP 2111. The claim language, broadly interpreted, is taught by the prior art references. First, prior art reference Bocklet teaches normalization through the use of mean and standard deviation for each time band. Bocklet P0012-P0014. Examiner does not interpret mean normalization with the formula described in applicants’ response since the as filed disclosure does not incorporate the formula to define mean normalization. Second, prior art reference Fernandes teaches the use of dimensional reduction and normalization in the process of emotion recognition. Fernandes also teaches the reduction into a single dimension for each audio sample by taking the mean across each mel coefficients. Fernandes Section IV. Dimensionality Reduction. The claim limitation “taking the mean value for each time band” is broad and is interpreted as taking the mean value across each mel coefficients.
Furthermore, Applicant has amended dependent claim 17. Applicant asserts that Kim does not teach RNN or layers to “reduce data overfitting” or “transforming data into useful numbers”. Examiner respectfully disagrees. Kim teaches recurrent neural network and associated layers to transform data into useful numbers. Kim Col. 31, Lines 19-36.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-5, 12, 16-1721, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (U.S. Patent No. 11501794), hereinafter Kim, in view of Eleftheriou et al. (U.S. PG Pub No. 20200075039), hereinafter Eleftheriou, in view of Fernandes et al. ("Speech Emotion Recognition using Mel Frequency Cepstral Coefficient and SVM Classifier"), hereinafter Fernandes, and in further view of Bocklet et al. (U.S. PG Pub No. 20180322863), hereinafter Bocklet.
Regarding claim 1 Kim teaches:
A system for enabling a user to tactilely feel the emotion in a verbal input, said system comprising: (Fig. 1, Columns 3 and 25, lines 1-7 and 32-42 respectively: The system can detect a sentiment (emotion) of a user. The satisfaction estimator 850 of Fig. 8 can analyze the output of the speech processing component 240 to determine whether the user’s speech is positive, negative or neutral. Also, in column 67, lines 23-29, Kim teaches that the system can include a haptic output device that provides output that produces particular touch sensations to the user)
a verbal input receiving device for receiving a spoken input signal; (Refer to Figs. 1-2, Device 110 and Columns 3-4, lines 64-67 and lines 1-15 and 64-67 respectively: The device 110 can be a speech-enabled device and can detect audio 11 by the user 5 by receiving input audio data using a microphone. The device 110 can determine that audio 11 includes a wake word and may then send audio data corresponding to the audio 11 to the system(s) 120);
a processor and memory configured with machine readable code to define a pre-processing stage and an emotion model in the form of a trained artificial neural network stage for extracting at least one emotion associated with the verbal input (Fig. 9, Fig. 21 no.: 2116 and Fig. 24, Columns 3, 30 and 50, lines 8-29, 37-55 and 18-36 respectively: The device 110 includes one or more processors, which includes a central processing unit for processing data and computer-readable instructions, and a memory for storing data instructions. The device may preprocess the input data to improve the accuracy of the sentiment (emotion) data. The sentiment detection component 275 of Fig. 9 includes a voice activity detection component 905, an encoder component 920, an utterance feature vector 960, and a trained model 965. The trained model component 965 can process the utterance feature vector(s) 960 using a fully-connected neural network. The trained model component 965 can output score(s) 970 indicating a sentiment category 975 for the user audio 915 (memory configured with machine readable code for extracting emotional content). The sentiment (emotion) categories may be angry, neutral and/or happy);
one or more haptic feedback device worn by one or more users, which are configured to emit a haptic feedback signal associated with said at least one emotion, (Kim col. 67, lines 23-29, output devices include a haptic output device that can produce a particular touch sensation to the user such as a buzz or vibration to simulate the purr of a cat).
Kim does not specifically teach:
one or more haptic feedback device worn by one or more users, which are configured to emit a haptic feedback signal associated with said at least one emotion,
wherein the pre-processing stage is configured to generate a multi-dimensional Mel Spectrogram or Mel-Frequency Cepstral Coefficient (MFCC) matrix and to reduce the multi-dimensional matrix to a single dimensional output by taking the mean value for each time band as well as running mean normalization across all the data and feeding this into the artificial neural network stage.
Eleftheriou, however, teaches:
one or more haptic feedback device worn by one or more users, which are configured to emit a haptic feedback signal associated with said at least one emotion, (Fig. 1, Paragraphs 23 and 68 – Generally, in blocks S170, S180, and S190 – the wearable device can access the emotion model of the user and detect an instance of a target emotion (e.g., a happy or sad moment) and signal the user of the instance via the wearable device (e.g., by haptic feedback, emitting a tone, etc.)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system for enabling a user to tactilely feel the emotion in a verbal input receiving device of Kim with a wearable haptic device as disclosed by Eleftheriou, at least because doing so would alert the user to be mindful of the user’s emotional state. Moreover, a wearable device would allow the user to have targeted emotions in an unobtrusive manner throughout the day and thus reducing the time required to generate an accurate emotion model for the user. (Eleftheriou P0023, P0068).
Kim in view of Eleftheriou does not specifically teach:
wherein the pre-processing stage is configured to generate a multi-dimensional Mel Spectrogram or Mel-Frequency Cepstral Coefficient (MFCC) matrix and to reduce the multi-dimensional matrix to a single dimensional output by taking the mean value for each time band as well as running mean normalization across all the data and feeding this into the artificial neural network stage.
Fernandes, however, teaches:
wherein the pre-processing stage is configured to generate a multi-dimensional Mel Spectrogram or Mel-Frequency Cepstral Coefficient (MFCC) matrix and to reduce the multi-dimensional matrix to a single dimensional output by taking the mean value for each time band as well as running mean normalization across all the data and feeding this into the artificial neural network stage. (Abstract, Using Mel Frequency Cepstral Coefficient (MFCC) feature extraction; III, MFCC Feature Extraction, calculated by applying a Mel-scale filter bank to the Fourier transform of a windowed signal and later a DCT transforms the log spectrum into a cepstral.; Abstract, Section IV, A supervised learning approach for Speech Emotion Recognition using Mel-Frequency Cepstral Coefficient (MFCC) feature extraction. The CC matrix is reduced in dimension from two-dimensional 12xF matrix to a 1 x 60 single dimension output.; Section IV, the 12XF matrix (two-dimensional matrix) is reduced to a 1 x 60 matrix (single dimension) by extracting mean, median, standard deviation, kurtosis and skewness across each mel coefficients.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a two-dimensional Mel-Frequency Cepstral Coefficient (MFCC) matrix and reducing the 2D matrix to a 1D matrix as disclosed by Fernandes, at least because doing so would allow the efficiency/accuracy in detecting emotions to increase. Note that MFFCs enable a signal representation that is closer to human hearing perception and hence are an extremely useful feature in emotion recognition. (Fernandes, Page 200). Additionally, reducing the two-dimensional matrix to a single-dimension matrix output by means of mean normalization or other dimensionality reduction would prevent attributes in greater numeric ranges from dominating those in the smaller numeric ranges, as suggested by Fernandes (Fernandes, Page 203).
Kim, in view of Eleftheriou, and further view of Fernandes does not specifically teach:
wherein the pre-processing stage is configured to generate a multi-dimensional Mel Spectrogram or Mel-Frequency Cepstral Coefficient (MFCC) matrix and to reduce the multi-dimensional matrix to a single dimensional output by taking the mean value for each time band as well as running mean normalization across all the data and feeding this into the artificial neural network stage.
Bocklet, however, teaches:
wherein the pre-processing stage is configured to generate a multi-dimensional Mel Spectrogram or Mel-Frequency Cepstral Coefficient (MFCC) matrix and to reduce the multi-dimensional matrix to a single dimensional output by taking the mean value for each time band as well as running mean normalization across all the data and feeding this into the artificial neural network stage. (P0004, Analyze the speech for Mel-Frequency Cepstral Coefficients (MFCCs). In order to improve the accuracy and reliability of the MFCC, Cepstral Mean Subtraction (CMS) is used in combination with Cepstral Variance Normalization (CVN).; P0013, The features are normalized by taking the feature for each frame, subtracting the mean and diving by the variance.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize mean normalization where mean is subtracted from the feature. It would have been obvious to combine the references because mean normalization improves the accuracy and reliability of the MFCC where the process leads to the removal of stationary noise. (Bocklet P0004)
Regarding claim 3 Kim in view of Eleftheriou, in view of Fernandes, and further view of Bocklet teach claim 1.
Kim does not specifically teach:
wherein the haptic feedback device comprises a wristband or other wearable item in communication with an output from the artificial neural network stage.
Eleftheriou, however, teaches:
wherein the haptic feedback device comprises a wristband or other wearable item in communication with an output from the artificial neural network stage. (Fig. 1, Fig. 7, and Paragraphs 23 and 68 – Generally, in blocks S170, S180, and S190 – the wearable device can access the emotion model of the user and detect an instance of a target emotion (e.g., a happy or sad moment) and signal the user of the instance via the wearable device/wristband/watch (e.g., by haptic feedback, emitting a tone, etc.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system for enabling a user to tactilely feel the emotion in a verbal input receiving device of Kim and Eleftheriou and incorporate a wristband or other wearable device as disclosed by Eleftheriou, at least because doing so would alert the user to be mindful of the user’s emotional state. Moreover, a wearable device would allow the user to have targeted emotions in an unobtrusive manner throughout the day and thus reducing the time required to generate an accurate emotion model for the user. Eleftheriou (P0023, P0068).
Regarding claim 4 Kim in view of Eleftheriou, in view of Fernandes, and further view of Bocklet teach claim 3.
Kim further teaches:
wherein the haptic feedback signal comprises a vibration that is unique for each emotion or combination of emotions extracted by the neural network stage. (Fig. 9, Columns 30 and 67, lines 37-55 and 23-29 respectively: The system can include a haptic output device that provides output that produces particular touch sensations to the user. A motor with an eccentric weight may be used to create a buzz or vibration. The trained model component 965 can process the utterance feature vector(s) 960 using a fully-connected neural network. The trained model component 965 can output score(s) 970 indicating a sentiment category 975 for the user audio 915. The sentiment (emotion) categories may be angry, neutral and/or happy).
Regarding claim 5 Kim in view of Eleftheriou, in view of Fernandes, and further view of Bocklet teach claim 1.
Kim further teaches:
wherein the users are speakers taking part in live, face-to-face conversation, and the verbal input receiving device comprises a microphone (Fig. 1, Fig. 23A no.:2320, Columns 3-5, lines 1-7, 64-67 and 1-2 respectively: As shown in Fig. 1, the device 110 can receive (130) first input audio data corresponding to a user 5. The device 110 can capture the first input audio data using a microphone and the first input audio data may include speech or sounds from the user 5 and/or speech and sounds from at least one other person. The device may also detect a sentiment (e.g., emotion) of a user while speaking to the system or to another person (live face-to-face conversation between two speakers)).
Regarding claim 12 Kim in view of Eleftheriou, in view of Fernandes, and further view of Bocklet teach claim 1.
Kim further teaches:
wherein the haptic feedback signal comprises a vibration that is unique to each emotion or combination of emotions identified by the neural network (Fig. 9, Columns 30 and 67, lines 37-55 and 23-29 respectively: The system can include a haptic output device that provides output that produces particular touch sensations to the user. A motor with an eccentric weight may be used to create a buzz or vibration. The trained model component 965 can process the utterance feature vector(s) 960 using a fully-connected neural network. The trained model component 965 can output score(s) 970 indicating a sentiment category 975 for the user audio 915. The sentiment (emotion) categories may be angry, neutral and/or happy).
Regarding claim 16 Kim in view of Eleftheriou, in view of Fernandes, and further view of Bocklet teach claim 1.
Kim does not specifically teach:
further comprising representing the at least one emotion in auditory form to a speaking participant.
Eleftheriou, however, teaches:
further comprising representing the at least one emotion in auditory form to a speaking participant. (Fig. 1, Paragraphs 23 and 68 – Generally, in blocks S170, S180, and S190 – the wearable device can access the emotion model of the user and detect an instance of a target emotion (e.g., a happy or sad moment) and signal the user of the instance via the wearable device (e.g., by haptic feedback, emitting a tone, etc.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system for enabling a user to tactilely feel the emotion in a verbal input receiving device of Kim as modified by Pandey and Fernandes and incorporate a wearable haptic device as disclosed by Eleftheriou, at least because doing so would alert the user to be mindful of the user’s emotional state. Moreover, a wearable device would allow the user to have targeted emotions in an unobtrusive manner throughout the day and thus reducing the time required to generate an accurate emotion model for the user. (Eleftheriou P0023, P0068).
Regarding claim 17 Kim in view of Eleftheriou, in view of Fernandes, and further view of Bocklet teach claim 1.
Kim further teaches:
wherein the artificial neural network stage is a recurrent neural network (RNN) that includes layers for performing one or more of: reducing data overfitting, and transforming data into useful numbers. (Col. 31, Lines 19-36, A neural network may be structured with an input layer, middle layer(s), and an output layer. … Each connection from one node to another node in the next layer may be associated with a weight or score. A neural network may output a single output or a weighted set of possible outputs. … The output of the hidden layer may be fed back into the hidden layer for processing of the next set of inputs. A neural network incorporating recurrent connections may be referred to as a recurrent neural network (RNN).)
Regarding claim 21 Kim teaches:
A system for enabling a user to tactilely feel the emotion in a verbal input, said system comprising: (Fig. 1, Columns 3 and 25, lines 1-7 and 32-42 respectively: The system can detect a sentiment (emotion) of a user. The satisfaction estimator 850 of Fig. 8 can analyze the output of the speech processing component 240 to determine whether the user’s speech is positive, negative or neutral. Also, in column 67, lines 23-29, Kim teaches that the system can include a haptic output device that provides output that produces particular touch sensations to the user)
a verbal input receiving device for receiving a spoken input signal; (Refer to Figs. 1-2, Device 110 and Columns 3-4, lines 64-67 and lines 1-15 and 64-67 respectively: The device 110 can be a speech-enabled device and can detect audio 11 by the user 5 by receiving input audio data using a microphone. The device 110 can determine that audio 11 includes a wake word and may then send audio data corresponding to the audio 11 to the system(s) 120);
a processor and memory configured with machine readable code to define a pre-processing stage and an emotion model receiving the output of the pre-processing stage as its input, wherein the emotion model comprises a trained neural network stage for extracting at least one emotion associated with the verbal input (Fig. 9, Fig. 21 no.: 2116 and Fig. 24, Columns 3, 30 and 50, lines 8-29, 37-55 and 18-36 respectively: The device 110 includes one or more processors, which includes a central processing unit for processing data and computer-readable instructions, and a memory for storing data instructions. The device may preprocess the input data to improve the accuracy of the sentiment (emotion) data. The sentiment detection component 275 of Fig. 9 includes a voice activity detection component 905, an encoder component 920, an utterance feature vector 960, and a trained model 965. The trained model component 965 can process the utterance feature vector(s) 960 using a fully-connected neural network. The trained model component 965 can output score(s) 970 indicating a sentiment category 975 for the user audio 915 (memory configured with machine readable code for extracting emotional content). The sentiment (emotion) categories may be angry, neutral and/or happy);
a haptic feedback device worn by a user, which are configured to emit a haptic feedback signal associated with said at least one emotion. (Kim col. 67, lines 23-29, output devices include a haptic output device that can produce a particular touch sensation to the user such as a buzz or vibration to simulate the purr of a cat).
Kim does not specifically teach:
wherein the pre-processing stage is configured to generate a multi-dimensional Mel Spectrogram or a two-dimensional Mel-Frequency Cepstral Coefficient (MFCC) matrix from time bands defined in the spoken input signal, and take the mean value for each time band and also run mean normalization across all the data to reduce the multi-dimensional matrix to a single dimensional output; and
a haptic feedback device worn by a user, which are configured to emit a haptic feedback signal associated with said at least one emotion.
Eleftheriou, however, teaches:
a haptic feedback device worn by a user, which are configured to emit a haptic feedback signal associated with said at least one emotion. (Fig. 1, Paragraphs 23 and 68 – Generally, in blocks S170, S180, and S190 – the wearable device can access the emotion model of the user and detect an instance of a target emotion (e.g., a happy or sad moment) and signal the user of the instance via the wearable device (e.g., by haptic feedback, emitting a tone, etc.)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system for enabling a user to tactilely feel the emotion in a verbal input receiving device of Kim with a wearable haptic device as disclosed by Eleftheriou, at least because doing so would alert the user to be mindful of the user’s emotional state. Moreover, a wearable device would allow the user to have targeted emotions in an unobtrusive manner throughout the day and thus reducing the time required to generate an accurate emotion model for the user. (Eleftheriou P0023, P0068).
Kim in view of Eleftheriou does not specifically teach:
wherein the pre-processing stage is configured to generate a multi-dimensional Mel Spectrogram or a two-dimensional Mel-Frequency Cepstral Coefficient (MFCC) matrix from time bands defined in the spoken input signal, and take the mean value for each time band and also run mean normalization across all the data to reduce the multi-dimensional matrix to a single dimensional output; and
Fernandes, however, teaches:
wherein the pre-processing stage is configured to generate a multi-dimensional Mel Spectrogram or a two-dimensional Mel-Frequency Cepstral Coefficient (MFCC) matrix from time bands defined in the spoken input signal, and take the mean value for each time band and also run mean normalization across all the data to reduce the multi-dimensional matrix to a single dimensional output; and (Abstract, Using Mel Frequency Cepstral Coefficient (MFCC) feature extraction; III, MFCC Feature Extraction, calculated by applying a Mel-scale filter bank to the Fourier transform of a windowed signal and later a DCT transforms the log spectrum into a cepstral.; Abstract, Section IV, A supervised learning approach for Speech Emotion Recognition using Mel-Frequency Cepstral Coefficient (MFCC) feature extraction. The CC matrix is reduced in dimension from two-dimensional 12xF matrix to a 1 x 60 single dimension output.; Section IV, the 12XF matrix (two-dimensional matrix) is reduced to a 1 x 60 matrix (single dimension) by extracting mean, median, standard deviation, kurtosis and skewness across each mel coefficients.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a two-dimensional Mel-Frequency Cepstral Coefficient (MFCC) matrix and reducing the 2D matrix to a 1D matrix as disclosed by Fernandes, at least because doing so would allow the efficiency/accuracy in detecting emotions to increase. Note that MFFCs enable a signal representation that is closer to human hearing perception and hence are an extremely useful feature in emotion recognition. (Fernandes, Page 200). Additionally, reducing the two-dimensional matrix to a single-dimension matrix output by means of mean normalization or other dimensionality reduction would prevent attributes in greater numeric ranges from dominating those in the smaller numeric ranges, as suggested by Fernandes (Fernandes, Page 203).
Kim, in view of Eleftheriou, and further view of Fernandes does not specifically teach:
wherein the pre-processing stage is configured to generate a multi-dimensional Mel Spectrogram or a two-dimensional Mel-Frequency Cepstral Coefficient (MFCC) matrix from time bands defined in the spoken input signal, and take the mean value for each time band and also run mean normalization across all the data to reduce the multi-dimensional matrix to a single dimensional output; and
Bocklet, however, teaches:
wherein the pre-processing stage is configured to generate a one-dimensional Mel Spectrogram or a two-dimensional Mel-Frequency Cepstral Coefficient (MFCC) matrix, and takes the mean value for each time band and also runs mean normalization across the entire data set; and (P0004, Analyze the speech for Mel-Frequency Cepstral Coefficients (MFCCs). In order to improve the accuracy and reliability of the MFCC, Cepstral Mean Subtraction (CMS) is used in combination with Cepstral Variance Normalization (CVN).; P0013, The features are normalized by taking the feature for each frame, subtracting the mean and diving by the variance.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize mean normalization where mean is subtracted from the feature. It would have been obvious to combine the references because mean normalization improves the accuracy and reliability of the MFCC where the process leads to the removal of stationary noise. (Bocklet P0004)
Regarding claim 24 Kim teaches:
A method for analyzing spoken audio input obtained from one or more speaking participants for emotional content, comprising: (Col. 3, Lines 1-3, To improve an interaction with the user, the device may detect a sentiment (e.g., emotion) of a user while speaking to the system or to another person.; Col. 3, Lines 9-10, Methods are disclosed that perform cross-modal or multimodal sentiment detection.; )
capturing time bands of the spoken audio input, (Refer to Figs. 1-2, Device 110 and Columns 3-4, lines 64-67 and lines 1-15 and 64-67 respectively: The device can be a speech-enabled device and can detect audio by the user by receiving input audio data using a microphone.; Col. 58, Lines 10-15, The AFE may divide the raw audio data into frames representing time intervals for which the AFE determines a number of values (i.e., features) representing qualities of the raw audio data, along with a set of those values (i.e., a feature vector or audio feature vector) representing features/qualities of the raw audio data within each frame.)
pre-processing the time bands of the spoken audio input by transforming the time bands into a multi-dimensional Mel-Frequency Cepstral Coefficient (MFCC) matrix or Mel Spectrogram, and (Col. 58, Lines 16-24, A frame may be a certain period of time, for example a sliding window of 25 ms of audio data taken every 10 ms, or the like. A number of approaches may be used by the AFE to process the raw audio data 2343, such as mel-frequency cepstral coefficients (MFCCs).)
feeding the single dimensional output into a neural network to identify at least one emotion in the audio input, and (Fig. 9, Fig. 21 no.: 2116 and Fig. 24, Col. 3, 30 and 50, Lines 8-29, 37-55 and 18-36, The sentiment detection component of Fig. 9 includes a voice activity detection component, an encoder component, an utterance feature vector, and a trained model. The trained model component can process the utterance feature vector(s) using a fully-connected neural network. The trained model component can output score(s) indicating a sentiment category for the user audio (memory configured with machine readable code for extracting emotional content). The sentiment (emotion) categories may be angry, neutral and/or happy.)
Kim does not specifically teach:
reducing the multi-dimensional matrix to a single-dimension output by taking the mean value for each time band as well as running mean normalization across all the data, the method further comprising,
converting the at least one emotion into haptic feedback.
Eleftheriou, however, teaches:
converting the at least one emotion into haptic feedback. (Fig. 1, Paragraphs 23 and 68 – Generally, in blocks S170, S180, and S190 – the wearable device can access the emotion model of the user and detect an instance of a target emotion (e.g., a happy or sad moment) and signal the user of the instance via the wearable device (e.g., by haptic feedback, emitting a tone, etc.)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system for enabling a user to tactilely feel the emotion in a verbal input receiving device of Kim with a wearable haptic device as disclosed by Eleftheriou, at least because doing so would alert the user to be mindful of the user’s emotional state. Moreover, a wearable device would allow the user to have targeted emotions in an unobtrusive manner throughout the day and thus reducing the time required to generate an accurate emotion model for the user. (Eleftheriou P0023, P0068).
Kim in view of Eleftheriou does not specifically teach:
reducing the multi-dimensional matrix to a single-dimension output by taking the mean value for each time band as well as running mean normalization across all the data, the method further comprising,
Fernandes, however, teaches:
reducing the multi-dimensional matrix to a single-dimension output by taking the mean value for each time band as well as running mean normalization across all the data, the method further comprising, (Abstract, Using Mel Frequency Cepstral Coefficient (MFCC) feature extraction; III, MFCC Feature Extraction, calculated by applying a Mel-scale filter bank to the Fourier transform of a windowed signal and later a DCT transforms the log spectrum into a cepstral.; Abstract, Section IV, A supervised learning approach for Speech Emotion Recognition using Mel-Frequency Cepstral Coefficient (MFCC) feature extraction. The CC matrix is reduced in dimension from two-dimensional 12xF matrix to a 1 x 60 single dimension output.; Section IV, the 12XF matrix (two-dimensional matrix) is reduced to a 1 x 60 matrix (single dimension) by extracting mean, median, standard deviation, kurtosis and skewness across each mel coefficients.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a two-dimensional Mel-Frequency Cepstral Coefficient (MFCC) matrix and reducing the 2D matrix to a 1D matrix as disclosed by Fernandes, at least because doing so would allow the efficiency/accuracy in detecting emotions to increase. Note that MFFCs enable a signal representation that is closer to human hearing perception and hence are an extremely useful feature in emotion recognition. (Fernandes, Page 200). Additionally, reducing the two-dimensional matrix to a single-dimension matrix output by means of mean normalization or other dimensionality reduction would prevent attributes in greater numeric ranges from dominating those in the smaller numeric ranges, as suggested by Fernandes (Fernandes, Page 203).
Kim, in view of Eleftheriou, and further view of Fernandes does not specifically teach:
reducing the multi-dimensional matrix to a single-dimension output by taking the mean value for each time band as well as running mean normalization across all the data, the method further comprising,
Bocklet, however, teaches:
wherein the pre-processing stage is configured to generate a one-dimensional Mel Spectrogram or a two-dimensional Mel-Frequency Cepstral Coefficient (MFCC) matrix, and takes the mean value for each time band and also runs mean normalization across the entire data set; and (P0004, Analyze the speech for Mel-Frequency Cepstral Coefficients (MFCCs). In order to improve the accuracy and reliability of the MFCC, Cepstral Mean Subtraction (CMS) is used in combination with Cepstral Variance Normalization (CVN).; P0013, The features are normalized by taking the feature for each frame, subtracting the mean and diving by the variance.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize mean normalization where mean is subtracted from the feature. It would have been obvious to combine the references because mean normalization improves the accuracy and reliability of the MFCC where the process leads to the removal of stationary noise. (Bocklet P0004)
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Kim, in view of Eleftheriou, in view of Fernandes, in view of Bocklet, and further view of Pandey et al. ("Deep Learning Techniques for Speech Emotion Recognition: A Review"), hereinafter Pandey.
Regarding claim 22 Kim in view of Eleftheriou, in view of Fernandes, and further view of Bocklet teach claim 21.
Kim in view of Eleftheriou, in view of Fernandes, and in further view of Bocklet does not specifically teach:
wherein the neural network stage comprises a recurrent neural network (RNN) architecture the includes layers for recurrently recognizing patterns, and layers for linearly altering data sizes.
Pandey, however, teaches:
wherein the neural network stage comprises a recurrent neural network (RNN) architecture the includes layers for recurrently recognizing patterns, and layers for linearly altering data sizes. (C. Long Short Term Memory, Recurrent network architecture. LSTM layer learn long term dependencies. Equation 4 representing hidden layers prior to activation, output of the previous layer into next layer, layers having respective weights.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system for enabling a user to tactilely feel the emotion in a verbal input receiving device of Kim as modified by Eleftheriou with a neural network comprises a RNN that includes one or more of the claimed layers as disclosed by Pandey, at least because doing so would help the neural network model to learn more robust features. (Pandey, Section D).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL WONSUK CHUNG whose telephone number is (571)272-1345. The examiner can normally be reached Monday - Friday (7am-4pm)[PT].
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, PIERRE-LOUIS DESIR can be reached at (571)272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL W CHUNG/Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659