DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Introduction
This office action is in response to communications filed 07/18/2024. Claims 1-20 are pending and likewise have been examined.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 07/18/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 5 recites the limitation "the pre-interruption and post-interruption speaking duration speaking durations" in line 2-3, and “the pre-interruption and post-interruption voice energy”. There is insufficient antecedent basis for this limitation in the claim. While Pre and post interruption speaking rates have been established prior to this point in the claim, speaking durations and voice energy have not.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1-19 of U.S. Patent No. 12100417. Although the claims at issue are not identical, they are not patentably distinct from each other. See table below for matching limitations.
Instant application
U.S. Patent No. 12100417
Claim 1: A system comprising: one or more processors; and a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to:
Claim 1: A system comprising:
one or more processors; and
a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to:
receive an audio file comprising a first channel and a second channel, the first channel comprising first voice activity of a first user and the second channel comprising second voice activity of a second user;
receive an audio file comprising a first channel and a second channel, the first channel comprising first voice activity of a first user and the second channel comprising second voice activity of a second user;
detect, using a deep neural network (DNN), one or more moments of interruption between the first and second users from the audio file by classifying portions of the audio file as either a speech portion or a non-speech portion based on one or more phonetic representations mapped to each portion of the audio file;
detect, using a deep neural network (DNN), one or more moments of interruption between the first and second users from the audio file by classifying portions of the audio file as either a speech portion or anon-speech portion based on one or more phonetic representations mapped to each portion of the audio file;
extract, using the DNN, one or more vocal features from the one or more moments of interruption;
extract, using the DNN, one or more vocal features from the one or more moments of interruption,
determine, using a machine learning model and based on the one or more vocal features, whether a threshold number of moments of the one or more moments of interruption corresponds to a first emotion type;
determine, using a machine learning model and based on the one or more vocal features, whether a threshold number of moments of the one or more moments of interruption corresponds to a first emotion type,
when the threshold number of moments corresponds to the first emotion type, transmit a first message comprising a first binary indication;
when the threshold number of moments corresponds to the first emotion type, transmit a first message comprising a first binary indication; and
and when the threshold number of moments does not correspond to the first emotion type, transmit a second message comprising a second binary indication.
when the threshold number of moments does not correspond to the first emotion type, transmit a second message comprising a second binary indication.
Claim 2: The system of claim 1, wherein the DNN comprises long short-term memory (LSTM).
Claim 2: The system of claim 1, wherein the DNN comprises long short-term memory (LSTM).
Claim 3: The system of claim 1, wherein the instructions are further configured to cause the system to: separate, using the DNN, the audio file into one or more portions;
Claim 3: The system of claim 1, wherein the instructions are further configured to cause the system to: separate, using the DNN, the audio file into one or more portions; and
map, using the DNN, each portion of the one or more portions to one or more phonetic representations;
map, using the DNN, each portion of the one or more portions to one or more phonetic representations.
and classify, using the DNN, each portion of the one or more portions as either a speech portion or a non-speech portion based on each portion’s respective one or more phonetic representations.
Claim 1: detect, using a deep neural network (DNN), one or more moments of interruption between the first and second users from the audio file by classifying portions of the audio file as either a speech portion or anon-speech portion based on one or more phonetic representations mapped to each portion of the audio file;
Claim 4: The system of claim 3, wherein the one or more phonetic representations comprise one or more of vowels and consonants.
Claim 4: The system of claim 3, wherein the one or more phonetic representations comprise one or more of vowels and consonants.
Claim 5: The system of claim 1, wherein a length of interruption comprises seconds, a pre-interruption and post-interruption speaking rates comprise number of syllables per second, the pre-interruption and post-interruption speaking durations comprise seconds, a voice activity ratio comprises active speech duration in seconds to pause duration in seconds, and the pre-interruption and post-interruption voice energy comprise decibels.
Claim 1: extract, using the DNN, one or more vocal features from the one or more moments of interruption, the one or more vocal features comprising one or more of pre-interruption speaking rate, post-interruption speaking rate, pre-interruption speaking duration, post-interruption speaking duration, voice activity ratio, or combinations thereof;
Claim 5:
The system of claim 1, wherein both the pre-interruption and the post-interruption speaking rates are extracted and comprise number of syllables per second, the pre-interruption and post-interruption speaking durations are extracted and comprise seconds, and the voice activity ratio are extracted and comprises active speech duration in seconds to pause duration in seconds.
Claim 6: The system of claim 1, wherein transmitting the first message further comprises classifying the audio file as associated with user agitation.
Claim 6: The system of claim 1, wherein transmitting the first message further comprises classifying the audio file as associated with user agitation.
Claim 7: The system of claim 1, wherein transmitting the second message further comprises classifying the audio file as associated with user non-agitation.
Claim 7: The system of claim 1, wherein transmitting the second message further comprises classifying the audio file as associated with user non-agitation.
Claim 8: A system comprising: one or more processors; and a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to:
Claim 8: A system comprising:
one or more processors; and a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to:
receive a dual-channel audio file comprising an interaction between a first user and a second user;
receive a dual-channel audio file comprising an interaction between a first user and a second user;
detect, using a neural network, one or more moments of interruption between the first and second users from the dual-channel audio file;
detect, using a neural network, one or more moments of interruption between the first and second users from the dual-channel audio file
extract, using the neural network, one or more vocal features associated with each of the one or more moments of interruption;
extract, using the neural network, one or more vocal features associated with each of the one or more moments of interruption
determine, based on the one or more vocal features, whether a threshold number of moments of the one or more moments of interruption corresponds to a first emotion type;
determine, using a machine learning model and based on the one or more vocal features, whether a threshold number of moments of the one or more moments of interruption corresponds to a first emotion type,
when the threshold number of moments corresponds to the first emotion type, transmit a first message comprising a first binary indication;
when the threshold number of moments corresponds to the first emotion type, transmit a first message comprising a first binary indication; and
and when the threshold number of moments does not correspond to the first emotion type, transmit a second message comprising a second binary indication.
when the threshold number of moments does not correspond to the first emotion type, transmit a second message comprising a second binary indication.
Claim 9: The system of claim 8, wherein the neural network is a deep neural network (DNN).
Claim 9: The system of claim 8, wherein the neural network is a deep neural network (DNN).
Claim 10: The system of claim 9, wherein the DNN comprises long short-term memory (LSTM).
The system of claim 9, wherein the DNN comprises long short-term memory (LSTM).
Claim 11: The system of claim 10, wherein the instructions are further configured to cause the system to: separate the dual-channel audio file into one or more portions;
Claim 11: The system of claim 10, wherein the instructions are further configured to cause the system to:
separate the dual-channel audio file into one or more portions;
map each portion of the one or more portions to one or more phonetic representations;
and map each portion of the one or more portions to one or more phonetic representations.
and classify each portion of the one or more portions as either a speech portion or a non-speech portion based on each portion’s respective one or more phonetic representations,
Claim 8: detect, using a neural network, one or more moments of interruption between the first and second users from the dual-channel audio file by classifying portions of the dual-channel audio file as either a speech portion or anon-speech portion based on one or more phonetic representations mapped to each portion of the dual-channel audio file
wherein the one or more phonetic representations comprise one or more of vowels and consonants.
wherein the one or more phonetic representations comprise one or more of vowels and consonants;
Claim 12: The system of claim 8, wherein transmitting the first message further comprises classifying the dual-channel audio file as associated with user agitation.
Claim 12: The system of claim 8, wherein transmitting the first message further comprises classifying the dual-channel audio file as associated with user agitation.
Claim 13: The system of claim 8, wherein transmitting the second message further comprises classifying the dual-channel audio file as associated with user non-agitation.
Claim 13: The system of claim 8, wherein transmitting the second message further comprises classifying the dual-channel audio file as associated with user non-agitation.
Claim 14: A system comprising: one or more processors; and a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to: receive an audio file comprising an interaction between a first user and a second user;
Claim 14: A system comprising:
one or more processors; and
a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to:
receive an audio file comprising an interaction between a first user and a second user;
detect, using a first machine learning model, one or more moments of interruption between the first and second users from the audio file;
detect, using a first machine learning model, one or more moments of interruption between the first and second users from the audio file
when a threshold number of one or more moments of interruption corresponds to a first emotion type, classify the audio file as associated with user agitation;
when the threshold number of moments corresponds to the first emotion type, classify the audio file as associated with user agitation; and
and when the threshold number of one or more moments of interruption does not correspond the first emotion type, classify the audio file as associated with user non-agitation.
when the threshold number of moments does not correspond the first emotion type, classify the audio file as associated with user non-agitation.
Claim 15: The system of claim 14, wherein the first machine learning model comprises a neural network.
Claim 15: The system of claim 14, wherein the first machine learning model comprises a neural network.
Claim 16: The system of claim 15, wherein the first machine learning model comprises a deep neural network (DNN) comprising long short-term memory (LSTM).
Claim 16: The system of claim 14, wherein the first machine learning model comprises a deep neural network (DNN) comprising long short-term memory (LSTM).
Claim 17: The system of claim 16, wherein the instructions are further configured to cause the system to: extract, using the neural network, one or more vocal features associated with each of the one or more moments of interruption; and determine, based on the one or more vocal features, whether a threshold number of moments of the one or more moments of interruption corresponds to a first emotion type, wherein the one or more vocal features comprise one or more of length of interruption, pre-interruption speaking rate, post-interruption speaking rate, pre-interruption speaking duration, post-interruption speaking duration, voice activity ratio, pre-interruption voice energy, and post-interruption voice energy.
Claim 14: extract, using the first machine learning model, one or more vocal features associated with each of the one or more moments of interruption, the one or more vocal features comprising one or more of speaking rate, speaking duration, and voice activity ratio, or combinations thereof; determine, using a second machine learning model and based on the one or more vocal features, whether a threshold number of moments of the one or more moments of interruption corresponds to a first emotion type, wherein the second machine learning model is trained based on one or more first vocal features comprising one or more labels corresponding to one or more emotion types, wherein the threshold number of moments is a predetermined integer associated with a specific emotion type;
Claim 18: The system of claim 14, wherein the audio file comprises a dual-channel audio file.
Claim 17: The system of claim 14, wherein the audio file comprises a dual-channel audio file.
Claim 19: The system of claim 18, wherein the instructions are further configured to cause the system to: separate, using the first machine learning model, the dual-channel audio file into one or more portions;
Claim 18: The system of claim 17, wherein the instructions are further configured to cause the system to:
separate, using the first machine learning model, the dual-channel audio file into one or more portions; and
map each portion of the one or more portions to one or more phonetic representations;
map each portion of the one or more portions to one or more phonetic representations.
and classify each portion of the one or more portions as either a speech portion or a non-speech portion based on each portion’s respective one or more phonetic representations.
Claim 14: detect, using a first machine learning model, one or more moments of interruption between the first and second users from the audio file by classifying, using the first machine learning model, portions of the audio file as either a speech portion or a non-speech portion based on one or more phonetic representations mapped to each portion of the audio file;
Claim 20: The system of claim 19, wherein the one or more phonetic representations comprise one or more of vowels and consonants.
Claim 19: The system of claim 18, wherein the one or more phonetic representations comprise one or more of vowels and consonants.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 8-9, 12-15 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Haukioja et al. (US 20190253558 A1), and further in view of Sivasubramanian et al. (US 20210157834 A1).
Regarding Claim 8:
Haukioja teaches a system comprising: one or more processors; and a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to(Para [0011], Ln 1-8, computer readable medium comprising processor-executable code….. the processor-executable code may cause the processor to perform certain operations):
receive a dual-channel audio file comprising an interaction between a first user and a second user(Para [0007], Ln 1-7, Each call is recorded or monitored and sampled live in real-time by the system. Ideally, the calls would be recorded on multiple tracks/channels, wherein each track/channel contains the voice of just one person);
detect, using a neural network, one or more moments of interruption between the first and second users from the dual-channel audio file(Para [0059], Ln 1-10, training and testing models, the system is able to process and extract and generate hundreds (100's), or more, of unique features from the speech data in phone calls with MFCC extraction, feature splicing….. and DNN training. Para [0031], Ln 1-6, system preferably utilizes a feature extraction engine with pattern recognition and classification techniques to extract and recognize salient features in the the audio signal data. Para [0030], Ln features: 1) Caller's Speech Emotion Readings measured over the duration of the call. The emotion readings, also known as SER (abbreviation for Speech Emotion Recognition) labels, also contain temporal information;……3) Number of interruptions, as indicated by the amount of overlap between caller's voice and the agent's voice. Para [0052], Ln 1-10, training and pattern recognition approach to the system emotional labeling, feature extraction and classification, the system preferably applies MFCC extraction on the sample by feature splicing the signal. Thereafter LDA+MLLT transformation are preformed, i.e., linear discriminant analysis and maximum likelihood linear transform. Hidden markov model (HMM) training and deep neural network (DNN) training);
extract, using the neural network, one or more vocal features associated with each of the one or more moments of interruption(Para [0030], Ln 1-18, 3) Number of interruptions, as indicated by the amount of overlap between caller's voice and the agent's voice, see details below; 4) Emotional disparity during the interruption, in case the agent has to gently interrupt the caller to bring the conversation back into focus…… The system may use the Support Vector Machine Classifier (SVM) algorithm. Para [0052], Ln 1-10, training and pattern recognition approach to the system emotional labeling, feature extraction and classification, the system preferably applies MFCC extraction on the sample by feature splicing the signal. Thereafter LDA+MLLT transformation are preformed, i.e., linear discriminant analysis and maximum likelihood linear transform. Hidden markov model (HMM) training and deep neural network (DNN) training).
Haukioja does not teach determine, based on the one or more vocal features, whether a threshold number of moments of the one or more moments of interruption corresponds to a first emotion type; when the threshold number of moments corresponds to the first emotion type, transmit a first message comprising a first binary indication; and when the threshold number of moments does not correspond to the first emotion type, transmit a second message comprising a second binary indication.
In the same field of call monitoring, Sivasubramanian teaches determine, based on the one or more vocal features, whether a threshold number of moments of the one or more moments of interruption corresponds to a first emotion type(Para [0157], Ln 1-15, organizations can use contacts analytics output files for various use cases…set threshold associated with sentiments and define actions or categories based on certain thresholds being exceeded. For example, a run of N negative sentiments coupled with an overall negative sentiment score may be categorized as a bad interaction and a supervisor may be notified. Para [0116], Ln 16-22, categories can be defined based on content of communications as well as acoustic characteristics in the case of audio contacts. For example, calls may be categorized to identify instances of long silence, talking too fast, interruptions, more. Para [0042], Ln 9-17, calls are analyzed to extract different call characteristics which may include one or more of the following non-limiting examples: talk speed, interruptions, silence (e.g., gaps in speech), speaker energy, pitch, tone, and other voice characteristics);
when the threshold number of moments corresponds to the first emotion type, transmit a first message comprising a first binary indication(Para [0157], Ln 1-15, organizations can use contacts analytics output files for various use cases…set threshold associated with sentiments and define actions or categories based on certain thresholds being exceeded. For example, a run of N negative sentiments coupled with an overall negative sentiment score may be categorized as a bad interaction and a supervisor may be notified);
and when the threshold number of moments does not correspond to the first emotion type, transmit a second message comprising a second binary indication(Para [0157], Ln 1-15, organizations can use contacts analytics output files for various use cases…set threshold associated with sentiments and define actions or categories based on certain thresholds being exceeded. For example, a run of N negative sentiments coupled with an overall negative sentiment score may be categorized as a bad interaction. (sending message is being interpreted under broadest reasonable interpretation, the identification of whether or not the threshold is met and the result of that being stored in memory would read on sending a message). Para [0258], Ln 1-11, executable program instructions for the general administration and operation of that server and includes a computer-readable storage medium (e.g., a hard disk, random access memory, read only memory, etc.) storing instructions that, if executed by a processor of the server, cause or otherwise allow the server to perform its intended functions).
It would have been obvious for one skilled in the art, at the effective time of filling to modify Haukioja with the call center analytics system of Sivasubramanian, as it can help improve customer experience by notifying supervisors of poor customer experience(Para [0157], 1-15).
Regarding Claim 9:
The combination of Haukioja and Sivasubramanian teaches the system of claim 8, and Haukioja teaches wherein the neural network is a deep neural network (DNN)( Para [0052], Ln 1-10, training and pattern recognition approach to the system emotional labeling, feature extraction and classification, the system preferably applies MFCC extraction on the sample by feature splicing the signal. Thereafter LDA+MLLT transformation are preformed, i.e., linear discriminant analysis and maximum likelihood linear transform. Hidden markov model (HMM) training and deep neural network (DNN) training).
Regarding Claim 12:
The combination of Haukioja and Sivasubramanian teaches the system of claim 8, but does not teach wherein transmitting the first message further comprises classifying the dual-channel audio file as associated with user agitation.
In the same field of call monitoring, Sivasubramanian teaches wherein transmitting the first message further comprises classifying the dual-channel audio file as associated with user agitation(Para [0166], Ln 1-17, customers may exhibit positive sentiments (e.g., as illustrated in FIG. 18 by the customers with smiles) as well as negative sentiments (e.g., as illustrated in FIG. 18 by the angry customer). Para [0157], Ln 1-15, organizations can use contacts analytics output files for various use cases…set threshold associated with sentiments and define actions or categories based on certain thresholds being exceeded. For example, a run of N negative sentiments coupled with an overall negative sentiment score may be categorized as a bad interaction and a supervisor may be notified).
It would have been obvious for one skilled in the art, at the effective time of filling to modify the combination of Haukioja and Sivasubramanian with the call center analytics system of Sivasubramanian, as it can help improve customer experience by notifying supervisors of poor customer experience(Para [0157], 1-15).
Regarding Claim 13:
The combination of Haukioja and Sivasubramanian teaches the system of claim 8, but does not teach wherein transmitting the second message further comprises classifying the dual-channel audio file as associated with user non-agitation.
In the same field of call monitoring, Sivasubramanian teaches wherein transmitting the second message further comprises classifying the dual-channel audio file as associated with user non-agitation(Para [0166], Ln 1-17, customers may exhibit positive sentiments (e.g., as illustrated in FIG. 18 by the customers with smiles) as well as negative sentiments (e.g., as illustrated in FIG. 18 by the angry customer). Para [0157], Ln 1-15, organizations can use contacts analytics output files for various use cases…set threshold associated with sentiments and define actions or categories based on certain thresholds being exceeded. For example, a run of N negative sentiments coupled with an overall negative sentiment score may be categorized as a bad interaction and a supervisor may be notified).
It would have been obvious for one skilled in the art, at the effective time of filling to modify the combination of Haukioja and Sivasubramanian with the call center analytics system of Sivasubramanian, as it can help improve customer experience by notifying supervisors of poor customer experience(Para [0157], 1-15).
Regarding Claim 14:
Haukioja teaches a system comprising: one or more processors; and a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to(Para [0011], Ln 1-8, computer readable medium comprising processor-executable code….. the processor-executable code may cause the processor to perform certain operations):
receive an audio file comprising an interaction between a first user and a second user(Para [0007], Ln 1-7, Each call is recorded or monitored and sampled live in real-time by the system. Ideally, the calls would be recorded on multiple tracks/channels, wherein each track/channel contains the voice of just one person);
detect, using a first machine learning model, one or more moments of interruption between the first and second users from the audio file(Para [0059], Ln 1-10, training and testing models, the system is able to process and extract and generate hundreds (100's), or more, of unique features from the speech data in phone calls with MFCC extraction, feature splicing….. and DNN training. Para [0031], Ln 1-6, system preferably utilizes a feature extraction engine with pattern recognition and classification techniques to extract and recognize salient features in the the audio signal data. Para [0030], Ln features: 1) Caller's Speech Emotion Readings measured over the duration of the call. The emotion readings, also known as SER (abbreviation for Speech Emotion Recognition) labels, also contain temporal information;……3) Number of interruptions, as indicated by the amount of overlap between caller's voice and the agent's voice. Para [0052], Ln 1-10, training and pattern recognition approach to the system emotional labeling, feature extraction and classification, the system preferably applies MFCC extraction on the sample by feature splicing the signal. Thereafter LDA+MLLT transformation are preformed, i.e., linear discriminant analysis and maximum likelihood linear transform. Hidden markov model (HMM) training and deep neural network (DNN) training).
Haukioja does not teach when a threshold number of one or more moments of interruption corresponds to a first emotion type, classify the audio file as associated with user agitation; and when the threshold number of one or more moments of interruption does not correspond the first emotion type, classify the audio file as associated with user non-agitation.
In the same field of call monitoring, Sivasubramanian teaches when a threshold number of one or more moments of interruption corresponds to a first emotion type, classify the audio file as associated with user agitation(Para [0157], Ln 1-15, organizations can use contacts analytics output files for various use cases…set threshold associated with sentiments and define actions or categories based on certain thresholds being exceeded. For example, a run of N negative sentiments coupled with an overall negative sentiment score may be categorized as a bad interaction and a supervisor may be notified. Para [0116], Ln 16-22, categories can be defined based on content of communications as well as acoustic characteristics in the case of audio contacts. For example, calls may be categorized to identify instances of long silence, talking too fast, interruptions, more. Para [0042], Ln 9-17, calls are analyzed to extract different call characteristics which may include one or more of the following non-limiting examples: talk speed, interruptions, silence (e.g., gaps in speech), speaker energy, pitch, tone, and other voice characteristics. Para [0166], Ln 1-17, customers may exhibit positive sentiments (e.g., as illustrated in FIG. 18 by the customers with smiles) as well as negative sentiments (e.g., as illustrated in FIG. 18 by the angry customer));
and when the threshold number of one or more moments of interruption does not correspond the first emotion type, classify the audio file as associated with user non-agitation(Para [0157], Ln 1-15, organizations can use contacts analytics output files for various use cases…set threshold associated with sentiments and define actions or categories based on certain thresholds being exceeded. For example, a run of N negative sentiments coupled with an overall negative sentiment score may be categorized as a bad interaction and a supervisor may be notified. Para [0166], Ln 1-17, customers may exhibit positive sentiments (e.g., as illustrated in FIG. 18 by the customers with smiles) as well as negative sentiments (e.g., as illustrated in FIG. 18 by the angry customer)).
It would have been obvious for one skilled in the art, at the effective time of filling to modify Haukioja with the call center analytics system of Sivasubramanian, as it can help improve customer experience by notifying supervisors of poor customer experience(Para [0157], 1-15).
Regarding Claim 15:
The combination of Haukioja and Sivasubramanian teaches the system of claim 14, and Haukioja teaches wherein the first machine learning model comprises a neural network(Para [0052], Ln 1-10, training and pattern recognition approach to the system emotional labeling, feature extraction and classification, the system preferably applies MFCC extraction on the sample by feature splicing the signal. Thereafter LDA+MLLT transformation are preformed, i.e., linear discriminant analysis and maximum likelihood linear transform. Hidden markov model (HMM) training and deep neural network (DNN) training).
Regarding Claim 18:
The combination of Haukioja and Sivasubramanian teaches the system of claim 14, and Haukioja teaches wherein the audio file comprises a dual-channel audio file(Para [0007], Ln 1-7, Each call is recorded or monitored and sampled live in real-time by the system. Ideally, the calls would be recorded on multiple tracks/channels, wherein each track/channel contains the voice of just one person).
Claim(s) 10 and 16-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Haukioja and Sivasubramanian as applied to claim 9 and 15 above, and further in view of Krishnan et al. (US 20220093101 A1).
Regarding Claim 10:
The combination of Haukioja and Sivasubramanian teaches the system of claim 9, but does not specifically teach wherein the DNN comprises long short-term memory (LSTM).
In the same field of dialog monitoring, Krishnan teaches wherein the DNN comprises long short-term memory (LSTM)(Para [0470], Ln 1-8, user audio data 2015 may be input into the encoder component 2020 to determine frame feature vector(s) 2025. The encoder component 2020 may be a bidirectional LSTM.).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify the combination of Haukioja and Sivasubramanian with the LSTM based feature extraction encoder of Krishnan as it may prove the system’s ability to analyze the conversation(Para [0314], Ln 1-13).
Regarding Claim 16:
The combination of Haukioja and Sivasubramanian teaches the system of claim 15, but does not teach wherein the first machine learning model comprises a deep neural network (DNN) comprising long short-term memory (LSTM).
In the same field of dialog monitoring, Krishnan teaches wherein the first machine learning model comprises a deep neural network (DNN) comprising long short-term memory (LSTM)(Para [0470], Ln 1-8, user audio data 2015 may be input into the encoder component 2020 to determine frame feature vector(s) 2025. The encoder component 2020 may be a bidirectional LSTM.).
It would have been obvious for one skilled in the art, at the effective time of filling, to modify the combination of Haukioja and Sivasubramanian with the LSTM based feature extraction encoder of Krishnan as it may prove the system’s ability to analyze the conversation(Para [0314], Ln 1-13).
Regarding Claim 17:
The combination of Haukioja, Sivasubramanian and Krishnan teaches the system of claim 16, and Haukioja wherein the instructions are further configured to cause the system to: extract, using the neural network, one or more vocal features associated with each of the one or more moments of interruption(Para [0030], Ln 1-18, 3) Number of interruptions, as indicated by the amount of overlap between caller's voice and the agent's voice, see details below; 4) Emotional disparity during the interruption, in case the agent has to gently interrupt the caller to bring the conversation back into focus…… The system may use the Support Vector Machine Classifier (SVM) algorithm. Para [0052], Ln 1-10, training and pattern recognition approach to the system emotional labeling, feature extraction and classification, the system preferably applies MFCC extraction on the sample by feature splicing the signal. Thereafter LDA+MLLT transformation are preformed, i.e., linear discriminant analysis and maximum likelihood linear transform. Hidden markov model (HMM) training and deep neural network (DNN) training).
The combination of Haukioja, Sivasubramanian and Krishnan does not teach and determine, based on the one or more vocal features, whether a threshold number of moments of the one or more moments of interruption corresponds to a first emotion type, wherein the one or more vocal features comprise one or more of length of interruption, pre-interruption speaking rate, post-interruption speaking rate, pre-interruption speaking duration, post-interruption speaking duration, voice activity ratio, pre-interruption voice energy, and post-interruption voice energy.
In the same field of call monitoring, Sivasubramanian teaches and determine, based on the one or more vocal features, whether a threshold number of moments of the one or more moments of interruption corresponds to a first emotion type(Para [0157], Ln 1-15, organizations can use contacts analytics output files for various use cases…set threshold associated with sentiments and define actions or categories based on certain thresholds being exceeded. For example, a run of N negative sentiments coupled with an overall negative sentiment score may be categorized as a bad interaction and a supervisor may be notified. Para [0116], Ln 16-22, categories can be defined based on content of communications as well as acoustic characteristics in the case of audio contacts. For example, calls may be categorized to identify instances of long silence, talking too fast, interruptions, more. Para [0042], Ln 9-17, calls are analyzed to extract different call characteristics which may include one or more of the following non-limiting examples: talk speed, interruptions, silence (e.g., gaps in speech), speaker energy, pitch, tone, and other voice characteristics.),
wherein the one or more vocal features comprise one or more of length of interruption, pre-interruption speaking rate, post-interruption speaking rate, pre-interruption speaking duration, post-interruption speaking duration, voice activity ratio, pre-interruption voice energy, and post-interruption voice energy(Para [0042], Ln 9-17, calls are analyzed to extract different call characteristics which may include one or more of the following non-limiting examples: talk speed, interruptions, silence (e.g., gaps in speech), speaker energy, pitch, tone, and other voice characteristics. Para [0116], Ln 16-22, categories can be defined based on content of communications as well as acoustic characteristics in the case of audio contacts. For example, calls may be categorized to identify instances of long silence, talking too fast, interruptions, more).
It would have been obvious for one skilled in the art, at the effective time of filling to modify the combination of Haukioja, Sivasubramanian and Krishnan with the call center analytics system of Sivasubramanian, as it can help improve customer experience by notifying supervisors of poor customer experience(Para [0157], 1-15).
Allowable Subject Matter
Claims 1-7 would be allowable if rewritten, amended or a terminal disclaimer is filed, to overcome the double patenting rejection(s), set forth in this Office action, as well as the 112(b) rejection for Claim 5.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding Claim 1:
Haukioja teaches a system comprising: one or more processors; and a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to(Para [0011], Ln 1-8, computer readable medium comprising processor-executable code….. the processor-executable code may cause the processor to perform certain operations):
receive an audio file comprising a first channel and a second channel, the first channel comprising first voice activity of a first user and the second channel comprising second voice activity of a second user(Para [0007], Ln 1-7, Each call is recorded or monitored and sampled live in real-time by the system. Ideally, the calls would be recorded on multiple tracks/channels, wherein each track/channel contains the voice of just one person);
extract, using the DNN, one or more vocal features from the one or more moments of interruption(Para [0030], Ln 1-18, 3) Number of interruptions, as indicated by the amount of overlap between caller's voice and the agent's voice, see details below; 4) Emotional disparity during the interruption, in case the agent has to gently interrupt the caller to bring the conversation back into focus…… The system may use the Support Vector Machine Classifier (SVM) algorithm. Para [0052], Ln 1-10, training and pattern recognition approach to the system emotional labeling, feature extraction and classification, the system preferably applies MFCC extraction on the sample by feature splicing the signal. Thereafter LDA+MLLT transformation are preformed, i.e., linear discriminant analysis and maximum likelihood linear transform. Hidden markov model (HMM) training and deep neural network (DNN) training).
Haukioja does not teach determine, using a machine learning model and based on the one or more vocal features, whether a threshold number of moments of the one or more moments of interruption corresponds to a first emotion type; when the threshold number of moments corresponds to the first emotion type, transmit a first message comprising a first binary indication; and when the threshold number of moments does not correspond to the first emotion type, transmit a second message comprising a second binary indication.
In the same field of call monitoring, Sivasubramanian teaches determine, using a machine learning model and based on the one or more vocal features, whether a threshold number of moments of the one or more moments of interruption corresponds to a first emotion type(Para [0157], Ln 1-15, organizations can use contacts analytics output files for various use cases…set threshold associated with sentiments and define actions or categories based on certain thresholds being exceeded. For example, a run of N negative sentiments coupled with an overall negative sentiment score may be categorized as a bad interaction and a supervisor may be notified. Para [0116], Ln 16-22, categories can be defined based on content of communications as well as acoustic characteristics in the case of audio contacts. For example, calls may be categorized to identify instances of long silence, talking too fast, interruptions, more. Para [0042], Ln 9-17, calls are analyzed to extract different call characteristics which may include one or more of the following non-limiting examples: talk speed, interruptions, silence (e.g., gaps in speech), speaker energy, pitch, tone, and other voice characteristics);
when the threshold number of moments corresponds to the first emotion type, transmit a first message comprising a first binary indication(Para [0157], Ln 1-15, organizations can use contacts analytics output files for various use cases…set threshold associated with sentiments and define actions or categories based on certain thresholds being exceeded. For example, a run of N negative sentiments coupled with an overall negative sentiment score may be categorized as a bad interaction and a supervisor may be notified);
and when the threshold number of moments does not correspond to the first emotion type, transmit a second message comprising a second binary indication(Para [0157], Ln 1-15, organizations can use contacts analytics output files for various use cases…set threshold associated with sentiments and define actions or categories based on certain thresholds being exceeded. For example, a run of N negative sentiments coupled with an overall negative sentiment score may be categorized as a bad interaction. (sending message is being interpreted under broadest reasonable interpretation, the identification of whether or not the threshold is met and the result of that being stored in memory would read on sending a message). Para [0258], Ln 1-11, executable program instructions for the general administration and operation of that server and includes a computer-readable storage medium (e.g., a hard disk, random access memory, read only memory, etc.) storing instructions that, if executed by a processor of the server, cause or otherwise allow the server to perform its intended functions).
It would have been obvious for one skilled in the art, at the effective time of filling to modify Haukioja with the call center analytics system of Sivasubramanian, as it can help improve customer experience by notifying supervisors of poor customer experience(Para [0157], 1-15).
The prior art alone or in combination does not teach detect, using a deep neural network (DNN), one or more moments of interruption between the first and second users from the audio file by classifying portions of the audio file as either a speech portion or a non-speech portion based on one or more phonetic representations mapped to each portion of the audio file.
Claims 2-7 depend on a claim containing allowable subject matter, and therefore also contain allowable subject matter.
Claims 11 and 19-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, as well as having the double patenting rejections overcome.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding Claim 11:
The combination of Haukioja, Sivasubramanian and Krishnan teaches the system of claim 10, but does not teach wherein the instructions are further configured to cause the system to: separate the dual-channel audio file into one or more portions; map each portion of the one or more portions to one or more phonetic representations; and classify each portion of the one or more portions as either a speech portion or a non-speech portion based on each portion’s respective one or more phonetic representations, wherein the one or more phonetic representations comprise one or more of vowels and consonants.
The prior art of record alone or in combination does not teach the above limitations.
Regarding Claim 19:
The combination of Haukioja and Sivasubramanian teaches the system of claim 18, but does not teach wherein the instructions are further configured to cause the system to: separate, using the first machine learning model, the dual-channel audio file into one or more portions; map each portion of the one or more portions to one or more phonetic representations; and classify each portion of the one or more portions as either a speech portion or a non-speech portion based on each portion’s respective one or more phonetic representations.
The prior art of record alone or in combination does not teach the above limitations.
Claim 20 depends on a claim containing allowable subject matter, therefore also contains allowable subject matter.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Mizumoto et al. (US 20160307571 A1)
Conversation analysis using speech interruption and using phonemes for voice recognition.
Sato et al. (US 20240321273 A1)
Overlapping speech detection with voice recognition using phoneme or kana units.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEXANDER G MARLOW whose telephone number is (571)272-4536. The examiner can normally be reached Monday - Thursday 10:00 am - 8:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richmond Dorvil can be reached at (571)272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEXANDER G MARLOW/ Assistant Examiner, Art Unit 2658
/RICHEMOND DORVIL/ Supervisory Patent Examiner, Art Unit 2658
Prosecution Insights