Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claims 1-18 are pending. Claims 1 and 14 are independent.
This Application was published as U.S. 20260018269.
Apparent priority: 10 July 2024.
Based on the Drawings, Claim 8 includes the key goal of the instant Application and it is suggested that this claim is included in the independent Claim and its subject further elaborated upon.
PNG
media_image1.png
560
620
media_image1.png
Greyscale
Claim Objections
Use of capitalization in the Claims should be avoided unless the capitalized words have a special meaning.
35 U.S.C. 112(f) Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Such claim limitation(s) is/are: “service management module” in Claims 15-16. These limitations are generic in the context of the art and don’t refer to any specific structure and only serve as placeholders for the structure that performs the associated function(s) without providing any information about what that structure is. MPEP 2181 I A says:
For a term to be considered a substitute for "means," and lack sufficient structure for performing the function, it must serve as a generic placeholder and thus not limit the scope of the claim to any specific manner or structure for performing the claimed function. It is important to remember that there are no absolutes in the determination of terms used as a substitute for "means" that serve as generic placeholders. The examiner must carefully consider the term in light of the specification and the commonly accepted meaning in the technological art. Every application will turn on its own facts.
Based on the ordinary skill in the art and description of functions of these components in the Specification, they refer to processors or a combination of processor and memory or to a combination of software and hardware.
PLEASE NOTE: This is NOT a rejection. Please don’t address it as a rejection. If the Applicant does not agree with the INTERPRETATION, he may argue or amend to replace the terms interpreted under 112(f) with structural terms such as “microphone” or “processor” as appropriately supported by the Specification. In the alternative, he may let the interpretation stand if the intent was to include a means plus function limitation in the Claim.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 12 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
There is insufficient antecedent basis for “the plurality of Dimensional Emotion Qualities” in the claim.
12. The computer-implemented method of claim 1, wherein the concordance-discrepancy model comprises the following steps:
i. selecting the plurality of Dimensional Emotion Qualities of the Reported Emotion;
ii. selecting the plurality of Dimensional Emotion Qualities of the Detected Emotion; and
iii. undertaking at least one of A or B:
A. determining that the Reported Emotion and Detected Emotion are in alignment with each other in terms of their Dimensional Emotion Qualities;
B. determining that the Reported Emotion and Detected Emotion are not in alignment with each other in terms of their Dimensional Emotion Qualities.
Additionally, use of the possessive such as “their” is discouraged. Instead use “first dimensional emotion qualities pertaining to the reported emotion” and “second dimensional emotion qualities pertaining to the detected emotion” and “determining that the first dimensional emotion qualities and the second dimensional emotion qualities are in alignment.” Note that “dimensional emotion qualities” is not defined and further note that when the nature and character of “dimensional emotion qualities” is not defined, then the meaning of “alignment” is not defined and can be interpreted broadly. Key terms need to be defined inside the claim language. Also, as provided above, avoid capitalization unless it pertains to a particular industry term such as BluetoothTM.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 6, 8, and 11-14 are rejected under 35 U.S.C. 103 as being unpatentable over Bennett (U.S. 20100036660) in view of and Hill (U.S. 20130121591).
PNG
media_image2.png
614
824
media_image2.png
Greyscale
.
PNG
media_image3.png
586
828
media_image3.png
Greyscale
PNG
media_image4.png
578
882
media_image4.png
Greyscale
PNG
media_image5.png
644
756
media_image5.png
Greyscale
Regarding Claim 1, Bennett teaches:
1. A computer-implemented method to generate a Reported Emotion, a Detected Emotion, and a Concordance-Discrepancy Report, the method comprising:
i. based on an occurrence of a prompt, recording a digital audio sample representing an input utterance spoken by a user via a microphone of a mobile computing device associated with the user; [Bennett, Figures 1 and 5 show the input of speech that may be from a recording. “[0029] … recording a set of responses from the first group of persons to the series of questions;…” Figure 3 “animated character to guide the user 157” may provide the “prompt” of the Claim soliciting input from the user. Figure 8 shows under “user interface agent 400” a microphone icon. Speech input of the various drawings implies a microphone. Bennet includes a “Front-end Client 110/150/302” but does not specify it as a mobile device.]
ii. generating the Reported Emotion by: [Bennett uses syntactic cues and lexical cue of emotion from the text of a recognized speech.]
a. extracting via one or more processors from the digital audio sample a transcript comprising a sequence of natural-language words corresponding to the digital audio sample using speech-to-text processing; and [Bennett uses both text of the speech and prosody (sound) of the speech to detect emotion. Figures 1, 3, 5, “Speech Recognition (SRE) 115/155/306.” “[0098] A speech recognition agent 402 handles the process of recognizing a speech utterance and outputting a stream of recognized text….”]
b. determining the Reported Emotion from the transcript using a natural-language processing model; [Bennett, Figure 3 and 8, “NL agent 190/310.” “[0099] Of the other agents illustrated in FIG. 8, the natural language agent 310 takes the recognized text and outputs a natural language representation of it….” Figure 1 syntactic cues and lexical cue of emotion out of 121: “[0073] Accordingly emotion detector 100 as shown works in parallel with the speech recognition processes. It consists of three main sections: (1) a prosody analyzer 118 which operates based on extracted acoustic features of the utterance; (2) a parts-of-speech analyzer 121 which yields syntactic cues relating to the emotion state; and (3) a trained classifier 125 that accepts inputs from the prosody analyzer 118 and the parts-of-speech analyzer and outputs data values which correspond to the emotion state embedded in the utterance.” See an example regarding detecting Certainty vs Doubt from the sentence of the speaker at [0077]-[0080].]
iii. generating the Detected Emotion by: [Bennett uses prosodical indicators of emotion from the sound/prosody of speech.]
a. extracting a set of acoustic features via the one or more processors from the digital audio sample; and [Bennett, Figure 1, “prosody analysis 118.” “[0072] … Like the NLQS distributed speech recognition process, a significant portion of the emotion modeling and detection is implemented at the client side by a prosody analyzer 118. …” “[0073] Accordingly emotion detector 100 as shown works in parallel with the speech recognition processes. It consists of three main sections: (1) a prosody analyzer 118 which operates based on extracted acoustic features of the utterance;…” See also extraction of MFCC in Figure 5 and [0089].]
b. processing the set of acoustic features to identify the Detected Emotion using an emotion detection model; and [Bennett, “[0075] The prosody analysis as noted above is preferably based on three key acoustic features--Fundamental Frequency (FO), Amplitude (RMS) and Duration (DUR), extracted in real-time from the utterance. …” See an example regarding detecting emotions of Certainty and Doubt from the sound of the speaker also in the table of [0065]. ]
iv. generating the Concordance-Discrepancy Report by: [Bennett, Figure 1, “CART Decision tree 125.” “[0074] The outputs of the prosody analyzer 118 and the parts-of-speech analyzer 121 are fed preferably to a trained CART classifier 125. This classifier 125 is trained with data obtained during the off-line training phase described previously. The data which populate the history file contained within the trained CART trees, 250 represent data values for the emotion cues derived from the sample population of test subjects and using the sample utterances common to the content in question….”]
a. analyzing the Reported Emotion and the Detected Emotion using a concordance-discrepancy model to determine if there is a concordance or a discrepancy between the Reported Emotion and the Detected Emotion; and Bennett, Figure 1, “CART Decision tree 125” teaches the “concordance-discrepancy model” of the Claim because it evaluates both the lexical/syntactical cues obtained from the text of the speech / Reported Emotion of the Claim and the prosodical cues /Detected Emotion of the Claim and is trained to output a conclusion regarding the emotion of the user.]
b. producing a Concordance-Discrepancy Report comprising the output of the concordance-discrepancy model. [Bennett, Figure 1, “CART Decision tree 125” is trained to output a conclusion regarding the emotion of the user.]
Bennett has trained its model to arrive at a decision for the emotion from the combination of two types of information. The instant Application is directed to a particular situation when the user is saying something about his feelings which may not be true such that the words and the voice may contradict each other.
Hill teaches a say-feel gap which teaches the concordance-discrepancy of the Claim:
i. based on an occurrence of a prompt, recording a digital audio sample representing an input utterance spoken by a user via a microphone of a mobile computing device associated with the user; [Hill, Figure 1, the device 120 is shown in Figure 12 and may be a mobile phone or a distributed system of client and server and a phone inherently includes a microphone: “[0099] FIG. 12 illustrates a block diagram of an example of a machine 1200 upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform. In alternative embodiments, the machine 1200 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine 1200 may operate in the capacity of a server machine, a client machine, or both in server-client network environments. In an example, the machine 1200 may act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. The machine 1200 may be a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), or other computer cluster configurations.” “[0059] The audio processing module 405 can be configured to receive an audio stream of the subject 125. This audio stream can correspond in time to a sequence of visual observations of the subject 125. The audio processing module 405 can produce a transcript of speech uttered in the audio stream. For example, the audio processing module 405 can receive an audio track for video of the subject and produce a transcript of that speech. In an example, the transcript can be time-coded for later matching to the video images or for other purposes. …”]
…
iv. generating the Concordance-Discrepancy Report by: [Hill, Figure 4, the “Difference Module 425” teaches the “Concordance-Discrepancy model” of the Claim because it determines the difference/discrepancy between what the “subject 125” says and what her true emotions may be as reflected by her facial gestures.]
a. analyzing the Reported Emotion and the Detected Emotion using a concordance-discrepancy model to determine if there is a concordance or a discrepancy between the Reported Emotion and the Detected Emotion; and [Hill, Figures 4-5. A “say-feel gap” is determined between what the subject is saying and what she is actually feeling based on other emotional cues: “[0019] …. Emotional observations of the subject (politician) can be used in conjunction with semantic understanding of the politician's words or phrases to automatically augment the presentation to observers. A say-feel gap (e.g., the difference between the meaning of the word and the emotional state of the speaker) can be determined. This say-feel gap can be used to, for example, modify the pitch, tone, or magnitude of the speech's audio to indicate the confidence the speaker has in her words….” “[0058] FIG. 4 illustrates an example of a system 400 for say-feel gap identification and use. The system 400 can include an audio processing module 405, a sematic processing module 410, an image processing module 415, an emotion determination module 420, and a difference module 425. In an example, the system 400 can also include a presentation module 430. …”]
b. producing a Concordance-Discrepancy Report comprising the output of the concordance-discrepancy model. [Hill, “[0064] The presentation module 430 can be configured to present the correlation calculated by the difference module 425 to a user. In an example, the presentation module 430 can be configured to vary the intensity of the presentation based on the magnitude of the correlation. For example, if the image of flaming pants is applied to video of a speaker when the correlation indicates a say-feel gap, the size of the flames can be increased as the correlation decreases….”]
PNG
media_image6.png
398
498
media_image6.png
Greyscale
PNG
media_image7.png
340
492
media_image7.png
Greyscale
PNG
media_image8.png
486
372
media_image8.png
Greyscale
Bennett and Hill pertain to emotion detection and while Bennett uses a combination of emotion detected from the words of speech and emotion detected from the prosody of speech and Hill uses the emotion detected from the words of the speech and emotion detected from the visual facial gestures of the subject, Hill too acknowledges that emotion is reflected in prosody as well (“[0019] … Emotional observations of the subject (politician) can be used in conjunction with semantic understanding of the politician's words or phrases to automatically augment the presentation to observers. A say-feel gap (e.g., the difference between the meaning of the word and the emotional state of the speaker) can be determined. This say-feel gap can be used to, for example, modify the pitch, tone, or magnitude of the speech's audio to indicate the confidence the speaker has in her words. In this example, baseline operation (e.g., unmodified audio) can be presented of the speech until the promise, which the politician does not intend to keep. At this juncture, the audio can be changed to be lower in tone and quieter, to indicate the politician's lack of confidence. Conversely, the audio can be made louder to illustrate a portion of the speech in which the politician is particularly confident. In an example, the transcript of the speech can be presented to the observer. Strings (e.g., words or phrases) in the transcript can be marked (e.g., highlighted, enlarged, changed in color, etc.) to represent the emotional state of the speaker correlated to the string.”). It would have been obvious to modify the system of Bennett to add the “Difference Module 425” of Hill that measures the difference/correlation between what the subject is saying and what he/she is feeling. Or to modify the system of Hill to get the emotion from prosody instead of image. This combination falls under simple substitution of one known element for another to obtain predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Regarding Claim 2, Bennett has a client/server architecture but does not specify what its client device is.
Hill teaches:
2. The computer-implemented method of claim 1 whereby the mobile device comprises the microphone, the one or more processors, at least two digital storage units, and at least one digital display unit. [Hill, Figure 12 shows the components of the client/mobile device that is used and includes the components. See [0102]-[0103]: “[0102] … The machine 1200 may further include a display unit 1210, an alphanumeric input device 1212 (e.g., a keyboard), and a user interface (UI) navigation device 1214 (e.g., a mouse). In an example, the display unit 1210, input device 1212 and UI navigation device 1214 may be a touch screen display. …” “[0103] The storage device 1216 may include a machine readable medium 1222 on which is stored one or more sets of data structures or instructions 1224 (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions 1224 may also reside, completely or at least partially, within the main memory 1204, within static memory 1206, or within the hardware processor 1202 during execution thereof by the machine 1200. In an example, one or any combination of the hardware processor 1202, the main memory 1204, the static memory 1206, or the storage device 1216 may constitute machine readable media.”]
Bennett and Hill are directed to emotion detection and it would have been obvious to use the device detailed in Hill as the client device of Bennett which is not described in detail.
Regarding Claim 6, Bennett does not include a telephone expressly and does not teach that it is prompting for a specific input.
Hill teaches and the teachings suggest:
6. The computer-implemented method of claim 1, wherein the prompt relates to one and only one of:
(a) a phrase shown on the display unit of the device prompting the user to say what they are feeling, [Hill teaches this limitation. The subject may react to a stimulus presented to him such as an article or a post. [0028].]
(b) the initiation of an outbound telephone call on the device, or [Hill teaches the collection of speech on a device which may be a telephone and thus would pertain to inbound or outbound speech. The initiation is not expressly taught but is suggested by the use of the device.]
(c) the acceptance of an inbound telephone call to the device. [Hill teaches the collection of speech on a device which may be a telephone and thus would pertain to inbound or outbound speech.]
Rationale as provided for Claim 1.
Regarding Claim 8, Bennett deduces the emotion from the language but does not include an express example of when the user says I am angry or I am happy.
Hill teaches and the teachings suggest:
8. The computer-implemented method of claim 1, wherein the transcript of the input utterance spoken by the user contains a description of the emotion the user reports experiencing. [Hill suggests this by a combination of the nurse-patient scenario because a nurse is likely to ask how the patient feels and the politician scenario where the politician may be lying. “[0017] In an example the user and subject can be different, such as an application that monitors patients in a hospital. In this example, the user can be a nurse while the subject is a patient. Visual (or non-visual) emotional cues can be observed by a patient monitor and an emotional state for the patient can be determined. ….” “0019] An example application for observed emotional data is in measuring the veracity of a person's oral representations. For example, observed emotional data can be used to determine if a politician delivering a promise at a campaign event believes in the promise, or in her ability to deliver on the promise. …” The combination of these two teachings suggest that the patient may say something about how she feels and the veracity of her statements may be checked.]
Rationale for combination with Bennett as provided for Claim 1.
Regarding Claim 11, Bennett teaches:
11. The computer-implemented method of claim 1, wherein the emotion detection model comprises the following steps:
i. determining a feature vector corresponding to the digital audio sample, wherein the feature vector comprises the set of acoustic features; [Bennett, “[0024] … The SR process typically transfers speech data from an utterance to be recognized using a packet stream of extracted acoustic feature data including at least some cepstral coefficients. …” “[0071] … he Wagon Cart requires a special structure of input--a prosodic feature vector (PFV)--i.e a vector that contains prosodic features in both predictor and predictees. Each row of this prosodic feature vector represents one predictee (a part of the PFV that has information about the class value, e.g. the accented class), and one or more predictors, each row having the same order of the predictors with the predictee as the first element in the row….” “[0059] …For example in many cases it may be useful to incorporate certain acoustic features (such as MFCCs, Delta MFCCs) changes in energy, and other well-known prosodic related data.”]
ii. processing the feature vector as input to a trained multi-label classification neural network, the multi-label classification neural network configured to produce a plurality of Dimensional Emotion Qualities; and [Bennett, the CART classifier is a probabilistic multi-label classifier and produces a variety of emotion labels that teach the “dimensional emotion qualities” of the Claim. See also Figure 4 for a classic Dimensional Emotion wheel. “[0073] Accordingly emotion detector 100 as shown works in parallel with the speech recognition processes. It consists of three main sections: (1) a prosody analyzer 118 which operates based on extracted acoustic features of the utterance; (2) a parts-of-speech analyzer 121 which yields syntactic cues relating to the emotion state; and (3) a trained classifier 125 that accepts inputs from the prosody analyzer 118 and the parts-of-speech analyzer and outputs data values which correspond to the emotion state embedded in the utterance.” “[0074] The outputs of the prosody analyzer 118 and the parts-of-speech analyzer 121 are fed preferably to a trained CART classifier 125….” The Dimensional Emotion Qualities are taught by Figure 4 and “[0061] … These questions are designed so that the expected elicited answers aided by visual cues exhibit emotions of CERTAINTY, UNCERTAINTY and DOUBT. For example, questions that have obvious answers typically will have a response that is closely correlated to the emotion state of CERTAINTY and can be ascribed to be present in more than 90% of the answers, whereas questions which are difficult will elicit answers from which the person is not sure of and therefore contain the UNCERTAINTY emotion also in greater than 90% of the cases. The formulation of the questions can be performed using any of a variety of known techniques.” See Table 1 in [0065] for the Emotions and their indicators. “[0100] The prosody modeler agent 308 performs the functions described above. It should be noted that although CART decision trees are described above, prosody modeler agents according to embodiments of the invention may use any of a variety of machine learning algorithms as classifiers to classify acoustic data and map acoustic correlates to the prosodic structure, including boosting (AdaBoost), artificial neural networks (ANN), support vector machines (SVM) and nearest neighbor methods.”]
iii. storing the plurality of Dimensional Emotion Qualities as the Detected Emotion in the second of the at least two digital storage units. [Bennett, “[0069] … The best predictions based on the training data are stored in the leaf nodes of the CART.” “[0048] …Instead, embodiments of the invention may use data files, databases, or any other form of data repository to store data, and the term "data repository" should be read broadly to include any type of data storage.”]
PNG
media_image9.png
622
760
media_image9.png
Greyscale
Regarding Claim 12, Bennett does not evaluate the correlation/concordance of the emotion detected by the two different methods.
Hill teaches:
12. The computer-implemented method of claim 1, wherein the concordance-discrepancy model comprises the following steps:
i. selecting the plurality of Dimensional Emotion Qualities of the Reported Emotion; [Hill Figure 3 is like Figure 4 of Bennett and shows the DEQs of the Claim. Hill teaches the use of the magnitude of a feeling in calculating the correlation: “[0028] … That is, whether the subject 125 liked, disliked, or was neutral (in an example also including the magnitude of this feeling) on the article can be used to later inform others (e.g., friends, marketers, advertisers, etc.) of the subject's feelings towards the article.”]
ii. selecting the plurality of Dimensional Emotion Qualities of the Detected Emotion; and [Hill: “0024] The emotion determination module 110 can be configured to determine an emotional state of the subject 125 based on the sequence of visual observations. The emotion determination module 110 can be configured to automatically determine the emotional state based on an emotional determination system. Some examples of such systems are discussed below with regard to FIG. 3. However, any system by which an emotional state for a subject 125 can be determined by observing the subject 125 can be used. …”]
iii. undertaking at least one of A or B: [Hill performs both.]
A. determining that the Reported Emotion and Detected Emotion are in alignment with each other in terms of their Dimensional Emotion Qualities; [Hill is looking to check the veracity of a statement based on the detected emotion. “[0058] FIG. 4 illustrates an example of a system 400 for say-feel gap identification and use. The system 400 can include an audio processing module 405, a sematic processing module 410, an image processing module 415, an emotion determination module 420, and a difference module 425. In an example, the system 400 can also include a presentation module 430. Any one or more of these modules can be on a single device, or spread among several devices (such as in a cloud computing environment).”]
B. determining that the Reported Emotion and Detected Emotion are not in alignment with each other in terms of their Dimensional Emotion Qualities. [Hill, Figure 4, “Difference Module 425” determines whether there is a say-feel gap or not. “[0041] FIG. 3 illustrates an example of an emotional state chart 300. The chart 300 illustrates several useful emotional concepts for use in the systems and methods described herein. Engagement (e.g., impact or intensity) represents the degree of emotional response by a subject. For example, without regard to the type of emotion observed, high engagement represents strong emotion while a low engagement represents little or no emotional response. Appeal (e.g., valence) represents whether the emotion was positive or negative….”]
Rationale for combination as provided for Claim 1. Hill was combined for this feature.
Regarding Claim 13, Bennett is not looking for discrepancy.
Hill teaches and the teaching suggests:
13. The computer implemented method of claim 5, wherein a harmony metric is computed by taking the ratio of discrepancies to the sum of discrepancies and concordances in the history of digital output reports. [Hill teaches the say-feel gap which teaches the “discrepancy/concordance” of the Claim. The “say-feel gap” of Hill is a correlation value. ([0019] … Emotional observations of the subject (politician) can be used in conjunction with semantic understanding of the politician's words or phrases to automatically augment the presentation to observers. A say-feel gap (e.g., the difference between the meaning of the word and the emotional state of the speaker) can be determined. …” “[0064] The presentation module 430 can be configured to present the correlation calculated by the difference module 425 to a user….” “[0063] The difference module can be configured to calculate a correlation value for the string. In an example, the correlation value can be calculated by comparing the meaning of the string to the determined emotional state of the subject 125 for the sequence of visual images corresponding to the utterance of the string. In an example, the correlation is a binary value. For example, the correlation can simply indicate that the spoken meaning of the words does, or does not, match the observed emotional state of the subject 125. In an example, the correlation can include a magnitude component. In an example, the correlation can include one or more of an engagement component, summarization component, or emotional response component (e.g., as discussed above with regard to FIG. 3).” Under the method of Hill, when the correlation is 1 there is concordance and when 0 there is discrepancy. Hill does not teach a “harmony metric” according to the ratio that is set forth. However, the harmony ratio of the Claim is a mere calculation of a percentage of discrepancies / correlation =0 with respect to all of the events. It is the ratio of A/(A+B). Hill has the A and B and the formula for calculating a percentage is an obvious way of expressing how frequent an event is.]
Rationale as provided for Claim 1.
Regarding Claim 14, Bennett teaches:
14. A system for generating a Reported Emotion, a Detected Emotion, and a Concordance-Discrepancy Report on the Reported Emotion and the Detected Emotion, comprising a mobile device associated with a user and a server with a connection to the mobile device. [Bennett as applied to Claim 1 teaches all of the limitations of this Claim except for the “mobile device” which is not expressly taught while implied by the client of Bennett. This Claim does not include the particularity of the Claim 1 and accordingly the “Concordance Discrepancy Report” is mapped to the CART classifier of Bennett.]
As applied to Claim 1, the Mobile device was taught by Hill.
PNG
media_image10.png
590
826
media_image10.png
Greyscale
Claims 3-5, 9, and 15-18 are rejected under 35 U.S.C. 103 as being unpatentable over Bennett and Hill in view of Chakrabarty (U.S. 20250378945).
Regarding Claim 3, Bennett teaches and the teachings suggest:
3. The computer-implemented method of claim 1 further comprising:
i. generating a digital output report comprising: [Hill Figure 4, “Presentation Module 430” and Figure 5, “Present Correlation to User 535.”]
(a) a unique identifier of the user's device, [Hill suggests this by teaching: “[0030] In an example, the application can be an interactive application. An interactive application refers to continually inputting information by the subject 125 and a user interface that is responsive to such inputs. Examples of such interactive applications can include games (e.g., video or computer games), forms (e.g., data entry into one or more fields, etc.), productivity applications (e.g., word processors, spreadsheets, presentation tools, graphical drafting tools, etc.), among others. …” For a gaming application the device id is important.]
(b) the user's name, [Hill discusses a scenario of healthcare worker and patient where the user and the subject are different which suggests that the report needs to have the name of the subject. “[0017] In an example the user and subject can be different, such as an application that monitors patients in a hospital. In this example, the user can be a nurse while the subject is a patient. Visual (or non-visual) emotional cues can be observed by a patient monitor and an emotional state for the patient can be determined. If, for example, the emotional state indicates surprise, alarm, or other emotional state that can indicate a problem, the monitor may changes to alert (e.g., via sound, changed text, changed colors, paging the user, etc.) the nurse….”]
(c) the user's email address, [Hill suggests this by teaching: “[0028] In an example, the application can be a social media application. Such an application can include an interface in which the subject 125 (user in this scenario) can post information (e.g., biographical information, pictures, group affiliations, etc.) about themselves, identify various circles of trust (e.g., friends) with other members, play games, etc. …” For a social media application the user email is generally used.]
(d) a timestamp indicating when the digital audio sample was received by the user's device, [ Hill, “[0059] … In an example, the transcript can be time-coded for later matching to the video images or for other purposes. …” Because the timestamp is generated and also the teachings regarding the display of the video in [0077] actually teach the use of the timestamp for alignment of text and detected emotion and therefor indirect presentation of timestamp and also suggest presentation of the timestamp directly.]
(e) the transcript, [Hill Figure 5, “produce transcript of speech 510.” In one type of presentation, the text/transcript is presented with graphics to indicate the emotion that is detected: “[0076] In an example, presenting the correlation can include presenting a visual indication of the emotional state in a representation of the transcript corresponding to the string. In this example, one or more strings of text can be produced from the audio stream. These strings can be presented to the user. …” ]
(f) the Reported Emotion, [Hill Figure 5, “determine meaning of string in transcript 515.” This teaches the “reported emotion” of the Claim which is what the subject expresses as his/her emotion. The “reported emotion” of the Claim would be in the text that is presented: [0076] above.]
(g) the Detected Emotion, and [Hill Figure 5, “determine emotional state of the subject 525” based on observations at 520 which teaches the “detected emotion” of the Claim. “[0076] … For example the presentation can include closed-captioning in video streams or written transcripts (e.g., as on a webpage), among others. In an example, the string can be color coded (e.g., highlighted in a particular color) to represent the emotional state. In an example, the string can be enlarged or shrunk to represent the emotional state. Other modifications to the string, such as changing the font, replacing the string with a graphic, adding an accompanying graphic, can also be used to represent the emotional state.”]
(h) the Concordance-Discrepancy Report; and [Hill Figure 5, “Present Correlation to User 535.” [0075]-[0079] detail the variations in the ways that the correlation/concordance-discrepancy is presented.]
ii. displaying for the user via the at least one display unit of the device associated with the user a Generative Artificial Intelligence (AI) Large Language Model (LLM) interpretation of the Concordance-Discrepancy Report.
Bennett is from pre-LLM era and Hill does not teach the use of an LLM either.
Chakrabarty teaches:
ii. displaying for the user via the at least one display unit of the device associated with the user a Generative Artificial Intelligence (AI) Large Language Model (LLM) interpretation of the Concordance-Discrepancy Report. [Chakrabarty: Figure 4, the output of the “Detect Sentiment” and “NLP” is input to “Using LLM to create embeddings” and to “build prompt” for “generative AI” which is providing back Audio and Visual presentation to the user. See also “Use LLM to generate content” which is part of the loop. “[0064] (1) Embeddings and Content Search—Using the LLM embeddings, the system may search its knowledge base to find relevant information. [0065] (2) Building Prompts and Generative AI—Custom prompts are built to facilitate the Generative AI in creating responses or documentation that are contextually relevant and personalized to the patient's needs.” “[0035] … The user device may also include a display for presenting information to the user, for example, an LCD screen. …”]
PNG
media_image11.png
764
544
media_image11.png
Greyscale
Bennett/Hill and Chakrabarty pertain to detection of emotion from speech and text and it would have been obvious to add the interpretation and presentation feature of Chakrabarty which uses an LLM to the system of combination to modernize the system of combination. This is a tandem addition of a post processing step to the main portion of the task. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Regarding Claim 4, Bennett teaches:
4. The computer-implemented method of claim 3, wherein the Generative AI LLM interpretation of the Concordance-Discrepancy Report comprises the output from an application programming interface (API) to the Generative AI LLM using a natural-language prompt that asks for a clear and simple rewrite of the Concordance-Discrepancy Report. [Bennett, Figure 7 showing the APIs for contacting different applications 372, 376. “[0095] FIG. 7 is a schematic diagram illustrating the agent concept in more detail. In FIG. 7, requesting agents or requesters 370 control and execute a number of applications 372, each application 372 having an application programming interface (API). The requesters 370 make queries of any one of a number of providing agents or providers 374 as necessary using ICL. The providers 374 control and execute their own respective applications 376 to provide answers to the queries. A facilitating application or facilitator 378 facilitates the communication between the agents 370, 374, and a meta-agent 380 controls the facilitator 378.”]
PNG
media_image12.png
488
482
media_image12.png
Greyscale
Bennett and Hill do not teach the use of a post-processing stage by an LLM.
Chakrabarty teaches:
wherein the Generative AI LLM interpretation of the Concordance-Discrepancy Report comprises the output from an application programming interface (API) to the Generative AI LLM using a natural-language prompt that asks for a clear and simple rewrite of the Concordance-Discrepancy Report. [Chakrabarty teaches the use of the LLM as an interpretive post-processor as shown in the mapping of Claim and also teaches the generation of Prompt for input to the LLM as shown in Figure 3 of this reference shown above. “[0077] … The system is designed to be integrated with existing hospital and EHR database systems like EPIC through secure API calls. …” See also Table 1 for API calls.]
Rationale as provided for Claim 3. This is an implementation detail for contacting or connecting to other applications.
Regarding Claim 5, Bennett teaches:
5. The computer-implemented method of claim 3 further comprising repeating steps i and ii to produce a history of digital output reports and a history of Generative AI LLM interpretations of the Concordance-Discrepancy Reports.[ Bennett, teaches that its training is based on previously collected history. Figure 5, “History 316.” “[0096] In system 300 of FIG. 5, the dialogue manager agent 312 acts as a meta agent. More particularly, it monitors the execution of dialogue strategies and is able to change plans as unplanned events occur. In general, it interweaves high-level tutorial planning with adaptive on-the-fly plans. As shown in FIG. 5, the dialogue manager agent 312 interfaces with agents that handle inference 314 and history-tracking 316 tasks, as well as a knowledge base agent 318 that accesses sources and models….” “19. The system of claim 18, wherein said classifier uses a history file containing data values for emotion cues derived from a sample population of test subjects and using a set of sample utterances common to content associated with the real-time recognition system.” ]
Bennett and Hill do not teach the LLM agent of the Claim.
Chakrabarty as applied to Claim 3 teaches the use of the LLM which operates in place of the “dialogue manage 312” of Bennett.
Rationale is a rationale of substitution or that provided for Claim 3.
Regarding Claim 9, Bennett teaches:
9. The computer-implemented method of claim 1, wherein the natural-language processing model comprises the following steps: [Bennett: “[0093] In a particularly advantageous embodiment, the system 300 may be constructed as a number of modular components functioning as software agents and adhering to the SRI Open Agent Architecture (OAA) framework. The SRI Open Agent Architecture (OAA) is a framework for integrating the various components that comprise a spoken dialogue system. … The term agent refers to a software process that meets the conventions of the OAA framework, where communication between each agent using the Interagent Communication Language (ICL) is via a solvable, a specific query that can be solved by special agents. Each application agent as shown can be interfaced to an existing legacy application such as a speech recognition engine or a library via a wrapper that calls a pre-existing application programming interface (API).”]
i. submitting the transcript to the API to the Generative AI LLM using a natural-language prompt asking which of a set of emotions most closely fits the transcript; [Bennett in Figure 7 teaches using APIs to access outside programs and to submit questions and receive responses from them. “[0095] FIG. 7 is a schematic diagram illustrating the agent concept in more detail. In FIG. 7, requesting agents or requesters 370 control and execute a number of applications 372, each application 372 having an application programming interface (API). The requesters 370 make queries of any one of a number of providing agents or providers 374 as necessary using ICL. The providers 374 control and execute their own respective applications 376 to provide answers to the queries. A facilitating application or facilitator 378 facilitates the communication between the agents 370, 374, and a meta-agent 380 controls the facilitator 378.”]
ii. selecting the output of the API to the Generative AI LLM as the Emotion Category; [Bennett, Figure 7, “[0035] … The providers 374 control and execute their own respective applications 376 to provide answers to the queries….”]
iii. submitting the Emotion Category to the API to the Generative AI LLM using a prompt, the prompt asking which dimensional emotion qualities are associated with the aforementioned Emotion Category; [Bennett, Figure 7, “[0035] … requesting agents or requesters 370 control and execute a number of applications 372, each application 372 having an application programming interface (API). ….”]
iv. selecting the output of the API to the Generative AI LLM as a plurality of Dimensional Emotion Qualities; and [Bennett, Figure 7, “[0035] … The providers 374 control and execute their own respective applications 376 to provide answers to the queries….”]
v. storing the plurality of Dimensional Emotion Qualities as the Reported Emotion in the first of the at least two digital storage units. [Bennett, Figure 4 teaches the “dimensional emotion qualities” and Figure 5, “History 316” teaches the storing. “[0055] A key concept in emotion theory is the representation of emotion as a two-dimensional activation--evaluation space. As seen in FIG. 4, the activation of the emotion state--the vertical axis, represents the activity of the emotion state, e.g. exhilaration represents a high level of activation, whereas boredom involves a small amount of activation. The evaluation of the emotion state--the horizontal axis, represents the feeling associated with the emotional state. For example, happiness is a very positive, whereas despair is very negative. Psychologists [see references 1, 2, 3, 4, 5 above] have long used this two dimensional circle to represent emotion states. The circumference of the circle defines the extreme limits of emotion intensity such as bliss, and the center of the circle is defined as the neutral point. Strong emotions such as those with high activation and very positive evaluation are represented on the periphery of the circle. An example of a strong emotion is exhilaration, an emotional state which is associated with very positive evaluation and high activation. Common emotions such as bored, angry etc. are placed within the circle at activation-evaluation coordinates calculated from values derived from tables published by Whissell referenced above.” “[0094] As one example, cognitive reasoning models implemented within cognitive reasoning agents may be used to create models of tutors that can be embedded in the interactive learning environment. These cognitive reasoning agents monitor and assess the student's performance, coach and guide the student as needed, and keep a record of what knowledge or skills the student has demonstrated and areas where there is need for improvement. Cognitive reasoning agents may also create a profile or characterization of the student before and after the lesson.”]
Bennett interacts with outside applications and sends queries and receives answers but does not teach these applications to be LLMs. See [0093]. Neither does Hill.
Chakrabarty as applied to Claim 3 teaches the use of the LLM which operates in place of the various Applications of Figure 7 of Bennett that can Request information as Requesters 372 or provide the response as Providers 376. Chakrabarty as applied to Claim 3 and in its Figure 4, “Build Prompt,” shows the generation of the appropriate Prompt for the “Generative AI” as well. “[0064] (1) Embeddings and Content Search—Using the LLM embeddings, the system may search its knowledge base to find relevant information. [0065] (2) Building Prompts and Generative AI—Custom prompts are built to facilitate the Generative AI in creating responses or documentation that are contextually relevant and personalized to the patient's needs.”
Rationale is a rationale of substitution or that provided for Claim 3.
Regarding Claim 15, Bennett teaches:
15. The system of claim 14, wherein the mobile device associated with the user comprises:
i. a microphone, one or more processors, at least two digital storage units, at least one digital display unit, and the capacity to send and receive phone calls and text messages;
ii. a client application on the mobile device configured to accept spoken user input, display system outputs, send user data to the server, and send natural-language prompts to an API to a Generative AI LLM on the server; [Bennett Figure 7 showing the APIs for communicating with different applications or agents: “[0093] … Each application agent as shown can be interfaced to an existing legacy application such as a speech recognition engine or a library via a wrapper that calls a pre-existing application programming interface (API).” “[0095] FIG. 7 is a schematic diagram illustrating the agent concept in more detail. In FIG. 7, requesting agents or requesters 370 control and execute a number of applications 372, each application 372 having an application programming interface (API)….”
iii. a client application on the mobile device configured to execute a speech-to-text processing system and one or more trained multi-label classification neural networks, and to store digital output reports of the user; [Bennett Figure 3 teaches ASR and emotion detection being performed at either the Client or the Server sides. SRE Client Side 155. Bennett teaches a client application that performs ASR as shown in Figure 1 and includes a trained multi-label classification model as shown in Figures 2 and 5 which show the corpus and the process of training. Figure 1. “[0100] The prosody modeler agent 308 performs the functions described above. It should be noted that although CART decision trees are described above, prosody modeler agents according to embodiments of the invention may use any of a variety of machine learning algorithms as classifiers to classify acoustic data and map acoustic correlates to the prosodic structure, including boosting (AdaBoost), artificial neural networks (ANN), support vector machines (SVM) and nearest neighbor methods.”]
iv. a battery for providing power to the mobile device;
v. a network interface for establishing a connection with the server and configured to facilitate communication between the client application and the server; and [Bennett, the Client-Server arrangement requires communication over a network and Figure 3 shows the “internet 160” between Client side 150 and server side 180. “[0003] The invention relates to a system and an interactive method for detecting and processing prosodic elements of speech based user inputs and queries presented over a distributed network such as the Internet or local intranet. …”]
vi. a service management module configured to: [Bennett teaches the concept of executing the programs where more appropriate: “[0035] In other preferred embodiments an amount of prosodic data to be transferred to the server device is determined on a case by case basis in accordance with one or more of the following parameters: a) computational capabilities of the respective devices; b) communications capability of a network coupling the respective devices; c) loading of the server device; d) a performance requirement of a speech recognition task associated with a user query. Both prosodic data and acoustic feature data or other representative speech data may or may not be packaged within a common data stream as received at the server device, depending on the nature of the data, the content of the data streams, available bandwidth, prioritizations required, etc. Different payloads may be used for transporting prosodic data and acoustic feature data for speech recognition within their respective packets.”]
a. monitor the current network connectivity; [Bennett, “[0035] … b) communications capability of a network coupling the respective devices; …”]
b. monitor the battery life of the mobile device;
c. determine the complexity of the task to be processed; and [Bennett, “[0035] … d) a performance requirement of a speech recognition task associated with a user query….”]
d. dynamically switch the execution of the speech-to-text processing system, the execution of the one or more trained multi-label classification neural networks, and the storage of digital output reports of the user between the mobile device and the server based on the monitored network connectivity, mobile device battery life, and task complexity. [Bennett, Figure 3 in particular, determines where to conduct how much of the processing, which includes speech recognition and emotion detection, both being multi-label classification tasks, depending on the factors listed in [0035] above. “A prosody analyzer enhances the interpretation of natural language utterances. The analyzer is distributed over a client/server architecture, so that the scope of emotion recognition processing tasks can be allocated on a dynamic basis based on processing resources, channel conditions, client loads etc. The partially processed prosodic data can be sent separately or combined with other speech data from the client device and streamed to a server for a real-time response. …” Abstract. “[0045] … As seen in FIG. 3 the processing for NLQS 100 is generally distributed across a client side system 150, a data link 160, and a server-side system 180….” “[0049] Because the speech processing is broken up in this fashion, it is possible to achieve real-time, interactive, human-like dialog consisting of a large, controllable set of questions/answers….”
Bennett does not specify its Client device as a mobile device and is not particular on the hardware.
Hill teaches:
15. The system of claim 14, wherein the mobile device associated with the user comprises:
i. a microphone, one or more processors, at least two digital storage units, at least one digital display unit, and the capacity to send and receive phone calls and text messages; [Hill, Figure 12 and rejection of Claim 2.]
ii. a client application on the mobile device configured to accept spoken user input, display system outputs, send user data to the server, and send natural-language prompts to an API to a Generative AI LLM on the server; [Hill, Figure 12 and rejection of Claim 2. The use of a client/server architecture is taught by Hill as one of its possible configurations. [0099] … n a networked deployment, the machine 1200 may operate in the capacity of a server machine, a client machine, or both in server-client network environments….”]
iii. a client application on the mobile device configured to execute a speech-to-text processing system and one or more trained multi-label classification neural networks, and to store digital output reports of the user; [Hill, Figures 4-5. “Audio processing 405” which is speech recognition 510 of Figure 5 and presentation of the report to the user 450/535. Both speech recognition and emotion detection are multi-label classification tasks. “[0095] In an example, the emotional determination system can include an artificial intelligence (e.g., neural network) system trained against a number of faces. …” “[0043] Specific raw emotions, such as happiness, surprise, sadness, fear, anger, disgust, or contempt can also be used directly. Determining these raw emotions can be accomplished in a variety of ways. In an example, an artificial intelligence system can be trained on subject body or face. …”]
iv. a battery for providing power to the mobile device; [Hill teaches the types of devices it may use and many imply the presence of a battery. [0099].]
v. a network interface for establishing a connection with the server and configured to facilitate communication between the client application and the server; and [Hill, Figure 12, “network interface device 1220.” [0099] and [0102].]
…
Rationale as provided for Claim 1. Additionally, because battery life is a processing parameter it would have been obvious to include it in the considerations of the dividing the processing load between the Client and Server in Bennett.
Neither reference teaches one of their applications/agents to be a generative AI LLM.
Chakrabarty teaches:
ii. a client application on the mobile device configured to accept spoken user input, display system outputs, send user data to the server, and send natural-language prompts to an API to a Generative AI LLM on the server; [Chakrabarty as applied to Claim 3 teaches the use of a “Generative AI” in Figure 4 of this reference as receiving the results of sentiment detection and then providing an audio-visual presentation back to the user. Generation of the appropriate prompt is also shown in Figure 4.]
Rationale for combination as provided for Claim 3. LLM is used as a post-processor for presentation purposes.
Regarding Claim 16, Bennett teaches (this Claim is a parallel of Claim 15 and expresses the same factors only from the viewpoint of the server. The mapping of Claim 15 applies to this Claim as well. In Bennett the distributed computing is fluid between client and server sides; both can have all or some of the functions and are interchangeable. “…The analyzer is distributed over a client/server architecture, so that the scope of emotion recognition processing tasks can be allocated on a dynamic basis based on processing resources, channel conditions, client loads etc. The partially processed prosodic data can be sent separately or combined with other speech data from the client device and streamed to a server for a real-time response….” Bennett Abstract.):
16. The system of claim 14, wherein the server with a connection to the mobile device comprises:
i. a server processor configured to run an API to the Generative AI LLM, receive natural-language prompts for the AI LLM from the client application, and send responses back to the client application; [Bennett, Figures 1, 3, and 5 show a client-server configuration and Figure 7 shows that the server is communicating with different Application via APIs 372, 376.]
ii. a server processor configured to execute the speech-to-text processing system, execute the one or more trained multi-label classification neural networks, and store digital output reports of the user; [Bennett does teach that either the Client or the Server can perform ASR as shown in Figure 3: SRE Client Side 155 and SRE Server Side 182. Bennett includes a trained multi-label classification model for ASR and emotion detection as shown in Figures 2 and 5 which show the corpus and the process of training. The Dialog Manager (DM) is for reporting back to the user.]
iii. a network interface for establishing the connection with the mobile device and configured to facilitate communication between the client application and the mobile device; and [Bennett, the Client-Server arrangement requires communication over a network and Figure 3 shows the “internet 160” between Client side 150 and server side 180.]
iv. a service management module configured to: [Bennett, this is implied because its functions are performed by Bennett: “A prosody analyzer enhances the interpretation of natural language utterances. The analyzer is distributed over a client/server architecture, so that the scope of emotion recognition processing tasks can be allocated on a dynamic basis based on processing resources, channel conditions, client loads etc….” Abstract.]
a. communicate with the mobile device to receive data regarding network connectivity, mobile device battery life, and task complexity; and [Bennett, [0035] see rejection of Claim 15.]
b. accept the execution of the speech-to-text processing system and the one or more trained multi-label classification neural networks and the storage of the digital output reports of a user from the mobile device when determined to be optimal based on the received network connectivity data, mobile device battery life, and task complexity. [Bennett, [0035] see rejection of Claim 15.]
See mapping of Claim 15.
Hill is combined for a Mobile Device.
Chakrabarty is combined for LLM presenter.
Rationale as provided for Claim 15.
Regarding Claim 17, Bennett and Hill teach and the teachings suggest:
17. The system of claim 15 wherein the system is configured to execute the speech-to-text processing system, execute one or more trained multi-label classification neural networks, and store digital output reports of the user on the mobile device when the network connectivity is poor, the mobile device battery life is sufficient, and the task complexity is low. [Bennett, Abstract. This pair of Claims 17-18 are directed to concept of task distribution in a distributed computing environment (such as the Client-Server architecture of Bennett) where the tasks are distributed depending on certain conditions. See Rejection of Claim 15. Obviously, if connectivity is poor, and processing resources are sufficient for the task, the task is performed locally.]
Regarding Claim 18, Bennett and Hill teach and the teachings suggest:
18. The system of claim 16 wherein the system is configured to execute the speech-to-text processing system, execute one or more trained multi-label classification neural networks, and store digital output reports of the user on the server when the network connectivity is strong, the mobile device battery life is low, or the task complexity is high. [Bennett, Abstract. This pair of Claims 17-18 are directed to concept of task distribution in a distributed computing environment (such as the Client-Server architecture of Bennett) where the tasks are distributed depending on certain conditions. See Rejection of Claim 15. Obviously, if connectivity is high, and processing resources are insufficient for the task, the task is performed remotely at the server that has the better processing capabilities.]
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Bennett and Hill in view of Atlaf (U.S. 12525224).
Regarding Claim 7, Bennett and Hill teach the use of extracted acoustic features but not the specific type called by this Claim by name although the types of features used by Bennett and Hill include pitch and loudness that are included in the eGeMAPS.
Bennett teaches:
7. The computer-implemented method of claim 1, wherein the set of acoustic features correspond to the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS). [Altaf: “By using any number of techniques, the analytics server 202 may extract Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) low-level descriptor features from the audio to determine emotional features. The eGeMAPS may be a set of low-level descriptor features that are commonly used in the recognition of emotions in human voice. The eGeMAPS feature set may include different low-level descriptors, which capture various aspects of the acoustic signal such as pitch, loudness, spectral shape, and temporal dynamics. The eGeMAPS feature set may provide a comprehensive set of low-level descriptors to extract a wide range of acoustic features from human voice. By combining these features with machine learning algorithms and emotion models, the analytics server 202 may recognize or determine emotions in human voice.”]
Bennett/Hill and Atlaf pertain to detection of emotion from speech and it would have been obvious to use the particular type of acoustic features used by Atlaf that are stated as being particularly suited for emotion detection in place of the more generic acoustic features of the combination. This combination falls under simple substitution of one known element for another to obtain predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Bennett and Hill and Chakrabarty and further in view of Harper (U.S. 20210233623).
Regarding Claim 10, Bennett/Hill and Chakrabarty as applied to Claim 9 teaches the following portions of the Claim noting that the LLM is used as a black box where you provide an input query and it responds with an answer such that the Claim becomes a method of asking the LLM to determine a value without having invented the LLM:
10. The computer-implemented method of claim 1, wherein the emotion detection model comprises the following steps:
i. determining a feature vector corresponding to the digital audio sample, wherein the feature vector comprises the set of acoustic features: [Bennett: “[0067] Acoustic features or other representative speech features are extracted by a routine shown as 220….” “[0070] During the training phase of the CART decision tree 260, data is fed to the tree from a Prosodic Description File 240 and training data from Prosodic Feature Vectors 230 and the values of key parameters such as stop value and balance are optimized so that the output results of the tree have maximum correspondence with the results of the manual annotations.”]
ii. processing the feature vector as input to a trained multi-label classification neural network, the multi-label classification neural network configured to produce a plurality of emotion pairs, each emotion pair comprising an emotion name and an emotion score, wherein the emotion score of the emotion pair represents the probability that the digital audio sample expresses the named emotion of the emotion pair; [Bennett, the CART classifier of Bennett is a multi-label probabilistic classifier. "[0068] Decision tree classifiers, such as shown in FIG. 2, are probabilistic classifiers that transform data inputted to it into a binary question based on the attributes of the data that is supplied. …”]
iii. processing the emotion pairs one by one in an outer loop in accordance with a predetermined statistically significant threshold by undertaking at least one of A, B or C for each iteration of the loop:
A. determining that the emotion score in the emotion pair satisfies the threshold, determining that the score is the optimal such score so far, and storing the emotion name in the second of the at least two digital storage units;
B. determining that the emotion score in the emotion pair satisfies the threshold, and determining that the score is not the optimal such score so far;
C. determining that the emotion score in the emotion pair does not satisfy the threshold;
iv. selecting the emotion name in the second of the at least two digital storage units as the Emotion Category; [Bennett see Figure 4 for the emotion names/categories and their intensity value.]
v. submitting the Emotion Category to the API to the Generative AI LLM using a prompt, the prompt asking which dimensional emotion qualities are associated with the aforementioned Emotion Category; [Bennett Figure 7 using APIs to contact outside requester applications 372 in a modular arrangement of the system.]
vi. selecting the output of the API to the Generative AI LLM as the plurality of Dimensional Emotion Qualities; and [Bennett Figure 7 receiving the response from provider applications 376.]
vii. storing the plurality of Dimensional Emotion Qualities as the Detected Emotion in the second of the at least two digital storage units. [Bennett, Figure 1, History 316 indicating storage.]
Hill also teaches the use of outside applications: “[0024] … In an example, the emotion determination module 110 can be configured to relay application state information to an external source where the emotional state determination can be performed. The emotion determination module 110 can then communicate the emotional state determination to other modules of the device 120.”
Chakrabarty, Figure 4, shows the building of the prompt and submission to the generative AI to receive an audio-visual presentation of the results.
Bennett and Hill and Chakrabarty do not teach a score that is compared against a threshold.
Harper teaches:
ii. processing the feature vector as input to a trained multi-label classification neural network, the multi-label classification neural network configured to produce a plurality of emotion pairs, each emotion pair comprising an emotion name and an emotion score, wherein the emotion score of the emotion pair represents the probability that the digital audio sample expresses the named emotion of the emotion pair; [Harper detects emotion from speech and if the probability of classification of a certain emotion is above a threshold, it activates further processes: Figure 4: “[0046] According to one embodiment of the method 400, the classified user status can be an emotional or affective state of the user, or the status can be a physiological state, along with a probability of the classification. As such, if the probability is determined to exceed a predetermined threshold (YES in FIG. 4), in step 404, a cued health assessment is activated and is provided to a user at step 406. According to another embodiment, if the probability of the classification is determined to not have exceed the predetermined threshold (NO in FIG. 4), the method 400 activates an passive assessment in step 407 to continuously capture speech samples.”]
iii. processing the emotion pairs one by one in an outer loop in accordance with a predetermined statistically significant threshold by undertaking at least one of A, B or C for each iteration of the loop: [Harper, Figure 4.]
A. determining that the emotion score in the emotion pair satisfies the threshold, determining that the score is the optimal such score so far, and storing the emotion name in the second of the at least two digital storage units; [Harper, Figure 4. “Classify Health status 403” to “Clinically Actionable? 404” to Yes [Wingdings font/0xE8] emotion detected at a probability score above a threshold [Wingdings font/0xE8] determination is credible and will be given effect.]
B. determining that the emotion score in the emotion pair satisfies the threshold, and determining that the score is not the optimal such score so far;
C. determining that the emotion score in the emotion pair does not satisfy the threshold; [Harper, Figure 4. “Classify Health status 403” to “Clinically Actionable? 404” to No.]
Bennett/Harper/Chakrabarty and Harper pertain to detection of emotion from speech and it would have been obvious to use the threshold method of Harper with the system of combination to arrive at decisions regarding the importance of a detected emotion. This combination falls under combining prior art elements according to known methods to yield predictable results or use of known technique to improve similar devices (methods, or products) in the same way. See MPEP 2141, KSR, 550 U.S. at 418, 82 USPQ2d at 1396.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
LaBorde (U.S. 20230138557):
“[0220] The acoustic properties extracted from the audio signal may include a plurality of attributes enabling the assessment of the veracity of verbal statements using acoustic-prosodic features (e.g., formant frequencies, speech intensity) and lexical features (e.g., verb tense, use of negative emotion words) in utterances.
Adler (U.S. 20130173269):
PNG
media_image13.png
548
792
media_image13.png
Greyscale
Watanabe (U.S. 10891458):
PNG
media_image14.png
768
508
media_image14.png
Greyscale
PNG
media_image15.png
762
500
media_image15.png
Greyscale
Subramanian (US 7983910):
PNG
media_image16.png
628
788
media_image16.png
Greyscale
Osborne (U.S. 20200286505)
[0106] The two-dimensional emotion-mood space shown in FIG. 1 is an adaptation of the Geneva Emotion Wheel (Scherer 2005), where moods and emotions are manually arranged in a circular fashion by subjects manually assigning a location of their mood in the circular space. Low sympathetic autonomic arousal is located at the bottom of the wheel, while high sympathetic autonomic arousal is located at the top of the wheel. Low valence is valence located at the left and high valence is located at the right of the wheel.
PNG
media_image17.png
462
478
media_image17.png
Greyscale
Jin (US 20190122071):
[0029] In some embodiments, the different emotions can include discrete emotions. For example, in the field of emotion classification, there are six discrete emotion types, including joy, surprise, disgust, sadness, anger, and fear. In other embodiments, the different emotions can include discrete emotions as well as dimensional emotions. For example, dimensional models of emotion attempt to conceptualize human emotions by defining where they lie in two or three dimensions. Most dimensional models incorporate valence and arousal or intensity dimensions. Dimensional models of emotion suggest that a common and interconnected neurophysiological system is responsible for all affective states. These models contrast theories of basic emotion, which propose that different emotions arise from separate neural systems. The two-dimensional models that are most prominent are the circumplex model, the vector model, and the positive activation model.
Zimmerman (U.S. 20180124242):
PNG
media_image18.png
322
438
media_image18.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to FARIBA SIRJANI whose telephone number is (571)270-1499. The examiner can normally be reached 9 to 5, M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Fariba Sirjani/
Primary Examiner, Art Unit 2659