DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 1/21/26 has been entered.
Notice to Applicant
The following is a Non-Final Office action. In response to Examiner’s Final Rejection of 9/18/25, Applicant, on 1/21/26, amended claims. Claims 1-21 are pending in this application and have been rejected below.
Response to Amendment
Applicant’s amendments are acknowledged.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-21 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e. an abstract idea) without reciting significantly more.
Step One - First, pursuant to step 1 in MPEP 2106.03, the claim 1 is directed to an apparatus which is a statutory category.
Step 2A, Prong One - MPEP 2106.04 - The claim 1 recites–
“An apparatus for calibrating evaluator feedback relating to one or more agent-customer interactions based on corresponding customer feedback, which comprises … to implement operations comprising:
storing the customer feedback, which is provided via a customer questionnaire associated with an agent-customer interaction, …;
querying one or more …, in which the customer feedback is stored for:
one or more agent-customer interactions satisfying one or more user-selected filter criteria, the queried one or more agent-customer interactions including the agent-customer interaction with which the customer questionnaire is associated (see claim 3 below for alternatives -one or more user-selected filter criteria comprise channel type, duration of interaction, customer scoring of interaction, customer sentiment during interaction); and
the customer questionnaire associated with the agent-customer interaction included in the queried one or more agent-customer interactions, the queried customer questionnaire including the customer feedback;
generating an agent evaluation form for the agent-customer interaction based on the customer feedback included in the queried customer questionnaire associated with the agent-customer interaction;
communicating, to one or more user-selected evaluators, the generated agent evaluation form for the agent-customer interaction and one or more media streams of the agent-customer interaction;
visualizing, audibilizing, or both, …each accessible by at least one of the one or more user-selected evaluators:
the one or more media streams of the agent-customer interaction; and
the generated agent evaluation form; and
receiving, from the one or more user-selected evaluators, evaluator feedback via completion of the communicated agent evaluation form for the agent- customer interaction;
calculating a customer feedback variance score, for the one or more user-selected evaluators,
wherein the customer feedback variance score is calculated such that:
a relatively higher customer feedback variance score results from relatively less effective evaluation of the agent-customer interaction; and
a relatively lower customer feedback variance score results from relatively more effective evaluation of the agent-customer interaction, and wherein the customer feedback variance score is calculated by comparing:
the evaluator feedback received via completion of the communicated agent evaluation form for the agent-customer interaction by the one or more user-selected evaluators; to
the customer feedback provided via the customer questionnaire associated with the agent-customer interaction.”
As drafted, this is, under its broadest reasonable interpretation, within the Abstract idea grouping of “certain methods of organizing human activity” (managing relationships between people – including social activities, teaching, and following rules or instructions – we get feedback on employees/agents from customers and evaluate how the employee/agent is performing) and a mathematical relationships [calculating variance score by comparing two numbers, where a lower variance represents better effectiveness of evaluating agents/employees]. Here, we have a series of steps for using criteria (e.g. claim 3 – by customer scoring, chat/phone call, duration of interaction), to select agent-customer interactions, looking at the customer’s feedback in a questionnaire [e.g. are the customers satisfied?], generating an “agent evaluation form” based on the feedback (e.g. questions/content based on customer scoring for “comparing” at end of claim), communicating the form and a stream of interaction (which could just be written text or transcript from a “digital” channel type – see [0024] as published stating “digital” can be channel type for interaction) for selecting agent-customer interactions, then visualizing the media stream (e.g. the “digital” channel interaction, which can be a visual of the text of the chat), along with the generated evaluation form (to ask the same questions that were asked to the customer), and then calculating a customer feedback score by comparing the evaluator feedback and the customer feedback which is considered an explicit mathematical relationship; lower variance represents a better evaluation of the interaction with the customer. Accordingly, claim 1 is directed to an abstract idea.
Step 2A, Prong Two - MPEP 2106.04 - This judicial exception is not integrated into a practical application. In particular, the claim 1 recites additional elements that are:
An apparatus for calibrating evaluator feedback relating to one or more agent-customer interactions based on corresponding customer feedback, which comprises
“one or more non-transitory computer readable media and a plurality of instructions stored on the one or more non-transitory computer readable media and executable by one or more processors to implement operations comprising.”
storing the customer feedback, which is provided via a customer questionnaire associated with an agent-customer interaction, in a database, or databases;
querying one or more databases… for…;
communicating, to one or more user-selected evaluators, the generated agent evaluation form for the agent-customer interaction and one or more media streams of the agent-customer interaction;
visualizing, audibilizing, or both, “via one or more output devices,” each accessible by at least one of the one or more user-selected evaluators;
(MPEP 2106.05f applies – limitations in claim involve a computer, , and is considered “apply it” [the abstract idea] on a computer; merely uses a computer as a tool to perform an abstract idea; “storing” customer feedback; “querying” databases; “communicating” to evaluators displaying text from a “digital channel” [0024 as published] interaction is considered just “using a computer”) and having a display and a database queried [i.e. data storage] is “field of use” MPEP 2106.05h.
Accordingly, the additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim also fails to recite any improvements to another technology or technical field, improvements to the functioning of the computer itself, use of a particular machine, effecting a transformation or reduction of a particular article to a different state or thing, and/or an additional element applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception. See 84 Fed. Reg. 55. The claim is directed to an abstract idea.
Step 2B in MPEP 2106.05 - The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of a processor/computer, computer-readable medium having stored thereon instructions executed by the processor/computer, and “datastores” to execute operations are MPEP 2106.05(f) (Mere Instructions to Apply an Exception – “Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible.” Alice Corp., 134 S. Ct. at 235) and “field of use” (MPEP 2106.05h). Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept.
In addition, the steps involving “storing” data then “querying one or more databases for…” is viewed as conventional functions at step 2B (See MPEP 2106.05d(II)(iv) - Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306).
The claim fails to recite any improvements to another technology or technical field, improvements to the functioning of the computer itself, use of a particular machine, effecting a transformation or reduction of a particular article to a different state or thing, adding unconventional steps that confine the claim to a particular useful application, and/or meaningful limitations beyond generally linking the use of an abstract idea to a particular environment. See 84 Fed. Reg. 55. The claim is not patent eligible. Viewed individually or as a whole, these additional claim element(s) do not provide meaningful limitation(s) to transform the abstract idea into a patent eligible application of the abstract idea such that the claim(s) amounts to significantly more than the abstract idea itself.
Independent claim 8 is directed to a system at step 1, which is a statutory category. Claim 8 recites similar limitations as claim 1 and is rejected for the same reasons at step 2a, prong one, 2a, prong 2, and step 2b. The claim is not patent eligible.
Independent claim 15 is directed to a method at step 1, which is a statutory category. Claim 15 recites similar limitations as claim 1 and claim 8 and is rejected for the same reasons at step 2a, prong one; step 2a, prong 2 and step 2b.
Claims 2, 9, and 16 narrow the abstract idea by “visualizing” the score to a user. To extent it is communicate “via output devices each accessible by the at least one of the one or more user-selected evaluators,” this is considered “apply it [abstract idea] on a computer at step 2a, prong two and step 2B. In addition, passing information to over a network is considered a conventional computer function at step 2B – See MPEP 2106.05d(II) Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362. Claims 3, 10, and 17 narrow the abstract idea by naming filters for how interactions selected (e.g. customer scoring, duration of interaction), narrowing the abstract idea for mathematical relationships of thresholds. Claims 4, 11, and 18 narrow the abstract idea by having “additional” or second interactions; claims 5-6, 12-13, and 19-20 further narrow the abstract idea by then having a form of questions based on “additional” interactions and communicating to evaluators (people). Claims 7, 14, and 21 narrow the abstract idea by specifying how the customer feedback variance score is from the evaluator person completing the form compared to customer feedback from “additional” questionnaires.
Therefore, the claim(s) are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
For more information on 101 rejections, see MPEP 2106.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-21 are rejected under 35 U.S.C. 103 as being unpatentable over Mokashi (US 202/0311209), Vymenets (US 2016/0350699), and Alon (US 2021/0287263).
Concerning claim 1, Mokashi discloses:
An apparatus for calibrating evaluator feedback relating to one or more agent-customer interactions based on corresponding … feedback (Mokashi – see par 63-67 - The variance (e.g. a measure of how far numbers are spread out from their average value) may be a measure of how different the evaluator answers and agents are. In the case that two people, such as an evaluator and agent participate, an evaluator score (summing an evaluator rating from the ratings associated with all of the answers provided by the evaluator) and agent_score (summing an agent rating from the ratings associated with all of the answers provided by the agent) are created.), which comprises one or more non-transitory computer readable media and a plurality of instructions stored on the one or more non-transitory computer readable media and executable by one or more processors to implement operations comprising (Mokashi – see par 40 – embodiments include… computer or process non-transitory storage medium, for example a memory, including or storing instructions, e.g., computer-executable instructions, which, when executed by a processor or controller, carry out methods disclosed herein):
Mokashi discloses having questions for evaluations of an agent in an interaction with a customer (See par 19, 46). Mokashi further discloses sending a form to a third-party evaluation, being a person different from the agent (See par 53).
Vymenets and Alon disclose questions of an interaction being sent to a person that is considered explicitly a customer:
An apparatus for calibrating evaluator feedback relating to one or more agent-customer interactions based on corresponding “customer” feedback… (Vymenets – see par 62 – the agent (1-n) can meet with the evaluator for feedback. Additionally or alternatively, evaluation frequency can be based on customer survey feedback…;
see also Alon –see par 33 - In some embodiments, storage module 104 may store one or more databases relating to agent data (e.g. agent profiles, schedules, etc.), customer data (e.g. customer profiles, attributes, history, etc.), interaction data (e.g. details of each interaction with a customer)…);
storing the customer feedback, which is provided via a customer questionnaire associated with an agent-customer interaction, in a database, or databases (Vymenets – see par 62 – the agent (1-n) can meet with the evaluator for feedback. Additionally or alternatively, evaluation frequency can be based on customer survey feedback. For example, if a customer was not satisfied with a call for a determined agent 127(1-n), a frequency of evaluations can increase;
see also Alon –see par 33 - In some embodiments, storage module 104 may store one or more databases relating to agent data (e.g. agent profiles, schedules, etc.), customer data (e.g. customer profiles, attributes, history, etc.), interaction data (e.g. details of each interaction with a customer, including reason for the interaction, disposition data, time on hold, handle time, etc.), and the like. According to some embodiments, some of the data (e.g. customer profile data) may be maintained in a customer relations management (CRM) database hosted in storage module 104 or elsewhere).
Mokashi, Vymenets, and Alon disclose:
querying one or more databases, including the database, or databases, in which the customer feedback is stored (Mokashi – see par 26 - Interaction data or documents may be stored, e.g., in files and/or databases: for example logger 40, menus logger 42, and web-page logger 44 may record information related to interactions, such as the content or substance of interactions (e.g. recordings and/or transcripts of telephone calls) and metadata (e.g. telephone numbers used, customer identification (ID), etc.). In the case that documents other than interactions are used, other databases may be used; see par 44 – distribution definition may include search terms or filters…including KPIs or other quality or evaluative measures, such as sentiment;
see also Alon – see par 49 - questionnaire module 110 may match questions with topics based, e.g., on an assigned metric assessing the relevance of a score to the interaction topic. see par 50-51 - In some embodiments, the questionnaire module 110 may determine a subset of questions selected from the dataset based, at least in part, on a relevancy assessment score and/or related assessment. ) for:
one or more agent-customer interactions satisfying one or more user-selected filter criteria, the queried one or more agent-customer interactions including the agent-customer interaction with which the customer questionnaire is associated (see claim 3 below for alternatives -one or more user-selected filter criteria comprise channel type, duration of interaction, customer scoring of interaction, customer sentiment during interaction) (Mokashi – see par 42 - In operation 400, a quality plan may be created. For example, a computer module such as form creation tool 64 may receive accept or receive input from a person (e.g. using a browser or other local software on a computer or terminal) such as a form designer or manager to create a form which may later be integrated in a quality plan; see par 44 - A distribution definition may include document search terms or filters, such as only interactions longer or shorter than a certain time;… those in certain categories; and those of a certain type (e.g. voice call, text interaction); ... Search terms or filters may include KPIs or other quality or evaluative measures, such as sentiment (e.g. the overall emotional tone of the call, such as positive, neutral, hostile, negative). Search terms or filters may include multiple parameters; See par 50- For example, selection parameters may include interactions with a KPI below a certain threshold (disclosing scoring), interactions that resulted in another call from a customer, etc.
see also Alon – see par 52 - For example, if that number or percentage is below a certain threshold, then the question may have limited relevance to most customers and therefore may be deemed to be potentially unnecessarily burdensome to answer. In some embodiments, relevance may refer, for example, to a topic which comes up in very few interactions. In some embodiments, questions may be selected when a topic identified in an interaction correlates with one or more other topics; see par 55 - method illustrated in FIG. 4 may generate a dynamic survey tree and/or path, where a survey of questions may be dynamically constructed based on dynamic real-time analysis of responses, attributes information, and/or topics. For example, questions may be presented according to a dynamically adjustable question tree, whereby subsequent questions that are presented to the contacting customer depend on selections from previous questions; see par 58 - In some embodiments, to identify tracked topics that are relevant to questions, question and tracked topics may be represented using a vector. Thus, similarity may be based on detecting a word or phrase similarity based on the vector representations, wherein a similarity between a question and a topic is the sum of the similarities of the individual words herein. If the resulting sum exceeds a threshold value, then it is inferred that there is a connection between the question and the topic)
Mokashi discloses having questions for evaluations of an agent in an interaction with a customer (See par 19, 46). Mokashi further discloses sending a form to a third-party evaluation, being a person different from the agent (See par 53).
Vymenets and Alon disclose questions of an interaction being sent to a person that is considered explicitly a customer:
the customer questionnaire associated with the agent-customer interaction included in the queried one or more agent-customer interactions, the queried customer questionnaire including the customer feedback (Vymenets – see par 62 – the agent (1-n) can meet with the evaluator for feedback. Additionally or alternatively, evaluation frequency can be based on customer survey feedback. For example, if a customer was not satisfied with a call for a determined agent 127(1-n), a frequency of evaluations can increase
Alon – see par 28 - a survey questionnaire may be generated as part of a quality monitoring (QM) process. In the context of a contact center, QM refers to the process of evaluating contact center agents to measure and ensure the quality of the service provided by the agents. Typically, quality monitoring is performed to measure agent performance during interactions (e.g., calls, text chats, and email exchanges) between the agents and customers, on such metrics as politeness, demeanor, efficiency, and/or effectiveness.)
Mokashi and Vymenets disclose:
generating an agent evaluation form for the agent-customer interaction based on the customer feedback included in the queried customer questionnaire associated with the agent-customer interaction (Vymenets – see par 30 - A goal of the contact centers can be to provide quality customer service. Systems and methods can provide for a quality management platform that builds forms to help evaluate interactions by customers with agents at contact centers. See par 54 - The feedback can also be used to produce questions for future forms for the agent; The additional questions can be temporarily added for a determined time, permanent, updated based on agent performance, etc. see par 70 - The analytics server 137 can also determine a content of the interaction using analytics. The forms manager can add specific questions to the evaluation form based on the interaction data, timing of the interaction and/or other criteria, for example questions about an understanding level or customer comprehension of the interaction if the agent is not a native speaker
To extent that the “customer” feedback is used to build the “evaluation form,” Alon discloses:
generating an agent evaluation form for the agent-customer interaction based on the “customer” feedback included in the queried “customer” questionnaire associated with the agent-customer interaction (Alon – see par 29 - A QM evaluation may be triggered by an evaluator or a supervisor of the contact center, who is charged with evaluating an agent's performance. The present disclosure provides for initiating a customer feedback process to include customer experience and perspective in the QM process. see par 30 - an automated personalized survey questionnaire generation process may generate a survey questionnaire may be used to solicit or garner customer feedback. Such customer feedback may in turn be used in conjunction with a QM evaluation of an agent's performance during an interaction that the agent participated in. in some embodiments, the survey questionnaire may include one or more questions (e.g., “was the agent attentive” or “was the agent courteous”). In some embodiments, the questionnaire may also include various question types, e.g., yes/no questions, multiple choice questions, numerical value questions (e.g., on a scale from 1 to 5), or free-form questions).
Mokashi and Vymenets disclose:
communicating, to one or more user-selected evaluators, the generated agent evaluation form for the agent-customer interaction and one or more media streams of the agent-customer interaction (Mokashi – See par 20 - A computerized evaluation form (including a set of questions) may be associated with, or may be used with, a distribution definition such that a distribution definition selects certain interactions to be evaluated with the certain evaluation form; for each interaction selected, the agent involved (e.g. participating in the interaction) is chosen to evaluate the form along with another evaluator. For evaluations, different forms may be assigned to or used with different distribution definitions; see par 25, FIG. 1 - User terminals 18 may allow users, such as contact or data center personnel, managers, evaluators, etc., to create, configure, execute and view evaluations (e.g. created sets of evaluation questions and). User terminals 18, agent terminals 16, and possibly other equipment may be used to display data and receive data or human input as discussed herein. see par 30 - FIG. 2 depicts an example analysis center 50 which may perform functions such as those shown in FIG. 4; additional data may be stored in database 52, for example … ratings produced typically after the call; See FIG. 7, par 57 – showing “audio recording” 454 for controlling audio playback for interaction; and questions in evaluation form – 456 and answer fields 457;
See also Vymenets – See par 54 - The feedback can also be used to produce questions for future forms for the agent; The additional questions can be temporarily added for a determined time, permanent, updated based on agent performance, etc. see par 70 - The analytics server 137 can also determine a content of the interaction using analytics. The forms manager can add specific questions to the evaluation form based on the interaction data;
Applicant’s [0026] as published gives example “ The one or more evaluation forms are then generated (on the fly) by the evaluation form-builder module 110 based on customer feedback contained in the queried one or more customer questionnaires, as shown in FIG. 1A. In this manner, the evaluation form-builder module 110 automatically (re) designs evaluation (i.e., scoring) criteria based on inputs (i.e., feedback) provided by customers”
See also Alon – discloses the same example as Applicant– par 24 - the present disclosure provides for automated generation of a personalized survey questionnaire configured to assess a quality of a contact center interaction; par 30 - an automated personalized survey questionnaire generation process may generate a survey questionnaire may be used to solicit or garner customer feedback. Such customer feedback may in turn be used in conjunction with a QM evaluation of an agent's performance during an interaction that the agent participated in. The questionnaire may also include various question types, e.g., yes/no questions, multiple choice questions, numerical value questions (e.g., on a scale from 1 to 5), or free-form questions; see par 49-51 - questionnaire module 110 may match questions with topics based, e.g., on an assigned metric assessing the relevance of a score to the interaction topic. For example, a question relating to agent demeanor may assigned different scores based on a number and/or occurrence frequency of topics and/or subtopics associated with agent conduct and manner, and/or based on an analysis of the frequency with which the agent uses polite phrasing in the interaction.);
visualizing, audibilizing, or both, via one or more output devices each accessible by at least one of the one or more user-selected evaluators:
the one or more media streams of the agent-customer interaction (Mokashi – see par 52 - Documents or interactions may include for example audio recording (e.g. of telephone or other calls), text communications (SMS (short message service), WhatsApp messages, chats, etc.), video or other documents or files. See par 55 - each of the evaluator and agent may review the interaction or document; e.g. each may listen to the call or interaction (or read a transcript, view a video, etc., as appropriate) and provide answers (e.g. agent answers and evaluator answers each associated with a question) to questions. In one embodiment the document should be reviewed or the audio of the interaction listened to; See FIG. 7, par 57 – showing “audio recording” 454 for controlling audio playback for interaction; and questions in evaluation form – 456 and answer fields 457;); and
the generated agent evaluation form for the agent-customer interaction (Mokashi – see par 53 - In operation 430, the form along with the selected document or interaction may be distributed or assigned (e.g., to form executor module 67), substantially simultaneously, or concurrently, to the agent involved in the interaction or document (e.g. the agent who participated in the interaction) and one or more third-party evaluator(s), an evaluator being a person different from the agent; See FIG. 7, par 57 – showing “audio recording” 454 for controlling audio playback for interaction; and questions in evaluation form – 456 and answer fields 457);
receiving, from the one or more user-selected evaluators, evaluator feedback via completion of the communicated agent evaluation form for the agent- customer interaction (Mokashi – see par 58, FIG. 2, 4 - In operations 460 and 462, each of the evaluator and agent when finished reviewing the questions on the form, may finalize or submit the form, which causes in operation 470 the answers and scores, and/or in some embodiments one number which is the cumulative score, to be stored (e.g. in database 52). For example, form executor module 67 may receive or accept from the agent and evaluator (e.g. via agent terminals 16 and user terminals 18) a submission or indication indicating that the person has completed the computerized evaluation form. Data may be stored for example in a JSON document in a MySQL table which includes the information of the form, with answers and possibly scores added: e.g. a JSON or other document including questions and answers may be provided, and form executor module 67 may add actual responses—e.g. answers with scores—to the form.
see also Vymenets – see par 54 - the evaluator can leave feedback about the agent 127(1-n) on a question basis and/or general feedback. The feedback can be stored as part of a history of an agent 127(1-n).
See also Alon – see par 29 - A QM evaluation may be triggered by an evaluator or a supervisor of the contact center, who is charged with evaluating an agent's performance. For example, a supervisor may listen in on an interaction, to provide a qualitative assessment and performance feedback. ;and
calculating a customer feedback variance score, for the one or more user-selected evaluators (Mokashi – See par 55 - In one embodiment scores or ratings associated with answers are not displayed (but rather are hidden) to the reviewer (e.g. agent or evaluator) at the time of evaluation but rather after the evaluations are over, at the time of the evaluation comparison. See par 61, FIG. 9 – showing variance between two human reviewer scores (471, 472); see par 63 - A variance of the total score each evaluator's answers sum to may be created. The variance (e.g. a measure of how far numbers are spread out from their average value) may be a measure of how different the evaluator answers and agents are. In the case that two people, such as an evaluator and agent participate, an evaluator score (summing an evaluator rating from the ratings associated with all of the answers provided by the evaluator) and agent_score (summing an agent rating from the ratings associated with all of the answers provided by the agent) are created. More participant scores may be created, e.g. if there are more participants in the evaluation process beyond one evaluator and one agent. See par 67 – variance formulas),
wherein the customer feedback variance score is calculated such that:
a relatively higher customer feedback variance score results from relatively less effective evaluation of the agent-customer interaction; and
a relatively lower customer feedback variance score results from relatively more effective evaluation of the agent-customer interaction, and wherein the customer feedback variance score is calculated by comparing (Mokashi [disclosing both of limitations above] – see par 25 - User terminals 18 may allow users, such as contact or data center personnel, managers, evaluators, etc., to create, configure, execute and view evaluations (e.g. created sets of evaluation questions and). see par 63 - A variance of the total score each evaluator's answers sum to may be created. The variance (e.g. a measure of how far numbers are spread out from their average value) may be a measure of how different the evaluator answers and agents are. In the case that two people, such as an evaluator and agent participate, an evaluator score (summing an evaluator rating from the ratings associated with all of the answers provided by the evaluator) and agent_score (summing an agent rating from the ratings associated with all of the answers provided by the agent) are created. More participant scores may be created, e.g. if there are more participants in the evaluation process beyond one evaluator and one agent):
the evaluator feedback received via completion of the communicated agent evaluation form for the agent-customer interaction by the one or more user-selected evaluators (Mokashi – see par 58, FIG. 2, 4 - In operations 460 and 462, each of the evaluator and agent when finished reviewing the questions on the form, may finalize or submit the form, which causes in operation 470 the answers and scores, and/or in some embodiments one number which is the cumulative score, to be stored (e.g. in database 52). For example, form executor module 67 may receive or accept from the agent and evaluator (e.g. via agent terminals 16 and user terminals 18) a submission or indication indicating that the person has completed the computerized evaluation form. Data may be stored for example in a JSON document in a MySQL table which includes the information of the form, with answers and possibly scores added: e.g. a JSON or other document including questions and answers may be provided, and form executor module 67 may add actual responses—e.g. answers with scores—to the form; see also FIG. 7 – submit button 458 for evaluation with score 455 of 100.00); to
the customer feedback provided via the customer questionnaire associated with the agent-customer interaction (Vymenets – see par 62 – the agent (1-n) can meet with the evaluator for feedback. Additionally or alternatively, evaluation frequency can be based on customer survey feedback. For example, if a customer was not satisfied with a call for a determined agent 127(1-n), a frequency of evaluations can increase;
Alon – see par 29 - A QM evaluation may be triggered by an evaluator or a supervisor of the contact center, who is charged with evaluating an agent's performance. For example, a supervisor may listen in on an interaction, to provide a qualitative assessment and performance feedback. the present disclosure provides for initiating a customer feedback process to include customer experience and perspective in the QM process.).
Mokashi, Vymenets, and Alon are analogous art as they are directed to evaluating agents interacting with customers (see Mokashi Abstract, par 48; Vymenets Abstract; Alon Abstract). 1) Mokashi discloses having questions for evaluations of an agent in an interaction with a customer (See par 19, 46). Mokashi further discloses sending a form to a third-party evaluation, being a person different from the agent (See par 53). Vymenets and Alon improve upon Mokashi by having the person giving feedback be explicitly the person titled “customer” (See Vymenets par 62; Alon par 33) as well as using feedback to help produce questions for future evaluation forms, such as based on agent performance (See par Vymenets 30, 54, 70); and using customer feedback in conjunction with Quality Management evaluation of agent performance (See Alon par 30) as well as having questions determined from a subset of questions (See Alon par 49-51, 55, 58) where questions can be “assigned different scores based on frequency/occurrence of topics” (See par 49). One of ordinary skill in the art would be motivated to further include surveys/feedback answered by people who are considered “customers” as well as user feedback to help build evaluation forms of customer-agent interactions in Vymenets and having different questions and scores from customer feedback in Alon to efficiently improve upon the comparisons of two different feedback scores of customer-agent interactions in Mokashi.
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the variance between different evaluations of an agent in Mokashi to further have surveys/feedback answered by people who are considered “customers” as well as use feedback to help build evaluation forms of customer-agent interactions as disclosed in Vymenets, to further have customer feedback incorporated into evaluations of agents, selection of questions, and assigned different scores as disclosed in Alon, since the claimed invention is merely a combination of old elements, and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable and there is a reasonable expectation of success.
Concerning independent claim 8, Mokashi, Vymenets, and Alon disclose:
A system for calibrating evaluator feedback relating to one or more agent-customer interactions based on corresponding customer feedback, which comprises:
one or more databases (Mokashi – see par 26 - Interaction data or documents may be stored, e.g., in files and/or databases);
one or more computing devices adapted to (Mokashi – see par 40 – embodiments include… computer or process non-transitory storage medium, for example a memory, including or storing instructions, e.g., computer-executable instructions, which, when executed by a processor or controller, carry out methods disclosed herein):
The remaining limitations are similar to claim 1 above. It would be obvious to combine Mokashi, Vymenets, and Alon for the same reasons as claim 1.
Concerning independent claim 16, Mokashi, Vymenets, and Alon disclose:
A method for calibrating evaluator feedback relating to one or more agent-customer interactions based on corresponding customer feedback (Mokashi – see par 40 – embodiments include… computer or process non-transitory storage medium, for example a memory, including or storing instructions, e.g., computer-executable instructions, which, when executed by a processor or controller, carry out methods disclosed herein), which comprises:
The remaining limitations are similar to claim 1 above. It would be obvious to combine Mokashi, Vymenets, and Alon for the same reasons as claim 1.
Concerning claims 2, 9, and 16, Mokashi, Vymenets, and Alon discloses:
The apparatus of claim 1, wherein the operations further comprise visualizing, audibilizing, or both, via the one or more output devices each accessible by the at least one of the one or more user-selected evaluators, the calculated customer feedback variance score (Mokashi – see FIG. 9 – visualizing the variance score on a display; par 61 - FIG. 9 depicts an example screen displayed after an evaluation is complete, according to one embodiment. Evaluator score 471 and agent score 472 may be displayed; these scores may be the total for all answers answered by the evaluator and agent respectively, or the cumulative score up to the question(s) currently displayed. Questions 473 and answers 474 may be shown, along with scores 475 corresponding to the answers. Variance 476 may be displayed).
PNG
media_image1.png
598
822
media_image1.png
Greyscale
Concerning claims 3, 10, and 17, Mokashi, Vymenets, and Alon disclose:
The apparatus of claim 1, wherein the one or more user-selected filter criteria comprise channel type, duration of interaction, customer scoring of interaction, customer sentiment during interaction, or any combination thereof (Mokashi – see par 44 - A distribution definition may include document search terms or filters, such as only interactions longer or shorter than a certain time (e.g. those lasting less than a minute; those lasting more than 2 minutes…); those in certain categories; and those of a certain type (e.g. voice call, text interaction); ... Search terms or filters may include KPIs or other quality or evaluative measures, such as sentiment (e.g. the overall emotional tone of the call, such as positive, neutral, hostile, negative). Search terms or filters may include multiple parameters; See par 50- For example, selection parameters may include interactions with a KPI below a certain threshold (disclosing scoring), interactions that resulted in another call from a customer, etc.).
Concerning claims 4, 11, and 18, Mokashi, Vymenets, and Alon disclose:
The apparatus of claim 1, wherein the operations further comprise querying the one or more databases for one or more additional customer questionnaires associated with one or more additional agent-customer interactions from the queried one or more agent-customer interactions (Vymenets – see par 41 - A router and conversation manager server 120 may be leveraged for routing intelligence, of which there may be a great variety, and for association of the instant call with previous calls or future calls that might be made. see par 62 – the agent (1-n) can meet with the evaluator for feedback. Additionally or alternatively, evaluation frequency can be based on customer survey feedback. For example, if a customer was not satisfied with a call for a determined agent 127(1-n), a frequency of evaluations can increase. See par 85 - Additionally, the calibration report 2202 can add contact to the rating. For example, the interaction was a fifth in a series of interactions to solve a customer issue, and in this context the interaction can be rated higher than if the interaction were considered in isolation. The calibration report 2202 can help reduce variance between analysts over time. Additionally or alternatively, a profile can be generated for the analysts to show the evaluators that consistently over/under rate;
see also MPEP 2144.04 – Duplication of Parts - the court held that mere duplication of parts has no patentable significance unless a new and unexpected result is produced; here, no new and unexpected result by further getting feedback on “additional” interactions and questionnaires;
see also Alon - see par 49-51 - questionnaire module 110 may match questions with topics based, e.g., on an assigned metric assessing the relevance of a score to the interaction topic. For example, a question relating to agent demeanor may assigned different scores based on a number and/or occurrence frequency of topics and/or subtopics associated with agent conduct and manner, and/or based on an analysis of the frequency with which the agent uses polite phrasing in the interaction).
It would be obvious to combine Mokashi, Vymenets, and Alon for the same reasons as claim 1.
Concerning claims 5, 12, and 19, Mokashi, Vymenets, and Alon disclose:
The apparatus of claim 4, wherein generating the agent evaluation form for the agent-customer interaction is further based on customer feedback provided via the one or more additional customer questionnaires associated with the one or more additional agent-customer interactions (Vymenets –See par 85 - Additionally, the calibration report 2202 can add contact to the rating. For example, the interaction was a fifth in a series of interactions to solve a customer issue, and in this context the interaction can be rated higher than if the interaction were considered in isolation. The calibration report 2202 can help reduce variance between analysts over time. Additionally or alternatively, a profile can be generated for the analysts to show the evaluators that consistently over/under rate; see also MPEP 2144.04 – Duplication of Parts - the court held that mere duplication of parts has no patentable significance unless a new and unexpected result is produced; here, no new and unexpected result by further getting feedback on “additional” interactions and questionnaires;
see also Alon see par 28-30 - the present disclosure provides for initiating a customer feedback process to include customer experience and perspective in the QM process. Such customer feedback may in turn be used in conjunction with a QM evaluation of an agent's performance during an interaction that the agent participated in. ).
It would be obvious to combine Mokashi, Vymenets, and Alon for the same reasons as claim 1.
Concerning claims 6, 13, and 20, Mokashi, Vymenets, and Alon disclose:
The apparatus of claim 5, wherein the operations further comprise:
communicating to the one or more user-selected evaluators, one or more additional media streams of the one or more additional agent-customer interactions (Mokashi – see par 30 - Analysis center 50 may include for example a database 52 (e.g. stored in memory 120 or storage 130, FIG. 3) including for example audio recordings 54 (e.g. interactions or conversations), screen recordings 56 (e.g. of screenshots or other data associated with interactions), and text communication recordings 58.); and
visualizing, audibilizing, or both, via the one or more output devices each accessible by the at least one of the one or more user-selected evaluators, the one or more additional media streams (Mokashi – see par 52 - Documents or interactions may include for example audio recording (e.g. of telephone or other calls), text communications (SMS (short message service), WhatsApp messages, chats, etc.), video or other documents or files. See par 55 - each of the evaluator and agent may review the interaction or document; e.g. each may listen to the call or interaction (or read a transcript, view a video, etc., as appropriate) and provide answers (e.g. agent answers and evaluator answers each associated with a question) to questions. In one embodiment the document should be reviewed or the audio of the interaction listened to; see also MPEP 2144.04 – Duplication of Parts - the court held that mere duplication of parts has no patentable significance unless a new and unexpected result is produced; here, no new and unexpected result by further getting feedback on “additional” interactions and questionnaires).
Concerning claims 7, 14, and 21, Mokashi, Vymenets, and Alon disclose:
The apparatus of claim 6, wherein the customer feedback variance score is calculated, for the one or more user-selected evaluators, by further comparing:
the evaluator feedback received via completion of the communicated agent evaluation form for the agent-customer interaction by the one or more user-selected evaluators (Mokashi – See par 54, FIG. 4 -operations 440 and 442, the evaluator and agent may view one or more tasks or evaluation assignments (e.g. via form painter module 68 using internet browsers 17 and 19), which may include a specifically instantiated form paired with a specific interaction or document to evaluate; see par 56 - agent terminals 16 and user terminals 18, each including one or more question/answer combinations) from the form to an agent and evaluator, and may receive or accept from the agent (e.g. via agent terminals 16), for each question, an agent answer associated with a question, and may receive or accept from one or more evaluators (e.g. via user terminals 18) for each question evaluator answer(s) associated with questions. Each answer may have an associated score or rating which is typically pre-set or predetermined, e.g. in operation 400. Each of the evaluator and agent may move or proceed through the different screens of the form, and answer the questions. Answering a question having an associated score may cause a module such as form executor module 67 to record the score;);
to the customer feedback provided via the one or more additional customer questionnaires associated with the one or more additional agent-customer interactions (Mokashi – see par 63- A variance of the total score each evaluator's answers sum to may be created. The variance (e.g. a measure of how far numbers are spread out from their average value) may be a measure of how different the evaluator answers and agents are. In the case that two people, such as an evaluator and agent participate, an evaluator score (summing an evaluator rating from the ratings associated with all of the answers provided by the evaluator) and agent_score (summing an agent rating from the ratings associated with all of the answers provided by the agent) are created. More participant scores may be created, e.g. if there are more participants in the evaluation process beyond one evaluator and one agent; see also FIG. 9, par 64-66 – can score all evaluator results relative to “agent results”; see par 69 - A “dispute” between the agent and another evaluator (a discrepancy among answers to the same question, or a large variance) may be, using embodiments of the present invention, an honest product of the process, and may be produced without the agent knowing there is a dispute, rather than an after-the-fact disagreement. A practical application of the formula and algorithm is realized by allowing agents involved in interactions or documents to more accurately and honestly rate their performance, and allowing agents to dispute their evaluation by others by providing evaluations without having seen others' evaluations. In such a manner a “dispute” is inherent in the difference between agents' and evaluators' answers;
see also Vymenets –see par 30 - A goal of the contact centers can be to provide quality customer service. Systems and methods can provide for a quality management platform that builds forms to help evaluate interactions by customers with agents at contact centers. See par 54 - The feedback can be stored as part of a history of an agent 127(1-n). The feedback can also be used to produce questions for future forms for the agent, e.g., specific questions related to the agent's tone, the way they state the policy and the way they greet the customer, etc. These questions can aid in coaching objectives for the agent. The additional questions can be temporarily added for a determined time, permanent, updated based on agent performance, etc. See par 62 - evaluation frequency can be based on customer survey feedback. For example, if a customer was not satisfied with a call for a determined agent 127(1-n), a frequency of evaluations can increase. see par 68 - The evaluation manager can display the evaluations 1302 based on a filter 1310, e.g., by template 1312 (training, performance, sales target, legal, voice, customer experience, etc.), creator 1314, date 1316, evaluations 1318, etc. The evaluations can be scheduled 1320, e.g., as set by drop down options including a one-time occurrence or recurring. Clicking 1322 on the evaluation adds the form to an evaluation summary palette. see par 70 - The analytics server 137 can also determine a content of the interaction using analytics. Therefore, the contact center 15 can perform an evaluation based on the content of the interaction, the conversation level, the agent' communications, the customer's communications, call data, etc. The forms manager can add specific questions to the evaluation form based on the interaction data, timing of the interaction and/or other criteria, for example questions about an understanding level or customer comprehension of the interaction if the agent is not a native speaker; see also MPEP 2144.04 – Duplication of Parts - the court held that mere duplication of parts has no patentable significance unless a new and unexpected result is produced; here, no new and unexpected result by further getting feedback on “additional” interactions and questionnaires).
It would be obvious to combine Mokashi, Vymenets, and Alon for the same reasons as claim 1.
Response to Arguments
Applicant's arguments filed 1/21/26 have been fully considered but they are not persuasive and/or are moot in view of the new rejections.
With regards to 103, Applicant’s arguments are moot in view of the new rejection also applying Alon.
With regards to 101, Applicant argues that the amendments make the claim eligible as amounting to “significantly more than an abstract idea” in light of the claims and the amendments. Remarks, pages 17. In response, Examiner respectfully disagrees. The amendments add database and querying a database. The revised 101 rejection addresses these “additional elements,” but at this time, even when viewed as a combination, the claims are still rejected. Examiner suggests focusing on amending “additional elements” in the claim and centering arguments on a technological improvement, to the extent supported.
The remaining arguments are moot in view of the revised rejections necessitated by the amendments.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IVAN R GOLDBERG whose telephone number is (571)270-7949. The examiner can normally be reached 830AM - 430PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anita Coupe can be reached at 571-270-3614. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IVAN R GOLDBERG/ Primary Examiner, Art Unit 3619
Prosecution Insights