DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status
This action is in response to the amendment filed on 3/19/2026. Claims 1-20 are pending. Claims 1, 6, 9-11, 16, 19-20, are amended. No claims have been added. No claims have been cancelled.
Response to Arguments
Applicant's arguments filed 3/19/2026 have been fully considered but they are not persuasive. The applicant has argued in view of Ex Parte Desjardins "the Applicant submits that the claims are directed to patent-eligible subject matter and that any alleged judicial exception is integrated into a practical application. In particular, the Applicant submits that the claims are directed to an invention that provides an improvement to a technical field.” The examiner respectfully disagrees. The claimed use of a binary (ex yes/no) survey is a collection choice. Applicant’s specification does not describe how the binary format improves the functioning of a computer component, processor, memory, or communication device. Any benefit would be that a binary response would be easier to analyze, which is a statement about the content of the data, not about any improvement to computer functionality. According to Desjardins the improvement must be apparent from the specification and reflected in the claims as something that changes how the computer system itself operates. At most applicant’s invention has modified what data is received. The claimed technology is cited at a high level of generality and perform conventional functions. Applicant’s specification does not establish that the claimed combination of the elements operates in any non-conventional manner.
The applicant has also argued “The Applicant submits that the claimed invention does not relate simply to fundamental economic practices, but to a technological implementation which alleviates technical challenges previously associated with the collection of accurate customer experience data which can be analyzed accurately and used for improved result accuracy. Specifically, the use of binary problem assessments related to the types of negative customer experiences stored in the problem type data set allows the system to collect binary (yes/no) answers which are inherently more suitable for computational analysis, and for translation to computer data structures than users providing word answers or ranking experiences along a continuum. The latter is inherently subjective, which makes data processing more fuzzy, whereas the presently claimed invention deliberately obtains responses from users in a binary format which alleviates the difficulty for computerized systems to manipulate and otherwise process the response data. The Applicant submits that this addresses a technical problem associated with converting subjective user experience data to digital data more accurately." The examiner respectfully disagrees. Processing customer feedback to predict financial impact is not specifically addressing a technical problem. The binary survey format, while argued to product more structured data, does not transform the abstract analytical process. The economic impact applied to the collected data are merely mathematical operations applied to the collected data.
Under the USPTO Guidance, we next determine under Step 2A, Prong Two, whether the claims recite additional elements that integrate the judicial exception into a practical application (see MPEP §§ 2106.05(a)–(c), (e)–(h)). To integrate the exception into a practical application, the additional claim
elements must, for example, improve the functioning of a computer or any other technology or technical field (see MPEP § 2106.05(a)), apply the judicial exception with a particular machine (see MPEP § 2106.05(b)), or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment (see MPEP § 2106.05(e)). The only elements in claim 1 beyond the abstract idea (i.e., the additional
elements) are “a communication interface,” “at least one processor,” “memory in communication with at least one processor,” “software code stored in memory,” “an electronic survey,” “a display screen,” and “a graphic user interface.” The additional elements of claim 1 do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. In particular, Appellant’s Specification discloses that these elements encompass generic computer components. See Spec. ¶¶ 53, 127-131 (describing these elements each at a high level of generality). Merely adding generic computer components to perform abstract ideas does not integrate those ideas into a practical application. See MPEP §§ 2106.04(a), (d) (citing 2019 Revised Guidance, 84 Fed. Reg. at 55 (identifying “merely includ[ing] instructions to implement an abstract idea on a computer” as an example of when an abstract idea has not been integrated into a practical application)). Thus, the applicant has not persuaded us that the additional elements recited in claim 1: (1) improve the functioning of a computer or other technology; (2) are applied with any particular machine (except for generic computing elements); (3) effect a transformation of a particular article to a different state; or (4) are applied in any meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception. See MPEP §§ 2106.05(a)–(c), (e)–(h). The previous 101 rejection is maintained below.
The applicant has argued the previous 103 rejection in view of newly amended limitations. An updated search was conducted and the previous 103 rejection has been updated in view of applicant’s amendments. An updated 103 rejection can be found below.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter because the claim(s) as a whole, considering all claim elements both individually and in combination, do not amount to significantly more than an abstract idea. The claim(s) is/are directed to the abstract idea of processing customer feedback to predict a financial impact. The claimed invention is directed to an abstract idea without significantly more.
Step 2A, Prong 1:
The claim(s) recite(s) (mathematical relationships/formulas, mental process or certain methods of organizing human activity). Specifically the independent claims recite:
(a) mental process: as drafted, the claim recites the limitations of maintaining data, maintaining a tree model, transmitting a link, presenting a survey, converting the responses to a survey, receiving feedback data, storing the feedback, generating a decision tree, determining an economic impact, and causing to render a visualization which is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. That is, other than reciting generic computing devices nothing in the claim precludes the determining step from practically being performed in the human mind. For example, but for the computer/processor language, the claim encompasses a user manually predicting the economic impact of a negative customer experience. The mere nominal recitation of a generic computing devices does not take the claim limitation out of the mental processes grouping. This limitation is a mental process.
(b) mathematical formula: The claim recites a mathematical concept (which can include a mathematical relationships, mathematical formulas or equations, and mathematical calculations), and in this case calculating the economic impact of negative customer experiences. Thus, the claim recites a mathematical concept. A claim that recites a mathematical calculation will be considered as falling within the “mathematical concepts” grouping. A mathematical calculation is a mathematical operation (such as multiplication) or an act of calculating using mathematical methods to determine a variable or number, e.g., performing an arithmetic operation such as exponentiation. There is no particular word or set of words that indicates a claim recites a mathematical calculation. That is, a claim does not have to recite the word “calculating” in order to be considered a mathematical calculation. For example, a step of “determining” a variable or number using mathematical methods or “performing” a mathematical operation may also be considered mathematical calculations when the broadest reasonable interpretation.
(c) certain methods of organizing human activity: The claim as a whole recites a method of organizing human activity. The claimed invention is a method that allows for users to take customer feedback and determine how negative customer feedback causes an economic impact which is both a fundamental economic practice and a commercial interaction. Thus, the claim recites an abstract idea.
“Fundamental Economic Practices or Principles”; Under the 2019 PEG, “fundamental economic principles or practices,” which describe subject matter relating to the economy and commerce, are considered to be a “certain method of organizing human activity.” According to the 2019 PEG, “fundamental economic principles or practices” include hedging, insurance, and mitigating risk. The term “fundamental” is not used in the sense of necessarily being “old” or “well-known,” although being old or well-known may indicate that the practice is “fundamental.” Commercial or Legal Interactions” According to the 2019 PEG, “commercial interactions” or “legal interactions” include subject matter relating to agreements in the form of contracts.
5. Dependent claims 2-4, 6-10, 12-14, 16-19, further narrow the abstract idea identified in the independent claims and do not introduce further additional elements for consideration.
6. Dependent claims 5, 15, will be evaluated under Step 2A, Prong 2 below.
Step 2A, Prong 2: Independent claims 1, 11, 20, do not integrate the judicial exception into a practical application. Claim 1 is a system comprising “a communication interface; at least one processor; memory in communication with said at least one processor; software code stored in said memory, which when executed at said at least one processor causes said system to: transmit, to one or more customer computing devices, a link to an electronic survey; electronically present an electronic survey, store said feedback data in said memory; and cause to render, at a display screen, a graphic user interface.” Claim 11 is a method that recites limitations “transmitting, to one or more customer computing devices, a link to an electronic survey” , “electronically presenting said survey,” “storing said feedback data in a memory” , “a display screen, a graphic user interface visualizing the computed economic.” Claim 20 further recites the additional elements of “A non-transitory computer-readable storage medium storing instructions”, “transmit, to one or more customer computing devices, a link to an electronic survey,” “electronically present said electronic survey to said one or more customer computing devices, by said link,” “store said feedback data in a memory,” “render, at a display screen, a graphic user interface visualizing the computed economic impact.” These additional elements are mere instructions to implement an abstract idea using a computer in its ordinary capacity, or merely uses the computer as a tool to perform the identified abstract idea. Use of a computer or other machinery in its ordinary capacity for performing the steps of the abstract idea or other tasks (e.g., to maintain, transmit, present, convert, store, generate, compute, and display data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., certain methods of organizing human activity) does not integrate a judicial exception into a practical application. See MPEP 2106.05(f).
Therefore, the additional elements of the independent claims, when considered both individually and in combination, are not sufficient to prove integration into a practical application.
Dependent claims 2-4, 6-10, 12-14, 16-19, further narrow the abstract idea identified in the independent claims and do not introduce further additional elements for consideration, which does not integrate the judicial exception into a practical application.
Dependent claims 5, 15 introduce the additional element of “wherein the binary tree is generated using machine learning.” This limitation does not integrate the judicial exception into a practical application because it is nothing more than generally linking the use of the judicial exception to a particular technological environment. See MPEP 2106.05(h).
Therefore, the additional elements of the dependent claims, when considered both individually and in the context of the independent claims, are not sufficient to prove integration into a practical application.
Step 2B: Independent claims 1, 11, and 20 do not comprise anything significantly more than the judicial exception. As can be seen above with respect to Step 2A, Prong 2, Claim 1 is a system comprising “a communication interface; at least one processor; memory in communication with said at least one processor; software code stored in said memory, which when executed at said at least one processor causes said system to: transmit, to one or more customer computing devices, a link to an electronic survey; electronically present an electronic survey, store said feedback data in said memory; and cause to render, at a display screen, a graphic user interface.” Claim 11 is a method that recites limitations “transmitting, to one or more customer computing devices, a link to an electronic survey” , “electronically presenting said survey,” “storing said feedback data in a memory” , “a display screen, a graphic user interface visualizing the computed economic.” Claim 20 further recites the additional elements of “A non-transitory computer-readable storage medium storing instructions”, “transmit, to one or more customer computing devices, a link to an electronic survey,” “electronically present said electronic survey to said one or more customer computing devices, by said link,” “store said feedback data in a memory,” “render, at a display screen, a graphic user interface visualizing the computed economic impact.” These additional elements are mere instructions to implement an abstract idea using a computer in its ordinary capacity, or merely uses the computer as a tool to perform the identified abstract idea. Use of a computer or other machinery in its ordinary capacity for performing the steps of the abstract idea or other tasks (e.g. to maintain, transmit, present, convert, store, generate, compute, and display data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., certain methods of organizing human activity) is not anything significantly more than the judicial exception. See MPEP 2106.05(f).
The additional elements of the independent claims, when considered both individually and in combination, do not comprise anything significantly more than the judicial exception.
Dependent claims 2-4, 6-10, 12-14, 16-19, further narrow the abstract idea identified in the independent claims and do not introduce further additional elements for consideration, which is not anything significantly more than the judicial exception.
Dependent claims 5, 15 introduce the additional element of “wherein the binary tree is generated using machine learning.” This limitation is not anything significantly more than the judicial exception because it is nothing more than generally linking the use of the judicial exception to a particular technological environment. See MPEP 2106.05(h).
Therefore based on the above analysis as conducted based on MPEP 2106 from the United States Patent and Trademark Office the claims are viewed as a court recognized abstract idea, are viewed as a judicial exception, does not integrate the claims into a practical application, does not provide significantly more, and does not provide an inventive concept, therefore the claims are ineligible.
The additional elements of the dependent claims, when considered both individually and in the context of the independent claims, are not anything significantly more than the judicial exception.
Accordingly, claims 1-20 are rejected under 35 USC 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 9, 11, 19, 20, is/are rejected under 35 U.S.C. 103 as being unpatentable over Bennett et al. (US 20210019357 A1) in view of L. Chen et al. “A study on review manipulation classification using decision tree”, published 2013 (referred to hereinafter as ‘Chen’) in view of Smith et al. (US 20180122256 A1) in view of Ghorbani et al. (US 20210150604 A1).
Regarding claim 1, Bennett teaches a computer-implemented system for determining economic impact of customer experiences (¶ 12-14, disclose the impact of different costumer experiences. ¶ 38, discloses the use of KPI when making a determination of impact of costumer experiences. ¶ 44-48, 53, 94, 95, 106, 4);
a communication interface; at least one processor; memory in communication with said at least one processor; software code stored in said memory, which when executed at said at least one processor (¶ 17, 38, 63, 66, 103-106, disclose the claimed technology for performing the steps of the invention.);
maintain a data set including a plurality of types of negative customer experiences (¶ 25, discloses the use of customer questionnaires for customer experiences. ¶ 45, 54, discloses features that negatively impact a financial performance of an entity. ¶ 38, 78);
maintain a tree model for predicting economic impact of one or more of the plurality of types of negative customer experiences (¶ 25, discloses receiving questionnaire data in multiple forms. ¶ 45, discloses features that negatively impact a financial performance of an entity. ¶ 83, As shown by reference number 220, the machine learning system may train a machine learning model using the set of observations and using one or more machine learning algorithms, such as a regression algorithm, a decision tree algorithm, a neural network algorithm, a k-nearest neighbor algorithm, a support vector machine algorithm, and/or the like. After training, the machine learning system may store the machine learning model as a trained machine learning model 225 to be used to analyze new observations. ¶ 54, 38, 78);
receive feedback data (¶ 78-80, discloses a feature set may include one or more of the following features: employee satisfaction rating, customer satisfaction rating, a customer perceived usability answer or rating, a customer perceived value answer or rating, a customer perceived risk answer or rating, a customer convenience answer or rating, a technical innovation answer or rating, a functional quality answer or rating, a service quality answer or rating, a customer to employee engagement answer or rating, a brand trust answer or rating, a customer loyalty answer or rating, a Net Promoter Score, an overall customer experience answer or rating, an activity in social media answer or rating, an employee collaboration answer or rating, an employee creativity answer or rating, an employee empowerment answer or rating, an employee diversity answer or rating, an employee personal financial health answer or rating, an employee personal growth answer or rating, a leadership/direct supervisor answer or rating, an employee belief in company competitiveness answer or rating, an overall employee experience answer or rating, a technology work complexity answer or rating, a process work complexity answer or rating, an employee culture answer or rating, an employee self-efficacy answer or rating, a workplace analytics rating, an average employee salary, an average amount of time employees spend working, an average amount of time employees spend working outside of normal business hours, an employee efficiency rating, an entity financial health rating, and/or the like. ¶ 17-19, discloses various ways of receiving feedback in various formats ¶ 27, 38, 49, 57, 78);
store said feedback data in said memory (¶ 103-105, discloses storing feedback data in a memory. ¶ 65-67);
generate a decision tree based on the tree model (¶ 83, As shown by reference number 220, the machine learning system may train a machine learning model using the set of observations and using one or more machine learning algorithms, such as a regression algorithm, a decision tree algorithm, a neural network algorithm, a k-nearest neighbor algorithm, a support vector machine algorithm, and/or the like. After training, the machine learning system may store the machine learning model as a trained machine learning model 225 to be used to analyze new observations.);
determine economic impact of at least one of the types of negative customer experiences using the feedback data (¶ 44-47, For example, for a feature, the data analytics model 102 may determine that below an influence threshold the feature does not impact business performance (e.g., financial performance) of an entity. That is, if the influence threshold is 55 and a current feature rank is 35, the data analytics model 102 may determine that even if the feature rank is increased from 35 to 50, the financial performance of the entity may not be impacted or may be minimally impacted. The data analytics model 102 may determine that above a feature rank of 55 (e.g., the influence threshold associated with the feature), the feature may begin to incrementally impact the financial performance of the entity as the feature rank of the feature is improved. ¶ 27, 38, 49, 57, 78, 17-19);
and cause to render, at a display screen, a graphic user interface visualizing the determined economic impact of at least one of the types of negative customer experiences (¶ 56-57, 63, discloses a data analytics platform for analyzing an economic impact of a customer experience. ¶ 45, 54, disclose a negative customer economic impact.)
Bennett does not specifically teach all the claimed details of the decision tree.
However the combination of Bennett and Chen teaches
generate a decision tree based on the tree model, the data set and the feedback data, the decision tree having a plurality of internal nodes with each internal node corresponding to a type of the plurality of types of negative customer experiences (Chen, Table I, pg. 1-2, Consequently, this study aims to introduce additional information such as readability, sentiment, product features and so on to improve the classification performance of review manipulation by using decision tree (DT) algorithm. In addition, we try to find the important review manipulation factors from a candidate attribute set via the analysis of extracted knowledge rules and correlation coefficients. It can help the potential consumers to realize the reality of online reviews and to reduce the uncertainty of purchase decision making. Pg. 2, When decision tree induction is used for feature selection, a tree is constructed from the given data. All attributes that do appear in the tree are assumed to be relevant. The set of attributes appearing in the tree form the reduced subset of attributes. Pg. 3, The implemental procedure of the employed method can be listed in Fig. 1. The concise steps can be described as follows. In fact, there are 7 steps. They are “data collection”, “data preparation”, “define review manipulation attributes”, “created term-document matrix (TDM)”, “implement decision tree”, “identify key manipulation attributes”, and “performance & drawing conclusions”. Pg. 4, Step 4: Create TDM Step 4.1: Construct term-document matrix Every single review has been converted into a vector of terms with Term Frequency–Inverse Document Frequency (TF-IDF) weights. Then, based on defined features in step 2 and 3, the collected documents will be transformed to a term-document matrix (TDM). Step 4.2: Construct training/test sets In this step, 5 fold cross validation experiment has been employed for these training data. Depending on the employed feature sets, we have three experiments. They are described as bellow. Experiment #1: We only use original TDM with text features (unigram) Experiment #2: Combine the 8 review manipulation attribute set into the original TDM. Therefore, we use both original text features and review manipulation attributes to describe the collected data. Experiment #3: We only use review manipulation attributes to describe the collected data. Step 5: Implement decision tree algorithm (C4.5) Step 6: Identify key review manipulation attributes In this step, we use decision tree and correlation analysis to be our feature selection methods. Step 6.1: Construct 5 trees and pick the one who has the best performance Step 6.2: Interpret the knowledge rules discovered by decision tree. Step 6.3: Implement statistical correlation analysis Step 6.4: Acquire key attributes Step 7: Performance evaluation and drawing conclusions In this work, we use PA (Positive Accuracy), NA (Negative Accuracy), G-mean (Geometric mean of PA and NA), OA (Overall Accuracy), and F1 (an integrated index by using precision and recall).
Bennett teaches computing the economic impact using the feedback data. The combination of Bennett and Chen teaches
determine economic impact of at least one of the types of negative customer experiences using the generated decision tree and the feedback data (Chen, Table I, pg. 3, 4) Positive Sentiment Using positive sentiment can guide readers easier than negative comments. Related works found that readers will ignore the negative comments, if they really want to purchase something [25]. 5) Negative Sentiment Berger et al. [9] found the negative comments of existing products will lead to the increase of sales, compared to the products which haven’t be discussed. 6) Sentiment One article which contains sentiments (no matter positive or negative) can influence the emotions of readers [20, 22]. This attribute combine the information involved in 4th and 5th attributes. pg. 1-2, Consequently, this study aims to introduce additional information such as readability, sentiment, product features and so on to improve the classification performance of review manipulation by using decision tree (DT) algorithm. In addition, we try to find the important review manipulation factors from a candidate attribute set via the analysis of extracted knowledge rules and correlation coefficients. It can help the potential consumers to realize the reality of online reviews and to reduce the uncertainty of purchase decision making. Pg. 2, When decision tree induction is used for feature selection, a tree is constructed from the given data. All attributes that do appear in the tree are assumed to be relevant. The set of attributes appearing in the tree form the reduced subset of attributes…Some data mining algorithms have built-in feature selections such as decision trees [35]. When decision tree induction is used for feature selection, a tree is constructed from the given data. All attributes that do appear in the tree are assumed to be relevant. The set of attributes appearing in the tree form the reduced subset of attributes. Pg. 3, The implemental procedure of the employed method can be listed in Fig. 1. The concise steps can be described as follows. In fact, there are 7 steps. They are “data collection”, “data preparation”, “define review manipulation attributes”, “created term-document matrix (TDM)”, “implement decision tree”, “identify key manipulation attributes”, and “performance & drawing conclusions”. Pg. 4, Step 4: Create TDM Step 4.1: Construct term-document matrix Every single review has been converted into a vector of terms with Term Frequency–Inverse Document Frequency (TF-IDF) weights. Then, based on defined features in step 2 and 3, the collected documents will be transformed to a term-document matrix (TDM). Step 4.2: Construct training/test sets In this step, 5 fold cross validation experiment has been employed for these training data. Depending on the employed feature sets, we have three experiments. They are described as bellow. Experiment #1: We only use original TDM with text features (unigram) Experiment #2: Combine the 8 review manipulation attribute set into the original TDM. Therefore, we use both original text features and review manipulation attributes to describe the collected data. Experiment #3: We only use review manipulation attributes to describe the collected data. Step 5: Implement decision tree algorithm (C4.5) Step 6: Identify key review manipulation attributes In this step, we use decision tree and correlation analysis to be our feature selection methods. Step 6.1: Construct 5 trees and pick the one who has the best performance Step 6.2: Interpret the knowledge rules discovered by decision tree. Step 6.3: Implement statistical correlation analysis Step 6.4: Acquire key attributes Step 7: Performance evaluation and drawing conclusions In this work, we use PA (Positive Accuracy), NA (Negative Accuracy), G-mean (Geometric mean of PA and NA), OA (Overall Accuracy), and F1 (an integrated index by using precision and recall.
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform the specific details of the decision tree, as taught/suggested by Chen. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to analyzing and using customer feedback. One of ordinary skill in the art would have recognized that applying the known technique of Chen would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Chen to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such specific details of the decision tree into similar systems. Further, applying the specific details of the decision tree, would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow for analysis of laid out options, consequences, and comparing outcomes.
Bennett teaches computing the economic impact using the feedback data. Bennett does not specifically teach the details of the electronic survey or binary assessments as claimed.
However, Smith teaches transmit, to one or more customer computing devices, a link to an electronic survey (¶ 146-148, discloses transmitting a link to an electronic survey. ¶ 41, 61, 168, Fig. 6);
electronically present said electronic survey to said one or more customer computing devices by said link, wherein said electronic survey comprises binary problem assessments, each problem having two possible states (¶ 48, 140, 147-148, discloses electronic surveys with links. ¶ 114, 120, discloses a question two selectable options, yes or no.);
converting responses to said electronic survey to feedback data said feedback data comprising structured data (¶ 45, 74-75, 79-81, 90, 140, 143, discloses structuring electronic survey data into a database.)
receive the feedback data (¶ 51, 53, discloses receiving feedback data.);
store said feedback data in said memory (171-177, discloses storing data in a memory. ¶ 188-189).
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform the specific details of an electronic survey, as taught/suggested by Smith. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to surveys for receiving user information. One of ordinary skill in the art would have recognized that applying the known technique of Smith would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Smith to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such electronic survey features into similar systems. Further, applying electronic survey features would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow for a survey that is less time consuming, they are cheaper, you get the results faster, and you can transfer and use the data in various applications to answer important questions.
Bennett teaches computing the economic impact using the feedback data. Bennett does not specifically teach the details of the electronic survey or binary assessments as claimed.
However, Ghorbani teaches problem type data sets (¶ 111-116, categorizing keywords and types based on them being negative. ¶ 132, 176) wherein said electronic survey comprises binary problem assessments relating to one or more of said types of negative customer experiences, each problem having only two possible states (¶ 170, 176, discloses the use of the binary survey. ¶ 111, 116, discloses determining a positive or negative sentiment. ¶ 140-141, discloses a negative response/experience. ¶ 152, discloses a negative affinity. ¶ 119, 172, 187, 197, 201, 207).
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform the specific details of an electronic survey, as taught/suggested by Ghorbani. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to surveys for receiving user information. One of ordinary skill in the art would have recognized that applying the known technique of Ghorbani would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Ghorbani to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such electronic survey features into similar systems. Further, applying electronic survey features would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow for a survey that is less time consuming, they are cheaper, you get the results faster, and you can transfer and use the data in various applications to answer important questions.
Regarding claims 9 and 19, Bennett teaches wherein the software code, when executed at said at least one processor, causes said system to determine the economic impact of at least one of the types of negative customer experiences by: determining, for the at least one type of negative customer experience, a frequency of occurrence among a plurality of customers based on the feedback data; determining, for the at least one type of negative customer experience, a financial impact on the plurality of customers based on the feedback data; and determining the economic impact of for the at least one type of negative customer experience by multiplying the frequency of occurrence by the financial impact (¶ 54-56, In some implementations, the subset of features may include features that together most positively impact an experience attribute (e.g., that if the feature rank of each feature included in the subset of features is improved, the entity will realize the largest positive impact on financial performance). In some implementations, the subset of features may include features that together most negatively impact an experience attribute. For example, the data analytics platform 102 may identify a subset of features that are most negatively impacting financial performance of the entity. In some implementations, the data analytics platform 102 may identify a subset of features that, if feature ranks of features included in the subset of features are decreased, would have the largest negative impact on an experience attribute and/or on the financial performance of the entity. ¶ 44-48, 51, 79-80).
Regarding claim 11, Bennett teaches a computer-implemented method for computing economic impact of customer experiences (¶ 12, 38, discloses an economic impact of customer experiences. ¶ 44-48, 53, 94, 95, 106, 4);
maintaining a data set including a plurality of types of negative customer experiences (¶ 25, discloses maintaining data sets. ¶ 45, 54, discloses features that negatively impact a financial performance of an entity. ¶ 38, 78);
maintaining a tree model for predicting economic impact of one or more of the plurality of types of negative customer experiences (¶ 25, discloses receiving questionnaire data in multiple forms. ¶ 45, discloses features that negatively impact a financial performance of an entity. ¶ 83, As shown by reference number 220, the machine learning system may train a machine learning model using the set of observations and using one or more machine learning algorithms, such as a regression algorithm, a decision tree algorithm, a neural network algorithm, a k-nearest neighbor algorithm, a support vector machine algorithm, and/or the like. After training, the machine learning system may store the machine learning model as a trained machine learning model 225 to be used to analyze new observations. ¶ 54, 38, 78);
receiving feedback data (¶ 78-80, discloses a feature set may include one or more of the following features: employee satisfaction rating, customer satisfaction rating, a customer perceived usability answer or rating, a customer perceived value answer or rating, a customer perceived risk answer or rating, a customer convenience answer or rating, a technical innovation answer or rating, a functional quality answer or rating, a service quality answer or rating, a customer to employee engagement answer or rating, a brand trust answer or rating, a customer loyalty answer or rating, a Net Promoter Score, an overall customer experience answer or rating, an activity in social media answer or rating, an employee collaboration answer or rating, an employee creativity answer or rating, an employee empowerment answer or rating, an employee diversity answer or rating, an employee personal financial health answer or rating, an employee personal growth answer or rating, a leadership/direct supervisor answer or rating, an employee belief in company competitiveness answer or rating, an overall employee experience answer or rating, a technology work complexity answer or rating, a process work complexity answer or rating, an employee culture answer or rating, an employee self-efficacy answer or rating, a workplace analytics rating, an average employee salary, an average amount of time employees spend working, an average amount of time employees spend working outside of normal business hours, an employee efficiency rating, an entity financial health rating, and/or the like. ¶ 17-19, discloses various ways of receiving feedback in various formats ¶ 27, 38, 49, 57, 78);
storing said feedback data in said memory (¶ 103-105, discloses storing feedback data in a memory. ¶ 65-67);
generating a decision tree based on the tree model (¶ 83, As shown by reference number 220, the machine learning system may train a machine learning model using the set of observations and using one or more machine learning algorithms, such as a regression algorithm, a decision tree algorithm, a neural network algorithm, a k-nearest neighbor algorithm, a support vector machine algorithm, and/or the like. After training, the machine learning system may store the machine learning model as a trained machine learning model 225 to be used to analyze new observations.);
determining economic impact of at least one of the types of negative customer experiences using the feedback data (¶ 44-47, For example, for a feature, the data analytics model 102 may determine that below an influence threshold the feature does not impact business performance (e.g., financial performance) of an entity. That is, if the influence threshold is 55 and a current feature rank is 35, the data analytics model 102 may determine that even if the feature rank is increased from 35 to 50, the financial performance of the entity may not be impacted or may be minimally impacted. The data analytics model 102 may determine that above a feature rank of 55 (e.g., the influence threshold associated with the feature), the feature may begin to incrementally impact the financial performance of the entity as the feature rank of the feature is improved. ¶ 27, 38, 49, 57, 78, 17-19);
and causing to render, at a display screen, a graphic user interface visualizing the determined economic impact of at least one of the types of negative customer experiences (¶ 56-57, 63, discloses a data analytics platform for analyzing an economic impact of a customer experience. ¶ 45, 54, disclose a negative customer economic impact.)
Bennett does not specifically teach all the claimed details of the decision tree.
However the combination of Bennett and Chen teaches
generate a decision tree based on the tree model, the data set and the feedback data, the decision tree having a plurality of internal nodes with each internal node corresponding to a type of the plurality of types of negative customer experiences (Chen, Table I, pg. 1-2, Consequently, this study aims to introduce additional information such as readability, sentiment, product features and so on to improve the classification performance of review manipulation by using decision tree (DT) algorithm. In addition, we try to find the important review manipulation factors from a candidate attribute set via the analysis of extracted knowledge rules and correlation coefficients. It can help the potential consumers to realize the reality of online reviews and to reduce the uncertainty of purchase decision making. Pg. 2, When decision tree induction is used for feature selection, a tree is constructed from the given data. All attributes that do appear in the tree are assumed to be relevant. The set of attributes appearing in the tree form the reduced subset of attributes. Pg. 3, The implemental procedure of the employed method can be listed in Fig. 1. The concise steps can be described as follows. In fact, there are 7 steps. They are “data collection”, “data preparation”, “define review manipulation attributes”, “created term-document matrix (TDM)”, “implement decision tree”, “identify key manipulation attributes”, and “performance & drawing conclusions”. Pg. 4, Step 4: Create TDM Step 4.1: Construct term-document matrix Every single review has been converted into a vector of terms with Term Frequency–Inverse Document Frequency (TF-IDF) weights. Then, based on defined features in step 2 and 3, the collected documents will be transformed to a term-document matrix (TDM). Step 4.2: Construct training/test sets In this step, 5 fold cross validation experiment has been employed for these training data. Depending on the employed feature sets, we have three experiments. They are described as bellow. Experiment #1: We only use original TDM with text features (unigram) Experiment #2: Combine the 8 review manipulation attribute set into the original TDM. Therefore, we use both original text features and review manipulation attributes to describe the collected data. Experiment #3: We only use review manipulation attributes to describe the collected data. Step 5: Implement decision tree algorithm (C4.5) Step 6: Identify key review manipulation attributes In this step, we use decision tree and correlation analysis to be our feature selection methods. Step 6.1: Construct 5 trees and pick the one who has the best performance Step 6.2: Interpret the knowledge rules discovered by decision tree. Step 6.3: Implement statistical correlation analysis Step 6.4: Acquire key attributes Step 7: Performance evaluation and drawing conclusions In this work, we use PA (Positive Accuracy), NA (Negative Accuracy), G-mean (Geometric mean of PA and NA), OA (Overall Accuracy), and F1 (an integrated index by using precision and recall).
Bennett teaches computing the economic impact using the feedback data. The combination of Bennett and Chen teaches
determine economic impact of at least one of the types of negative customer experiences using the generated decision tree and the feedback data (Chen, Table I, pg. 3, 4) Positive Sentiment Using positive sentiment can guide readers easier than negative comments. Related works found that readers will ignore the negative comments, if they really want to purchase something [25]. 5) Negative Sentiment Berger et al. [9] found the negative comments of existing products will lead to the increase of sales, compared to the products which haven’t be discussed. 6) Sentiment One article which contains sentiments (no matter positive or negative) can influence the emotions of readers [20, 22]. This attribute combine the information involved in 4th and 5th attributes. pg. 1-2, Consequently, this study aims to introduce additional information such as readability, sentiment, product features and so on to improve the classification performance of review manipulation by using decision tree (DT) algorithm. In addition, we try to find the important review manipulation factors from a candidate attribute set via the analysis of extracted knowledge rules and correlation coefficients. It can help the potential consumers to realize the reality of online reviews and to reduce the uncertainty of purchase decision making. Pg. 2, When decision tree induction is used for feature selection, a tree is constructed from the given data. All attributes that do appear in the tree are assumed to be relevant. The set of attributes appearing in the tree form the reduced subset of attributes…Some data mining algorithms have built-in feature selections such as decision trees [35]. When decision tree induction is used for feature selection, a tree is constructed from the given data. All attributes that do appear in the tree are assumed to be relevant. The set of attributes appearing in the tree form the reduced subset of attributes. Pg. 3, The implemental procedure of the employed method can be listed in Fig. 1. The concise steps can be described as follows. In fact, there are 7 steps. They are “data collection”, “data preparation”, “define review manipulation attributes”, “created term-document matrix (TDM)”, “implement decision tree”, “identify key manipulation attributes”, and “performance & drawing conclusions”. Pg. 4, Step 4: Create TDM Step 4.1: Construct term-document matrix Every single review has been converted into a vector of terms with Term Frequency–Inverse Document Frequency (TF-IDF) weights. Then, based on defined features in step 2 and 3, the collected documents will be transformed to a term-document matrix (TDM). Step 4.2: Construct training/test sets In this step, 5 fold cross validation experiment has been employed for these training data. Depending on the employed feature sets, we have three experiments. They are described as bellow. Experiment #1: We only use original TDM with text features (unigram) Experiment #2: Combine the 8 review manipulation attribute set into the original TDM. Therefore, we use both original text features and review manipulation attributes to describe the collected data. Experiment #3: We only use review manipulation attributes to describe the collected data. Step 5: Implement decision tree algorithm (C4.5) Step 6: Identify key review manipulation attributes In this step, we use decision tree and correlation analysis to be our feature selection methods. Step 6.1: Construct 5 trees and pick the one who has the best performance Step 6.2: Interpret the knowledge rules discovered by decision tree. Step 6.3: Implement statistical correlation analysis Step 6.4: Acquire key attributes Step 7: Performance evaluation and drawing conclusions In this work, we use PA (Positive Accuracy), NA (Negative Accuracy), G-mean (Geometric mean of PA and NA), OA (Overall Accuracy), and F1 (an integrated index by using precision and recall.
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform the specific details of the decision tree, as taught/suggested by Chen. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to analyzing and using customer feedback. One of ordinary skill in the art would have recognized that applying the known technique of Chen would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Chen to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such specific details of the decision tree into similar systems. Further, applying the specific details of the decision tree, would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow for analysis of laid out options, consequences, and comparing outcomes.
Bennett teaches computing the economic impact using the feedback data. Bennett does not specifically teach the details of the electronic survey or binary assessments as claimed.
However, Smith teaches transmitting, to one or more customer computing devices, a link to an electronic survey (¶ 146-148, discloses transmitting a link to an electronic survey. ¶ 41, 61, 168, Fig. 6);
electronically present said electronic survey to said one or more customer computing devices by said link, wherein said electronic survey comprises binary problem assessments, each problem having two possible states (¶ 48, 140, 147-148, discloses electronic surveys with links. ¶ 114, 120, discloses a question two selectable options, yes or no.);
converting responses to said electronic survey to feedback data said feedback data comprising structured data (¶ 45, 74-75, 79-81, 90, 140, 143, discloses structuring electronic survey data into a database.);
receiving the feedback data (¶ 51, 53, discloses receiving feedback data.);
storing said feedback data in said memory (171-177, discloses storing data in a memory. ¶ 188-189).
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform the specific details of an electronic survey, as taught/suggested by Smith. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to surveys for receiving user information. One of ordinary skill in the art would have recognized that applying the known technique of Smith would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Smith to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such electronic survey features into similar systems. Further, applying electronic survey features would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow for a survey that is less time consuming, they are cheaper, you get the results faster, and you can transfer and use the data in various applications to answer important questions.
Bennett teaches computing the economic impact using the feedback data. Bennett does not specifically teach the details of the electronic survey or binary assessments as claimed.
However, Ghorbani teaches problem type data sets (¶ 111-116, categorizing keywords and types based on them being negative. ¶ 132, 176) wherein said electronic survey comprises binary problem assessments relating to one or more of said types of negative customer experiences, each problem having only two possible states (¶ 170, 176, discloses the use of the binary survey. ¶ 111, 116, discloses determining a positive or negative sentiment. ¶ 140-141, discloses a negative response/experience. ¶ 152, discloses a negative affinity. ¶ 119, 172, 187, 197, 201, 207).
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform the specific details of an electronic survey, as taught/suggested by Ghorbani. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to surveys for receiving user information. One of ordinary skill in the art would have recognized that applying the known technique of Ghorbani would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Ghorbani to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such electronic survey features into similar systems. Further, applying electronic survey features would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow for a survey that is less time consuming, they are cheaper, you get the results faster, and you can transfer and use the data in various applications to answer important questions.
Regarding claim 20, Bennett teaches a non-transitory computer-readable storage medium storing instructions which when executed adapt at least one computing device (¶ 17, 94-95, discloses the claimed technology. ¶ 38, 63, 66, 103-106);
maintain a data set including a plurality of types of negative customer experiences (¶ 25, discloses the use of customer questionnaires for customer experiences. ¶ 45, 54, discloses features that negatively impact a financial performance of an entity. ¶ 38, 78);
maintain a tree model for predicting economic impact of one or more of the plurality of types of negative customer experiences (¶ 25, discloses receiving questionnaire data in multiple forms. ¶ 45, discloses features that negatively impact a financial performance of an entity. ¶ 83, As shown by reference number 220, the machine learning system may train a machine learning model using the set of observations and using one or more machine learning algorithms, such as a regression algorithm, a decision tree algorithm, a neural network algorithm, a k-nearest neighbor algorithm, a support vector machine algorithm, and/or the like. After training, the machine learning system may store the machine learning model as a trained machine learning model 225 to be used to analyze new observations. ¶ 54, 38, 78);
receive feedback data (¶ 78-80, discloses a feature set may include one or more of the following features: employee satisfaction rating, customer satisfaction rating, a customer perceived usability answer or rating, a customer perceived value answer or rating, a customer perceived risk answer or rating, a customer convenience answer or rating, a technical innovation answer or rating, a functional quality answer or rating, a service quality answer or rating, a customer to employee engagement answer or rating, a brand trust answer or rating, a customer loyalty answer or rating, a Net Promoter Score, an overall customer experience answer or rating, an activity in social media answer or rating, an employee collaboration answer or rating, an employee creativity answer or rating, an employee empowerment answer or rating, an employee diversity answer or rating, an employee personal financial health answer or rating, an employee personal growth answer or rating, a leadership/direct supervisor answer or rating, an employee belief in company competitiveness answer or rating, an overall employee experience answer or rating, a technology work complexity answer or rating, a process work complexity answer or rating, an employee culture answer or rating, an employee self-efficacy answer or rating, a workplace analytics rating, an average employee salary, an average amount of time employees spend working, an average amount of time employees spend working outside of normal business hours, an employee efficiency rating, an entity financial health rating, and/or the like. ¶ 17-19, discloses various ways of receiving feedback in various formats ¶ 27, 38, 49, 57, 78);
storing said feedback data in said memory (¶ 103-105, discloses storing feedback data in a memory. ¶ 65-67);
generate a decision tree based on the tree model (¶ 83, As shown by reference number 220, the machine learning system may train a machine learning model using the set of observations and using one or more machine learning algorithms, such as a regression algorithm, a decision tree algorithm, a neural network algorithm, a k-nearest neighbor algorithm, a support vector machine algorithm, and/or the like. After training, the machine learning system may store the machine learning model as a trained machine learning model 225 to be used to analyze new observations.);
compute economic impact of at least one of the types of negative customer experiences using the feedback data (¶ 44-47, For example, for a feature, the data analytics model 102 may determine that below an influence threshold the feature does not impact business performance (e.g., financial performance) of an entity. That is, if the influence threshold is 55 and a current feature rank is 35, the data analytics model 102 may determine that even if the feature rank is increased from 35 to 50, the financial performance of the entity may not be impacted or may be minimally impacted. The data analytics model 102 may determine that above a feature rank of 55 (e.g., the influence threshold associated with the feature), the feature may begin to incrementally impact the financial performance of the entity as the feature rank of the feature is improved. ¶ 27, 38, 49, 57, 78, 17-19);
and cause to render, at a display screen, a graphic user interface visualizing the computed economic impact of at least one of the types of negative customer experiences (¶ 56-57, 63, discloses a data analytics platform for analyzing an economic impact of a customer experience. ¶ 45, 54, disclose a negative customer economic impact.)
Bennett does not specifically teach all the claimed details of the decision tree.
However the combination of Bennett and Chen teaches
generate a decision tree based on the tree model, the data set and the feedback data, the decision tree having a plurality of internal nodes with each internal node corresponding to a type of the plurality of types of negative customer experiences (Chen, Table I, pg. 1-2, Consequently, this study aims to introduce additional information such as readability, sentiment, product features and so on to improve the classification performance of review manipulation by using decision tree (DT) algorithm. In addition, we try to find the important review manipulation factors from a candidate attribute set via the analysis of extracted knowledge rules and correlation coefficients. It can help the potential consumers to realize the reality of online reviews and to reduce the uncertainty of purchase decision making. Pg. 2, When decision tree induction is used for feature selection, a tree is constructed from the given data. All attributes that do appear in the tree are assumed to be relevant. The set of attributes appearing in the tree form the reduced subset of attributes. Pg. 3, The implemental procedure of the employed method can be listed in Fig. 1. The concise steps can be described as follows. In fact, there are 7 steps. They are “data collection”, “data preparation”, “define review manipulation attributes”, “created term-document matrix (TDM)”, “implement decision tree”, “identify key manipulation attributes”, and “performance & drawing conclusions”. Pg. 4, Step 4: Create TDM Step 4.1: Construct term-document matrix Every single review has been converted into a vector of terms with Term Frequency–Inverse Document Frequency (TF-IDF) weights. Then, based on defined features in step 2 and 3, the collected documents will be transformed to a term-document matrix (TDM). Step 4.2: Construct training/test sets In this step, 5 fold cross validation experiment has been employed for these training data. Depending on the employed feature sets, we have three experiments. They are described as bellow. Experiment #1: We only use original TDM with text features (unigram) Experiment #2: Combine the 8 review manipulation attribute set into the original TDM. Therefore, we use both original text features and review manipulation attributes to describe the collected data. Experiment #3: We only use review manipulation attributes to describe the collected data. Step 5: Implement decision tree algorithm (C4.5) Step 6: Identify key review manipulation attributes In this step, we use decision tree and correlation analysis to be our feature selection methods. Step 6.1: Construct 5 trees and pick the one who has the best performance Step 6.2: Interpret the knowledge rules discovered by decision tree. Step 6.3: Implement statistical correlation analysis Step 6.4: Acquire key attributes Step 7: Performance evaluation and drawing conclusions In this work, we use PA (Positive Accuracy), NA (Negative Accuracy), G-mean (Geometric mean of PA and NA), OA (Overall Accuracy), and F1 (an integrated index by using precision and recall).
Bennett teaches computing the economic impact using the feedback data. The combination of Bennett and Chen teaches
compute economic impact of at least one of the types of negative customer experiences using the generated decision tree and the feedback data (Chen, Table I, pg. 3, 4) Positive Sentiment Using positive sentiment can guide readers easier than negative comments. Related works found that readers will ignore the negative comments, if they really want to purchase something [25]. 5) Negative Sentiment Berger et al. [9] found the negative comments of existing products will lead to the increase of sales, compared to the products which haven’t be discussed. 6) Sentiment One article which contains sentiments (no matter positive or negative) can influence the emotions of readers [20, 22]. This attribute combine the information involved in 4th and 5th attributes. pg. 1-2, Consequently, this study aims to introduce additional information such as readability, sentiment, product features and so on to improve the classification performance of review manipulation by using decision tree (DT) algorithm. In addition, we try to find the important review manipulation factors from a candidate attribute set via the analysis of extracted knowledge rules and correlation coefficients. It can help the potential consumers to realize the reality of online reviews and to reduce the uncertainty of purchase decision making. Pg. 2, When decision tree induction is used for feature selection, a tree is constructed from the given data. All attributes that do appear in the tree are assumed to be relevant. The set of attributes appearing in the tree form the reduced subset of attributes…Some data mining algorithms have built-in feature selections such as decision trees [35]. When decision tree induction is used for feature selection, a tree is constructed from the given data. All attributes that do appear in the tree are assumed to be relevant. The set of attributes appearing in the tree form the reduced subset of attributes. Pg. 3, The implemental procedure of the employed method can be listed in Fig. 1. The concise steps can be described as follows. In fact, there are 7 steps. They are “data collection”, “data preparation”, “define review manipulation attributes”, “created term-document matrix (TDM)”, “implement decision tree”, “identify key manipulation attributes”, and “performance & drawing conclusions”. Pg. 4, Step 4: Create TDM Step 4.1: Construct term-document matrix Every single review has been converted into a vector of terms with Term Frequency–Inverse Document Frequency (TF-IDF) weights. Then, based on defined features in step 2 and 3, the collected documents will be transformed to a term-document matrix (TDM). Step 4.2: Construct training/test sets In this step, 5 fold cross validation experiment has been employed for these training data. Depending on the employed feature sets, we have three experiments. They are described as bellow. Experiment #1: We only use original TDM with text features (unigram) Experiment #2: Combine the 8 review manipulation attribute set into the original TDM. Therefore, we use both original text features and review manipulation attributes to describe the collected data. Experiment #3: We only use review manipulation attributes to describe the collected data. Step 5: Implement decision tree algorithm (C4.5) Step 6: Identify key review manipulation attributes In this step, we use decision tree and correlation analysis to be our feature selection methods. Step 6.1: Construct 5 trees and pick the one who has the best performance Step 6.2: Interpret the knowledge rules discovered by decision tree. Step 6.3: Implement statistical correlation analysis Step 6.4: Acquire key attributes Step 7: Performance evaluation and drawing conclusions In this work, we use PA (Positive Accuracy), NA (Negative Accuracy), G-mean (Geometric mean of PA and NA), OA (Overall Accuracy), and F1 (an integrated index by using precision and recall.
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform the specific details of the decision tree, as taught/suggested by Chen. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to analyzing and using customer feedback. One of ordinary skill in the art would have recognized that applying the known technique of Chen would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Chen to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such specific details of the decision tree into similar systems. Further, applying the specific details of the decision tree, would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow for analysis of laid out options, consequences, and comparing outcomes.
Bennett teaches computing the economic impact using the feedback data. Bennett does not specifically teach the details of the electronic survey or binary assessments as claimed.
However, Smith teaches transmit, to one or more customer computing devices, a link to an electronic survey (¶ 146-148, discloses transmitting a link to an electronic survey. ¶ 41, 61, 168, Fig. 6);
electronically present said electronic survey to said one or more customer computing devices by said link, wherein said electronic survey comprises binary problem assessments, each problem having two possible states (¶ 48, 140, 147-148, discloses electronic surveys with links. ¶ 114, 120, discloses a question two selectable options, yes or no.);
converting responses to said electronic survey to feedback data said feedback data comprising structured data (¶ 45, 74-75, 79-81, 90, 140, 143, discloses structuring electronic survey data into a database.);
receive the feedback data (¶ 51, 53, discloses receiving feedback data.);
store said feedback data in said memory (171-177, discloses storing data in a memory. ¶ 188-189).
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform the specific details of an electronic survey, as taught/suggested by Smith. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to surveys for receiving user information. One of ordinary skill in the art would have recognized that applying the known technique of Smith would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Smith to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such electronic survey features into similar systems. Further, applying electronic survey features would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow for a survey that is less time consuming, they are cheaper, you get the results faster, and you can transfer and use the data in various applications to answer important questions.
Bennett teaches computing the economic impact using the feedback data. Bennett does not specifically teach the details of the electronic survey or binary assessments as claimed.
However, Ghorbani teaches problem type data sets (¶ 111-116, categorizing keywords and types based on them being negative. ¶ 132, 176) wherein said electronic survey comprises binary problem assessments relating to one or more of said types of negative customer experiences, each problem having only two possible states (¶ 170, 176, discloses the use of the binary survey. ¶ 111, 116, discloses determining a positive or negative sentiment. ¶ 140-141, discloses a negative response/experience. ¶ 152, discloses a negative affinity. ¶ 119, 172, 187, 197, 201, 207).
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform the specific details of an electronic survey, as taught/suggested by Ghorbani. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to surveys for receiving user information. One of ordinary skill in the art would have recognized that applying the known technique of Ghorbani would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Ghorbani to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such electronic survey features into similar systems. Further, applying electronic survey features would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow for a survey that is less time consuming, they are cheaper, you get the results faster, and you can transfer and use the data in various applications to answer important questions.
Claim(s) 2- 8, 12-18, is/are rejected under 35 U.S.C. 103 as being unpatentable over Bennett et al. (US 20210019357 A1) in view of Chen (2013) in view of Smith et al. (US 20180122256 A1) in view of Ghorbani et al. (US 20210150604 A1) in further view of Valdes et al (US 20230186106 A1).
Regarding claim 2 and 12, Bennet teaches a tree model but does not specifically teach a CART model.
However, Valdes teaches wherein the tree model is a classification and regression tree (CART) model and the decision tree is a binary tree (¶ 27, discloses machine learning classification, and a regression tree. ¶ 35, discloses a binary tree. ¶ 31, 69-71, 86-87, 94-99).
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform the specific details of a CART model, as taught/suggested by Valdes. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to improving decision trees. One of ordinary skill in the art would have recognized that applying the known technique of Valdes would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Valdes to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such specific tree model details into similar systems. Further, applying a CART model would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow searches for patterns and to help uncover hidden structure even in highly complex data.
Regarding claim 3 and 13, Bennet teaches wherein the decision tree comprises a class label indicating a classification of a type of negative customer experience corresponding (¶ 79, 45, discloses a decision tree and classification of a negative customer experience. ¶ 89, 54).
Bennett does not specifically teach a leaf of the decision tree or the internal node of the decision tree.
However, the combination of Bennett and Valdes teaches wherein each leaf of the decision tree comprises a class label indicating a classification of a type of experience corresponding to a given internal node of the decision tree (Valdes, ¶ 27, 25, discloses classification and leaf of a decision tree. ¶ 31, 34-35, 40, 69-71, 86-87, 93-99).
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform the specific details of an internal node of a decision tree, as taught/suggested by Valdes. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to improving decision trees. One of ordinary skill in the art would have recognized that applying the known technique of Valdes would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Valdes to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such specific decision tree details into similar systems. Further, applying an internal node of a decision tree would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow show each internal node (non-leaf node) which denotes a test on an attribute.
Regarding claim 4 and 14, Bennett teaches wherein the class label comprises a real value between 0 and 1, and wherein a value equal to or greater than 0.5 indicates that the associated type of negative customer experience has a meaningful economical impact (¶ 12, 30-31, discloses customer experience data. ¶ 45, 54, discloses negative impact. ¶ 56-57, disclose the use of historical data when determining impact. ¶ 38, 44-47, 63).
Regarding claim 5 and 15, Bennett teaches machine learning but does not does not specifically teach wherein the binary tree is generated using machine learning.
However, the combination of Bennett and Valdes teaches wherein the binary tree is generated using machine learning (¶ 27-29, discloses the use of machine learning. ¶ 88, discloses forming a binary tree. ¶ 25, FIG. 13 illustrates a heatmap linearly interpolating the weights associated with each instance for a disjoint region defined by one of the four leaf nodes of the trained tree, according to at least one embodiment of the invention. ¶ 124, discloses the use of CART. ¶ 34-35, 39, 69-71, 86-87, 93-99).
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform wherein the binary tree is generated using machine learning, as taught/suggested by Valdes. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to improving decision trees. One of ordinary skill in the art would have recognized that applying the known technique of Valdes would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Valdes to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such specific decision tree details into similar systems. Further, applying wherein the binary tree is generated using machine learning would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow the system to store data in a way that is easy to search and retrieve.
Regarding claims 6 and 16, Bennett teaches wherein generating the binary tree comprises: splitting the data set comprising the plurality of types of negative customer experiences into two groups based on a first cost function; and for each of the two groups: splitting the types of negative customer experiences in each respective group into two subsets based on a second cost function; and iteratively splitting the types of negative customer experiences in each respective subset into further binary subsets using a recursive binary splitting procedure until a predetermined threshold is reached (¶ 82, discloses creating clusters or groups. ¶ 87-89, discloses clusters as well as machine learning. ¶ 112-113, For example, the device may process, using a machine learning model trained based on data relating to one or more other entities, the set of questionnaire responses and the contextual data to identify a set of features and a set of feature ranks, as described above. In some implementations, each feature is associated with an influence threshold. ¶ 51, In some implementations, the data analytics platform 102 may select the subset of features based on determining one or more features that, if the feature rank is changed, will have a largest impact on a financial performance of an entity. In some implementations, the data analytics platform 102 may select the subset of features based on determining one or more features that, if the feature rank is changed, will have a largest impact on a financial performance of an entity with a lowest output of resources (e.g., computing resources, time resources, financial resources, and/or the like) by the entity. For example, the data analytics platform 102 may generate a resource cost of changing the feature rank of a feature. The data analytics platform 102 may generate an estimated result of changing the feature rank (e.g., an impact on an output of the machine learning model, an impact on a financial performance of an entity, and/or the like). In this way, the data analytics platform may identify the subset of features that may result in the largest result from changing the feature ranks with the lowest resource cost. In other words, the data analytics platform 102 may perform a cost-benefit analysis of a resource cost associated with improving a feature rank of a feature and a benefit that would be yielded from improving the feature rank of the feature (e.g., based on how close the feature rank is to an influence threshold, based on an impact on an output of the machine learning model, and/or the like). This conserves resources that would have otherwise been used changing the feature ranks of features that may satisfy an influence threshold, but may have a relatively low result from changing the feature rank, with a relatively high resource cost. ¶ 44-47, 87-89, 61, 62). Limitations also taught by Valdes (¶ 33, 87, 92).
Bennett does not specifically teach the details of the problem type data as claimed.
However, Ghorbani teaches problem type data sets (¶ 111-116, categorizing keywords and types based on them being negative. ¶ 132, 176).
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform the specific details of the data type, as taught/suggested by Ghorbani. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to surveys for receiving user information. One of ordinary skill in the art would have recognized that applying the known technique of Ghorbani would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Ghorbani to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such electronic survey features into similar systems. Further, applying the problem data would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow for a survey to focus on specific desired information for improvements.
Regarding claims 7 and 17, Bennett teaches machine learning but does not does not specifically teach wherein the an internal node of a binary tree.
However, the combination of Bennett and Valdes teaches wherein the predetermined threshold is a count on a total number of training instances assigned to each internal node of the binary tree (¶ 25-27, Machine Learning has evolved dramatically in recent years, and now is being applied to a broad spectrum of problems from computer vision to medicine. Specifically in medicine, a query of “machine learning” on www.pubmed.gov returns approximately 10,000 articles. The transition to the clinic, however, has seen limited success, and there has been little dissemination into clinical practice. Machine learning algorithms generally have some degree of inaccuracy, which leaves a user (e.g., a physician) with the question of what to do when their intuition and experience disagree with an algorithm's prediction. Most users might ignore the algorithm, in these cases, without being able to interpret how the algorithm computed its result. For this reason, some of the most widely used machine-learning based scoring or classification systems are highly interpretable. However, these systems generally trade off interpretability for accuracy. In medicine and other fields where misclassification has a high cost, while average prediction accuracy is a desirable trait; interpretability is as well. This is the reason why decision trees such as C4.5, ID3, and CART are popular in medicine. They can simulate the way physicians think by finding subpopulations of patients that all comply with certain rules and have the same classification. In a decision tree, these rules may be represented by nodes organized in a hierarchy, leading to a prediction. ¶ 88, Introduction of tree-structured boosting (TSB) as a new mechanism for creating a hierarchical ensemble model that recursively partitions the instance space, forming a perfect binary tree of weak learners. Each path from the root node to a leaf represents the outcome of a gradient-boosted ensemble for a particular partition of the instance space. ¶ 124, When A=0, the weights have binary normalized values that produce a sharp differentiation of the surface defined by the leaf node, similar to the behavior of CART, as illustrated in FIG. 13(a). As A increases in value, the weights become more diffuse in FIGS. 13(b) and 13(c), until A becomes significantly greater than 1. At that point, the weights approximate the initial values as anticipated by theory. Consequently, the ensembles along each path to a leaf are trained using equivalent instance weights, and therefore are the same and equivalent to gradient boosting. ¶ 34-35, 39, 69-71, 86-87, 93-99).
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform an internal node of a binary tree, as taught/suggested by Valdes. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to improving decision trees. One of ordinary skill in the art would have recognized that applying the known technique of Valdes would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Valdes to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such specific decision tree details into similar systems. Further, applying an internal node of a binary tree would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow the system to store data in a way that is easy to search and retrieve.
Regarding claims 8 and 18, Bennett teaches wherein splitting the types of negative customer experiences comprises selecting one type from the types of negative customer experiences and setting the selected type as an internal node (¶ 35, discloses specific correlations for specific features. ¶ 82, discloses outputting patterns. ¶ 54, discloses a negative effect on experience. ¶ 112-113, For example, the device may process, using a machine learning model trained based on data relating to one or more other entities, the set of questionnaire responses and the contextual data to identify a set of features and a set of feature ranks, as described above. In some implementations, each feature is associated with an influence threshold. ¶ 51, In some implementations, the data analytics platform 102 may select the subset of features based on determining one or more features that, if the feature rank is changed, will have a largest impact on a financial performance of an entity. In some implementations, the data analytics platform 102 may select the subset of features based on determining one or more features that, if the feature rank is changed, will have a largest impact on a financial performance of an entity with a lowest output of resources (e.g., computing resources, time resources, financial resources, and/or the like) by the entity. For example, the data analytics platform 102 may generate a resource cost of changing the feature rank of a feature. The data analytics platform 102 may generate an estimated result of changing the feature rank (e.g., an impact on an output of the machine learning model, an impact on a financial performance of an entity, and/or the like). In this way, the data analytics platform may identify the subset of features that may result in the largest result from changing the feature ranks with the lowest resource cost. In other words, the data analytics platform 102 may perform a cost-benefit analysis of a resource cost associated with improving a feature rank of a feature and a benefit that would be yielded from improving the feature rank of the feature (e.g., based on how close the feature rank is to an influence threshold, based on an impact on an output of the machine learning model, and/or the like). This conserves resources that would have otherwise been used changing the feature ranks of features that may satisfy an influence threshold, but may have a relatively low result from changing the feature rank, with a relatively high resource cost. ¶ 44-47, 87-89, 61, 62).
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bennett et al. (US 20210019357 A1) in view of Chen (2013) in view of Smith et al. (US 20180122256 A1) in view of Ghorbani et al. (US 20210150604 A1) in further view of Valdes et al (US 20230186106 A1) in further view of Burton et al. (US 11379863 B1).
Regarding claim 10, Bennett teaches feedback and financial impact but does not does not specifically teach details the claimed details about spending.
However, the combination of Bennett and Burton teaches determining, based on the feedback data, a first average amount of spending among a first group of customers that did not experience the at least one type of negative customer experience, the first group of customers from the plurality of customers; determining, based on the feedback data, a second average amount of spending among a second group of customers that experienced the at least one type of negative customer experience, the second group of customers from the plurality of customers; and determining the financial impact based on a difference between the first average amount of spending and the second average amount of spending (Burton discloses, col. 8, line 55 – col. 9, line 2, In addition, in some embodiments, the customer metric management circuit 260 determines a predicted future cash flow for the merchant 101 based on customer probability models. For example, if a customer is predicted to have a 75% chance of returning to the merchant 101 within the next month, and the customer is determined to typically spend $100 at the merchant 101 (or at other similar merchants), the customer metric management circuit 260 predicts $75 of revenue in the next month from that customer (e.g., the probability of return of the customer multiplied by the predicted amount the customer will spend at the merchant 101). In some embodiments, the customer metric management circuit 260 performs revenue prediction for multiple customers for aggregation to determine a predicted revenue for the merchant 101 over a certain period of time. Col. 11, lines 45-60, In some embodiments, the customer metric management circuit 260 attributes a desirability indicator to a customer indicating the value of the customer to the merchant 101. The desirability indicator is based on a customer's tendency to leave negative or positive reviews of businesses, tendency to post about experiences on social media, level of influence on social media, how wealthy the customer is, and so on. As such, the customer metric management circuit 260 provides a ranking or otherwise indicate customers along with their respective level of desirability to the merchant 101, so that the merchant 101 can provide incentives for highly desirable customers and/or disincentives for lowly desirable customers. Incentives include sending coupons or offers to these highly desirable customers, while disincentives include sending nothing to the lowly desirable customers. col. 6, line 60 – col. 8, line 55, col. 14, lines 9-67).
It would have been obvious to one of ordinary skill in the art at the time of filing to modify Bennett to include/perform details about spending, as taught/suggested by Burton. This known technique is applicable to the system of Bennett as they both share characteristics and capabilities, namely, they are directed to analyzing customer data. One of ordinary skill in the art would have recognized that applying the known technique of Burton would have yielded predictable results and resulted in an improved system. It would have been recognized that applying the technique of Burton to the teachings of Bennett would have yielded predictable results because the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate such spending feature details into similar systems. Further, applying details about spending would have been recognized by those of ordinary skill in the art as resulting in an improved system that would allow the system to store and have further data points.
Other pertinent prior art includes Katoh (US 20200090076 A1) which discloses a prediction method, and a learning device. GHORBANI et al. (US 20210150604 A1) which discloses customized creation of reviews and more customized presentation of reviews. Groarke et al. (US 20180285944 A1) which discloses providing spend profiles for reviewers in response to requests for validation of reviews submitted by the reviewer.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAMIE H AUSTIN whose telephone number is (571)272-7363. The examiner can normally be reached Monday, Tuesday, Thursday, Friday 7am-2pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian Epstein can be reached at (571) 270 5389. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JAMIE H. AUSTIN
Examiner
Art Unit 3625
/JAMIE H AUSTIN/Primary Examiner, Art Unit 3625
Prosecution Insights