DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
Acknowledgment is made of the information disclosure statements filed on 12/01/2023, 09/10/2025 and 1/202/2026.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on non-statutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1, 2, 7-10, 16 and are provisionally rejected on the ground of non-statutory double patenting as being unpatentable over claims 1-20 of US 20250124475 A1. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the issued patent are narrower in scope than claims of the instant application. Therefore, the claims of the issued patent anticipate the claims of the instant application. Please see the below claim mapping.
Each of the other claim map as the following, where (I) represents the instant application and (P) represents the issued patent: Claim 1+7+8+9 (I), Claim 1 (P); Claim 10 (I), Claim 13(P); Claim 16 (I), Claim 19(P); Claim 2 (I), Claim 4+5+6 (P);
Instant Application 18/485,233
EFD: 10/11/2023
US 20250124475 A1
1. A method, comprising: accessing, via one or more processors, training data to train a first layer of a machine learning model used within a machine learning environment, wherein the training data includes instances of user feedback; training, via the one or more processors, the first layer of the machine learning model, wherein the training includes providing instances of user feedback obtained from the training data to the first layer, and determining a first binary output for each of the instances of user feedback; removing, via the one or more processors and based, at least in part on training the first layer, a portion of the training data to create second training data; training, via the one or more processors, a second layer of the machine learning model, wherein the training includes providing second instances of user feedback obtained from the second training data to the second layer and determining a second binary output for each of the second instances of user feedback; and saving the machine learning model to a data store.
7. The method of claim 1, wherein the training data includes instances of user feedback that relates to at least one of one or more products, one or more services, or one or more features associated with a cloud service provider (CSP).
8. The method of claim 7, wherein the user feedback is obtained directly from at least one of one or more computing devices of CSP
9. The method of claim 7, wherein the user feedback is obtained from one or more external computing devices associated with at least one of a social network, a messaging service, an Internet forum, or a chat room that is separate from the CSP.
accessing, via one or more processors, training data to train a first layer of a machine learning model used within a machine learning environment, wherein the training data includes instances of user feedback;
training, via the one or more processors, the first layer of the machine learning model, wherein the training includes providing instances of user feedback obtained from the training data to the first layer, and
determining a first binary output for each of the instances of user feedback;
removing, via the one or more processors and based, at least in part on training the first layer, a portion of the training data to create second training data; training, via the one or more processors, a second layer of the machine learning model, wherein the training includes providing second instances of user feedback obtained from the second training data to the second layer and
determining a second binary output for each of the second instances of user feedback; and saving the machine learning model to a data store.
1. (Currently Amended) A method, comprising: determining, via one or more processors, user feedback data, wherein the user feedback data includes first user feedback data that is obtained from at least one of one or more computing devices of a cloud service provider (CSP), and
second user feedback data that is obtained from one or more external computing devices associated with at least one of a social network, a messaging service, an Internet forum, or a chat room;
monitor operations to generate, via the one or more processors, metrics associated with categories including a mindshare category, a happiness category, and one or more other categories that include at least one of an adoption category, a success of tasks category, an engagement category, or a retention category;
generating, via the one or more processors, a happiness score based, at least in part on the first user feedback data, wherein the happiness score indicates responses to one or more products of the CSP, one or more services of the CSP, or one or more features of the CSP;
generating, via the one or more processors, a mindshare score based, at least in part on the second user feedback data, wherein the mindshare score indicates responses to one or more products of the CSP, one or more services of the CSP, or one or more features of the CSP;
wherein generating at least one of the happiness score or the mindshare score comprises: training a first layer of one or more machine learning models to produce a first binary output that indicates a satisfaction ranking for instances of training data;
creating modified training data by removing the instances of the training data that have the satisfaction ranking;
training a second layer of the one or more machine learning models using the modified training data to produce a second binary output indicating a neutral or dissatisfied ranking; and
ordering a plurality of layers of the one or more machine learning models based on an amount of noise caused by different theme values in the training data;
generating, via the one or more processors, one or more other scores for individual ones of the one or more other categories;
generating a unified user experience (UX) score, via the one or more processors, based at least in part on the happiness score, the mindshare score and the one or more other scores, wherein each category associated with the scores contributes a predetermined percentage of the unified UX score; and
causing information about the unified UX score to be provided to a computing device associated with a user.
2. The method of claim 1, wherein the machine learning model is a sentiment model that performs sentiment analysis, and wherein the first layer of the sentiment model is trained to output a first binary value and the second layer is trained to output a second binary value.
1+4+5+6
Claim 10 (I) is a system claim with limitations similar to the limitations of instant method Claim 1 and is rejected under similar rationale.
Claim 16 (I) is a computer-readable medium claim with limitations similar to the limitations of instant method Claim 1 and is rejected under similar rationale.
Claim 11 (I) is a system claim with limitations similar to the limitations of instant method Claim 2 and is rejected under similar rationale.
Claim 17 (I) is a computer-readable medium claim with limitations similar to the limitations of instant method Claim 2 and is rejected under similar rationale.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 2, 10, 11, and 16, 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rahman et al. (“Multi-Tier Sentiment Analysis of Social Media Text Using Supervised
Machine Learning”) in view of Liu et al. (“Exploratory Undersampling for Class-Imbalance Learning”) in view of Curtin et al. US 10891374 B1.
Regarding Claim 1, Rahman teaches:
1. A method, comprising: accessing, via one or more processors, training data to train a first layer of a machine learning model used within a machine learning environment, wherein the training data includes instances of user feedback; FIG. 1, Supervised machine learning model used. Rahman teaches M2 Binary classifier (i.e. first layer)
(“abstract … In this study, pre-processing techniques and machine learning models for the multi-class classification of sentiments were explored. To augment the performance, a multi-layer classification model has been proposed. Owing to similitude with social media text, the movie reviews dataset has been used for the implementation. Supervised machine learning models namely Decision Tree, Support Vector Machine, and Naïve Bayes models have been
implemented for the task of sentiment classification. We have compared the models of single-layer architecture with multi-tier model. …”) (“Multi-tier architecture has been proposed for the multi-class SA. Their Multi-tier model hierarchically implements ternary and binary classifiers. Hence Machine Learning models are trained. Fig. 1 shows the flow diagram of the proposed methodology. The multi-tier model is comprised of three models. The training of the models takes simultaneously. Training data for M1 consists of positive, negative, and neutral text. Thus, M1 is trained to classify text into ternary classes. M2 and M3 are trained for binary classification. M2 is trained for positive and strong positive data. M3 is trained for negative and string negative text. Each classifier was trained using Naïve Bayes, Decision Tree, and SVM [47–53].” Page 5532, section 3.1) by Rahman et al. (“Multi-Tier Sentiment Analysis of Social Media Text Using Supervised Machine Learning”) {IDS provided}
Rahman further teaches:
wherein the training data includes instances of user feedback; FIG. 1, movie review Rahman teaches (“… The training of the models takes simultaneously. Training data for M1 consists of positive, negative, and neutral text. Thus, M1 is trained to classify text into ternary classes. M2 and M3 are trained for binary classification. M2 is trained for positive and strong positive data. M3 is trained for negative and string negative text. Each classifier was trained using Naïve Bayes, Decision Tree, and SVM [47–53].” Page 5532, section 3.1) (“The performance of the ML model is highly data-driven. A labelled dataset is required to perform supervised ML. Assigning 5 levels of sentiment to social media is a complex task. Labelling can either be done through human annotation or ML. Human annotation, although good in accuracy, is laborious. Thus, annotated Sentiment Analysis on Movie Reviews data set [12] tends to provide an optimal solution to this problem. In this study, data implementation has been performed on a publicly available movie review dataset. Table 1 shows the distribution of movie reviews among the classes. Most of the reviews belong to the neutral class. The extreme classes i.e., strong positive and string neutral contain minimum classes.” Page 5534, section 3.3.1 Dataset) by Rahman et al. (“Multi-Tier Sentiment Analysis of Social Media Text Using Supervised Machine Learning”) {IDS provided}
training, via the one or more processors, the first layer of the machine learning model, wherein the training includes providing instances of user feedback obtained from the training data to the first layer, and determining a first binary output for each of the instances of user feedback; Tables 4 and 5 tabulate the results of binary classifiers M2 (i.e. first layer) and M3 (i.e. second layer)., (“ section 3.1 … M2 is trained for positive and strong positive data….” Page 5532) in Fig. 1, data processing, positive and strong positive instance of movie review. by Rahman et al. (“Multi-Tier Sentiment Analysis of Social Media Text Using Supervised Machine Learning”) {IDS provided}
training, via the one or more processors, a second layer of the machine learning model, wherein the training includes providing second instances of user feedback[[ obtained from the second training data to the second layer ]] and FIG. 1, model training, Supervised model training phase and FIG. 8. Rahman teaches (“Multi-tier architecture has been proposed for the multi-class SA. Their Multi-tier model hierarchically implements ternary and binary classifiers. Hence Machine Learning models are trained. Fig. 1 shows the flow diagram of the proposed methodology. The multi-tier model is comprised of three models. The training of the models takes simultaneously. Training data for M1 consists of positive, negative, and neutral text. Thus, M1 is trained to classify text into ternary classes. M2 and M3 are trained for binary classification. M2 is trained for positive and strong positive data. M3 is trained for negative and string negative text. Each classifier was trained using Naïve Bayes, Decision Tree, and SVM [47–53].” Page 5532, section 3.1) (“Preprocessing was applied to the entire dataset and thus the feature vector caters to the words that are present in both test and training data. Thus, the preprocessed data is split into dev, train, and test sets according to the 10:70:20 ratio, respectively.” Page 5533, line 2-4) (“The performance of the ML model is highly data-driven. A labelled dataset is required to perform supervised ML. Assigning 5 levels of sentiment to social media is a complex task. Labelling can either be done through human annotation or ML. Human annotation, although good in accuracy, is laborious. Thus, annotated Sentiment Analysis on Movie Reviews data set [12] tends to provide an optimal solution to this problem. In this study, data implementation has been performed on a publicly available movie review dataset. Table 1 shows the distribution of movie reviews among the classes. Most of the reviews belong to the neutral class. The extreme classes i.e., strong positive and string neutral contain minimum classes.” Page 5534, section 3.3.1 Dataset) by Rahman et al. (“Multi-Tier Sentiment Analysis of Social Media Text Using Supervised Machine Learning”) {IDS provided}
Rahman further teaches:
determining a second binary output for each of the second instances of user feedback;
FIG. 1, movie review and FIG. 8. Rahman teaches (“… The training of the models takes simultaneously. Training data for M1 consists of positive, negative, and neutral text. Thus, M1 is trained to classify text into ternary classes. M2 and M3 are trained for binary classification. M2 is trained for positive and strong positive data. M3 is trained for negative and string negative text. Each classifier was trained using Naïve Bayes, Decision Tree, and SVM [47–53].” Page 5532, section 3.1) (“The performance of the ML model is highly data-driven. A labelled dataset is required to perform supervised ML. Assigning 5 levels of sentiment to social media is a complex task. Labelling can either be done through human annotation or ML. Human annotation, although good in accuracy, is laborious. Thus, annotated Sentiment Analysis on Movie Reviews data set [12] tends to provide an optimal solution to this problem. In this study, data implementation has been performed on a publicly available movie review dataset. Table 1 shows the distribution of movie reviews among the classes. Most of the reviews belong to the neutral class. The extreme classes i.e., strong positive and string neutral contain minimum classes.” Page 5534, section 3.3.1 Dataset) by Rahman et al. (“Multi-Tier Sentiment Analysis of Social Media Text Using Supervised Machine Learning”) {IDS provided}
Rahman does not explicitly teach removing a portion of the training data to create second training data.
Liu teaches:
removing, via the one or more processors and based, at least in part on training the first layer, a portion of the training data to create second training data and training a second layer…obtained from the second training data to the second layer; Liu teaches remove the majority instances that H1 correctly classify from N to shrink the pool. Train H2 on the shrunken N together with P. and repeat the process. iteration i=2, H2 trained on shrunken N and P. again producing binary positive/negative prediction per instance. (“where in each step, the majority class examples that are correctly classified by the current trained learners are removed from further consideration. … … This method is called BalanceCascade since it is somewhat similar to the cascade classifier in [34]. The majority training set N is shrunk after every Hi is trained, and every node Hi is dealing with a balanced subproblem (|Ni| = |P|). However, the final classifier is different. A cascade classifier is the conjunction of all {Hi}i=1,...,T , i.e., H(x) predicts positive if and only if all Hi(x)(i = 1, 2, . . . , T) predict positive. Viola and Jones [34] used the cascade classifier mainly to achieve fast testing speed. While in BalanceCascade, sequential dependence between classifiers is mainly exploited for reducing the redundant information in the majority class. This sampling strategy leads to a restricted sample space for the following undersampling process to explore as much useful information as possible. …” page 3, 2nd column paragraph after equation) and Liu teaches (“since BalanceCascade removes correctly classified majority class examples in each iteration, it will be more efficient on highly imbalanced data sets. …” page 11, column 1, 2nd paragraph of conclusion). entire SECTION B. Experimental Settings (page 5, entire column 1); entire section iii, Liu teaches multiple classifiers, each classifiers can be different layer, and training aspect of different classifiers and finally, combine all classifiers value to give final output. (“The idea behind EasyEnsemble is simple. Similar to the Balanced Random Forests [13], EasyEnsemble generates T balanced subproblems. The output of the ith subproblem is AdaBoost classifier Hi, an ensemble with si weak classifiers {hi,j}. An alternative view of hi,j is to treat it as a feature that is extracted by the ensemble learning method and can only take binary values [41]. Hi, in this viewpoint, is simply a linear classifier built on these features. Features extracted from different subsets Ni thus contain information of different aspects of the original majority training set N. Finally, instead of counting votes from {Hi}i=1,...,T , we collect all the features hi,j (i = 1, 2, . . . , T, j = 1, 2, . . . , si) and form an ensemble classifier from them.” Page 3, Column 1, first paragraph, section iii) (“Algorithm 2 The BalanceCascade algorithm. 1: {Input: A set of minority class examples P, a set of majority class examples N, |P| < |N |, the number of subsets …” page 3, 2nd column and algorithm 2) (“Conclusion Both algorithms sample multiple subsets of the majority class, train an ensemble from each of these subsets, and combine all weak classifiers in these ensembles into a final output. …” page 11, conclusion, column 1) by Liu et al. (“Exploratory Undersampling for Class-Imbalance Learning”)
Liu is considered to be analogous to the claimed invention because it relates BalanceCascade trains the learners sequentially, where in each step, the majority class examples that are correctly classified by the current trained learners are removed from further consideration.
Therefore, it would have been obvious for someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Rahman to incorporate the teachings of Liu in order to include feature removing and iteratively training.
One could have been motivated to do so because system can correctly classify data. (“Abstract … BalanceCascade trains the learners sequentially, where in each step, the majority class examples that are correctly classified by the current trained learners are removed from further consideration) by Liu et al. (“Exploratory Undersampling for Class-Imbalance Learning”)
The combination does not explicitly teach saving the machine learning model to a data store.
The combination does not explicitly teach saving the machine learning model to a data store.
Curtin teaches:
saving the machine learning model to a data store. Curtin teaches (“(77) In addition, one or more of the modules described herein may transform data, physical devices, and/or representations of physical devices from one form to another. For example, one or more of the modules recited herein may receive unweighted samples to train cascade classifiers of a machine-learning classification system, transform the unweighted samples into a weighted training dataset, output a result of the transformation to the machine-learning classification system, use the result of the transformation to determine whether an unknown sample falls within a classification (e.g., malicious, benign, etc.), and store the result of the transformation to a storage system. Additionally, or alternatively, one or more of the modules recited herein may transform a processor, volatile memory, non-volatile memory, and/or any other portion of a physical computing device from one form to another by executing on the computing device, storing data on the computing device, and/or otherwise interacting with the computing device.”) by Curtin et al. US 10891374 B1
Curtin is considered to be analogous to the claimed invention because it relates to systems may depend on machine-learning classifiers to detect when unknown samples come from a population of interest (e.g., spam, targeted emails, or malware).
Therefore, it would have been obvious for someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Rahman in view of Liu to incorporate the teachings of Curtin in order to include feature saving ML model to a data store.
One could have been motivated to do so because system can more accurately classify the unknown samples, thus improving the performance and accuracy of the cascade classifier. (“(10) … Thus, by using a modified training dataset to train a cascade classifier of a machine-learning classification system, the systems and methods described herein may enable a machine-learning classification system to more accurately classify the unknown samples, thus improving the performance and accuracy of the cascade classifier. Col. 4, lines 15-23) by Curtin et al. US 10891374 B1
Claim 10 is a system claim with a limitation similar to the limitation of combination of method Claim 1 is rejected under similar rationale.
Claim 16 is a non-transitory computer-readable medium claim with a limitation similar to the limitation of combination of device Claim 1 and is rejected under similar rationale.
Regarding Claim 2, the combination teaches the method of claim 1 as identified above.
2. The method of claim 1, wherein the machine learning model is a sentiment model that performs sentiment analysis, and Rahman further teaches (“The performance of the ML model is highly data-driven. A labelled dataset is required to perform supervised ML. Assigning 5 levels of sentiment to social media is a complex task. Labelling can either be done through human annotation or ML. Human annotation, although good in accuracy, is laborious. Thus, annotated Sentiment Analysis on Movie Reviews data set [12] tends to provide an optimal solution to this problem. In this study, data implementation has been performed on a publicly available movie review dataset. Table 1 shows the distribution of movie reviews among the classes. Most of the reviews belong to the neutral class. The extreme classes i.e., strong positive and string neutral contain minimum classes.” Page 5534, section 3.3.1 Dataset) by Rahman et al. (“Multi-Tier Sentiment Analysis of Social Media Text Using Supervised Machine Learning”) {IDS provided}
Rahman further teaches:
wherein the first layer of the sentiment model is trained to output a first binary value and FIG. 1, FIG. 8. (“Abstract … pre-processing techniques and machine learning models for the multi-class classification of sentiments were explored. To augment the performance, a multi-layer classification model has been proposed. Owing to similitude with social media text, the movie reviews dataset has been used for the implementation. Supervised machine learning models namely Decision Tree, Support Vector Machine, and Naïve Bayes models have been implemented for the task of sentiment classification. We have compared the models of single-layer architecture with multi-tier model. The results of Multi-tier model have slight improvement over the single-layer architecture. Moreover, multi-tier models have better recall which allow our proposed model to learn more context. …”) (“a multi-tier model has been proposed to augment the performance of multiclass sentiment classification. The proposed model has been implemented on the Sentiment Analysis on Movie Reviews data set [12]. Movie Reviews are classified into 5 classes i.e., positive, strongly positive,negative, strong negative, and neutral. Before the training and evaluation of the model, pre-processing techniques have been implemented for the SA task. For the implementation pre-processing NLP library,NLTK(natural language toolkit) has been utilized. TreeBank word tokenizer tokenizes the text data.WordNet Lemmatizer fulfills the need for text normalization. For the removal of stop words, the built-in list of stop words has been used. Lastly, a term frequency–Inverse document frequency (TFIDF) vectorization is performed to represent the textual data into numerical vector space that can be used to train the ML model. The multi-tier model decomposes the multiclass classification into binary and ternary classification sub-tasks. The ML models perform more optimally in case of binary and ternary classification.” Page 5528, last para.) by Rahman et al. (“Multi-Tier Sentiment Analysis of Social Media Text Using Supervised Machine Learning”) {IDS provided}
Rahman further teaches:
the second layer is trained to output a second binary value. FIG. 1, FIG. 8. (“Abstract … pre-processing techniques and machine learning models for the multi-class classification of sentiments were explored. To augment the performance, a multi-layer classification model has been proposed. Owing to similitude with social media text, the movie reviews dataset has been used for the implementation. Supervised machine learning models namely Decision Tree, Support Vector Machine, and Naïve Bayes models have been implemented for the task of sentiment classification. We have compared the models of single-layer architecture with multi-tier model. The results of Multi-tier model have slight improvement over the single-layer architecture. Moreover, multi-tier models have better recall which allow our proposed model to learn more context. …”) (“a multi-tier model has been proposed to augment the performance of multiclass sentiment classification. The proposed model has been implemented on the Sentiment Analysis on Movie Reviews data set [12]. Movie Reviews are classified into 5 classes i.e., positive, strongly positive,negative, strong negative, and neutral. Before the training and evaluation of the model, pre-processing techniques have been implemented for the SA task. For the implementation pre-processing NLP library,NLTK(natural language toolkit) has been utilized. TreeBank word tokenizer tokenizes the text data.WordNet Lemmatizer fulfills the need for text normalization. For the removal of stop words, the built-in list of stop words has been used. Lastly, a term frequency–Inverse document frequency (TFIDF) vectorization is performed to represent the textual data into numerical vector space that can be used to train the ML model. The multi-tier model decomposes the multiclass classification into binary and ternary classification sub-tasks. The ML models perform more optimally in case of binary and ternary classification.” Page 5528, last para.) by Rahman et al. (“Multi-Tier Sentiment Analysis of Social Media Text Using Supervised Machine Learning”) {IDS provided}
Claim 11 is a system claim with a limitation similar to the limitation of combination of method Claim 2 is rejected under similar rationale.
Claim 17 is a non-transitory computer-readable medium claim with a limitation similar to the limitation of combination of device Claim 2 and is rejected under similar rationale.
Claims 3, 12, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rahman, Liu and Curtin in view of SHENG et al. US 20190356956 A1.
Regarding Claim 3, the combination teaches the method of claim 2 as identified above.
The combination does not explicitly teach wherein the instances of user feedback include a text string that includes a numeric value that indicates a satisfaction rating.
SHENG teaches:
3. The method of claim 2, wherein the instances of user feedback include a text string that includes a numeric value that indicates a satisfaction rating. FIG. 7, SHENG teaches the channel metrics 714, training dataset 708 includes a predefined risk quantifier 710 in association with each channel. (“[0089] The channel metrics shown in the training dataset 708 are meant to be exemplary. Here, the channel metric “category” assumes a numerical value representative of a category in which the content of the association channel can generally be classified. …”) (“[0090] In addition to the channel metrics 714, training dataset 708 includes a predefined risk quantifier 710 in association with each channel. In FIG. 7, the predefined risk quantifier 710 is a value indicating whether or not the channel has been pre-identified as one that is likely (according to a computed metric) to invoke strong or negative sentiment in viewers. In the example shown, the predefined risk quantifier 710 indicates whether the channel is safe (indicating a negative risk metric below a set threshold) or unsafe (indicating a negative risk metric above a set threshold). In one implementation, the channel provider (e.g., YouTube) creates the predefined risk quantifier 710 for each channel. For example, the predefined risk quantifier 710 may be switched from “safe” (or logical state ‘1’) to “unsafe” (logical state ‘0’) when the digital media platforms forcibly removes one or more (e.g., a threshold number) of content items from the channel. In another implementation, the predefined risk metric may be a descriptive label such as very unsafe, unsafe, safe, very safe (although the title and the number of labels may be different). In yet another implementation, the predefined risk metric may include a numerical value (such as 0 to 5) with a low number such as 0 indicating a very risky channel and a high number such as 5 indicating a very risky channel.”) by SHENG et al. US 20190356956 A1
SHENG is considered to be analogous to the claimed invention because it relates to systems for intelligently predicting viewer sentiments.
Therefore, it would have been obvious for someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Rahman, Liu and Curtin to incorporate the teachings of SHENG in order to include feature binary output to the system.
One could have been motivated to do so because system can provide benefited service. (“[0022] … For example, the predictive tools may be utilized to determine that a particular channel has a strong or weak association one or more predefined risk categories including without limitation categories such as nudity, gore, profanity, violence, racially divisive content, politically-charged content, etc. Determining this type of association between an individual web-based channel and one or more risk categories may provide several benefits, such as allowing a content promotor to more objectively determine whether the web-based channel is good medium for broadcasting promotional content (e.g., a message, brand, service, product).”) by SHENG et al. US 20190356956 A1
Claim 12 is a system claim with a limitation similar to the limitation of combination of method Claim 3 is rejected under similar rationale.
Claim 18 is a non-transitory computer-readable medium claim with a limitation similar to the limitation of combination of device Claim 3 and is rejected under similar rationale.
Claim 4 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rahman, Liu and Curtin and further in view of Varghese et al. US 20190195525 A1
Regarding Claim 4, the combination teaches the method of claim 1 as identified above.
Rahman further teaches:
the second layer is trained to output a second binary value. FIG. 1, FIG. 8. Rahman teaches (“Abstract … pre-processing techniques and machine learning models for the multi-class classification of sentiments were explored. To augment the performance, a multi-layer classification model has been proposed. Owing to similitude with social media text, the movie reviews dataset has been used for the implementation. Supervised machine learning models namely Decision Tree, Support Vector Machine, and Naïve Bayes models have been implemented for the task of sentiment classification. We have compared the models of single-layer architecture with multi-tier model. The results of Multi-tier model have slight improvement over the single-layer architecture. Moreover, multi-tier models have better recall which allow our proposed model to learn more context. …”) (“a multi-tier model has been proposed to augment the performance of multiclass sentiment classification. The proposed model has been implemented on the Sentiment Analysis on Movie Reviews data set [12]. Movie Reviews are classified into 5 classes i.e., positive, strongly positive,negative, strong negative, and neutral. Before the training and evaluation of the model, pre-processing techniques have been implemented for the SA task. For the implementation pre-processing NLP library,NLTK(natural language toolkit) has been utilized. TreeBank word tokenizer tokenizes the text data.WordNet Lemmatizer fulfills the need for text normalization. For the removal of stop words, the built-in list of stop words has been used. Lastly, a term frequency–Inverse document frequency (TFIDF) vectorization is performed to represent the textual data into numerical vector space that can be used to train the ML model. The multi-tier model decomposes the multiclass classification into binary and ternary classification sub-tasks. The ML models perform more optimally in case of binary and ternary classification.” Page 5528, last para.) by Rahman et al. (“Multi-Tier Sentiment Analysis of Social Media Text Using Supervised Machine Learning”) {IDS provided}
The combination does not explicitly teach theme classification model that performs theme classification.
Varghese teaches:
4. The method of claim 1, wherein the machine learning model is a theme classification model that performs theme classification, and Varghese teaches (“[0072] A machine learning model, such as an analytic engine, may be generated from the training data set in a variety of ways. For instance, the purpose of a machine learning algorithm (MLA) may be to generate a machine learning model, such as a SVM-based classifier, e.g., a binary classifier and/or a linear binary classifier, a multi-class classifier, a kernel-based SVM, etc., a distance-based classifier, e.g., a Euclidean distance-based classifier, or the like, or a non-classifier type machine learning model, such as a decision tree, a KNN predictive model, a neural network, and so forth. For illustrative purposes, examples of the present disclosure are described herein primarily in connection with classifier type MLAs/machine learning models. In one example, the training data set may include labeled data which may be used in training the machine learning model to discriminate positive examples from negative examples. In another example where the classifier comprises a SVM, the machine learning algorithm may calculate a hyper-plane in a hyper-dimensional space representing the features space of all possible triggering parameters. The hyper-plane may define a boundary in the feature space which separates positive examples from negative examples. Once a classifier, or other type of machine learning model/predictive model, is generated for a particular network service and for a particular theme, the classifier may be applied to new data.”) by Varghese et al. US 20190195525 A1
wherein the first layer of the theme classification model is trained to output a first binary value and Varghese teaches machine learning model is generated for a particular network service and for a particular theme, the classifier may be applied to new data (“[0071] In step 255, the processor performs a remedial analytics associated with the equipment. For example, the analytic engine may comprise machine learning models (MLMs) that take a large number of inputs and attempt to determine whether a remedial action is necessary. For instance, the machine learning model may be initially trained with various historical data and then subsequently updated based on newly received data. In general, a system for generating a predictive model (a machine learning model) may collect, aggregate, and/or store various types of data that may be used as training data and which may be processed via the machine learning models. In accordance with the present disclosure, an analytic engine 105 (a machine learning model) may be created (trained) for a given remedial action based upon various data that may be collected in connection with the various sensors. Once trained, the analytic engine may be applied to new data that is collected to generate a recommended course of action, broadly a remedial action.”) (“[0072] A machine learning model, such as an analytic engine, may be generated from the training data set in a variety of ways. For instance, the purpose of a machine learning algorithm (MLA) may be to generate a machine learning model, such as a SVM-based classifier, e.g., a binary classifier and/or a linear binary classifier, a multi-class classifier, a kernel-based SVM, etc., a distance-based classifier … … The hyper-plane may define a boundary in the feature space which separates positive examples from negative examples. Once a classifier, or other type of machine learning model/predictive model, is generated for a particular network service and for a particular theme, the classifier may be applied to new data.”) by Varghese et al. US 20190195525 A1
Varghese is considered to be analogous to the claimed invention because it relates to a method of monitoring the efficiency and health of the HVAC unit is to gather and analyze the HVAC unit over a period of time.
Therefore, it would have been obvious for someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Rahman Liu and Curtin to incorporate the teachings of Varghese in order to include feature numeric value that indicates a satisfaction rating.
One could have been motivated to do so because system can have labelled training data so that positive and negative sample can be easily distinguished to feed to the classifier. (“[0072] … the training data set may include labeled data which may be used in training the machine learning model to discriminate positive examples from negative examples. ... … Once a classifier, or other type of machine learning model/predictive model, is generated for a particular network service and for a particular theme, the classifier may be applied to new data.”) by Varghese et al. US 20190195525 A1
Claim 13 is a system claim with a limitation similar to the limitation of combination of method Claim 4 is rejected under similar rationale
Claim 5, 6, 14, 15, and 19, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rahman, Liu, Curtin, Varghese, and further in view of Garvey et al. US 20240354789 A1.
Regarding Claim 5, the combination teaches the method of claim 4 as identified above.
The combination does not explicitly teach a text string that includes a numeric value that indicates a theme.
Garvey teaches:
5. The method of claim 4, wherein the instances of user feedback include a text string that includes a numeric value that indicates a theme. Garvey teaches (“[0025] … Qualitative data may be used to identify patterns or themes that are not easily quantifiable. Quantitative results are based on numerical data that may be measured and analyzed using statistical methods. Quantitative data may measure or otherwise quantify various facets of a user experience, which may be useful to benchmark and compare the performance of a product relative to peers in the same or similar product space.”) by Garvey et al. US 20240354789 A1
Garvey is considered to be analogous to the claimed invention because it relates to a techniques are described for producing machine-generating findings associated with user experiences with products and/or services.
Therefore, it would have been obvious for someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Rahman, Liu, Curtin, Varghese to incorporate the teachings of Garvey in order to include a text string that includes a numeric value that indicates a theme.
One could have been motivated to do so because system can have test result which reflect customer experience that leads to improve overall experience of the product (“[0003] … Test results may include qualitative and quantitative data that provide insights into user experiences with a product. Such insights may help isolate problematic areas of a product's design and guide product design updates to improve the overall experience when using a product.”) by Garvey et al. US 20240354789 A1
Claim 14 is a system claim with a limitation similar to the limitation of combination of method Claim 5 is rejected under similar rationale.
Claim 19 is a non-transitory computer-readable medium claim with a limitation similar to the limitation of combination of device Claim 5 and is rejected under similar rationale.
Regarding Claim 6, the combination teaches the method of claim 4 as identified above.
Rahman further teaches:
6. The method of claim 4, further comprising: removing, via the one or more processors and based, at least in part on training the second layer, a portion of the training data from the second training data to create third training data; and Rahman teaches FIG. 1 data processing, FIG. 2 shows data processing includes text cleaning, normalization, tokenization and vector representation and then training “supervised model training” , then different classifier process different data. Finally start testing phase with training data in FIG. 1. Rahman teaches M3 (i.e. second layer). in figure 1 In figure 2, text cleaning. (“ section 3.1 … Training data for M1 consists of positive, negative, and neutral text. …”). In figure 2, shows text cleaning.
training, via the one or more processors, a third layer of the machine learning model, wherein the training includes providing third instances of user feedback obtained from the third training data to the third layer FIG. 1, ternary classifier M1 (i.e. third layer), FIG. 8. FIG. 1, model training, Supervised model training phase and FIG. 8. Rahman teaches ternary classifier, each classifier can be each layer. (“Multi-tier architecture has been proposed for the multi-class SA. Their Multi-tier model hierarchically implements ternary and binary classifiers. Hence Machine Learning models are trained. Fig. 1 shows the flow diagram of the proposed methodology. The multi-tier model is comprised of three models. The training of the models takes simultaneously. Training data for M1 consists of positive, negative, and neutral text. Thus, M1 is trained to classify text into ternary classes. M2 and M3 are trained for binary classification. M2 is trained for positive and strong positive data. M3 is trained for negative and string negative text. Each classifier was trained using Naïve Bayes, Decision Tree, and SVM [47–53].” Page 5532, section 3.1) (“Preprocessing was applied to the entire dataset and thus the feature vector caters to the words that are present in both test and training data. Thus, the preprocessed data is split into dev, train, and test sets according to the 10:70:20 ratio, respectively.” Page 5533, line 2-4) (“The performance of the ML model is highly data-driven. A labelled dataset is required to perform supervised ML. Assigning 5 levels of sentiment to social media is a complex task. Labelling can either be done through human annotation or ML. Human annotation, although good in accuracy, is laborious. Thus, annotated Sentiment Analysis on Movie Reviews data set [12] tends to provide an optimal solution to this problem. In this study, data implementation has been performed on a publicly available movie review dataset. Table 1 shows the distribution of movie reviews among the classes. Most of the reviews belong to the neutral class. The extreme classes i.e., strong positive and string neutral contain minimum classes.” Page 5534, section 3.3.1 Dataset) (“Abstract … pre-processing techniques and machine learning models for the multi-class classification of sentiments were explored. To augment the performance, a multi-layer classification model has been proposed. Owing to similitude with social media text, the movie reviews dataset has been used for the implementation. Supervised machine learning models namely Decision Tree, Support Vector Machine, and Naïve Bayes models have been implemented for the task of sentiment classification. We have compared the models of single-layer architecture with multi-tier model. The results of Multi-tier model have slight improvement over the single-layer architecture. Moreover, multi-tier models have better recall which allow our proposed model to learn more context. …”) (“a multi-tier model has been proposed to augment the performance of multiclass sentiment classification. The proposed model has been implemented on the Sentiment Analysis on Movie Reviews data set [12]. Movie Reviews are classified into 5 classes i.e., positive, strongly positive,negative, strong negative, and neutral. Before the training and evaluation of the model, pre-processing techniques have been implemented for the SA task. For the implementation pre-processing NLP library,NLTK(natural language toolkit) has been utilized. TreeBank word tokenizer tokenizes the text data.WordNet Lemmatizer fulfills the need for text normalization. For the removal of stop words, the built-in list of stop words has been used. Lastly, a term frequency–Inverse document frequency (TFIDF) vectorization is performed to represent the textual data into numerical vector space that can be used to train the ML model. The multi-tier model decomposes the multiclass classification into binary and ternary classification sub-tasks. The ML models perform more optimally in case of binary and ternary classification.” Page 5528, last para.) by Rahman et al. (“Multi-Tier Sentiment Analysis of Social Media Text Using Supervised Machine Learning”) {IDS provided}
and determining a third binary output for each of the third instances of user feedback. FIG. 1 shows instance of movie review, neutral at data processing in FIG.1. Also, M1 ternary classifier. FIG. 3 shows histogram of sentiment distribution. FIG. 8. Table 3 depicts the results of ternary
classifier M1. Rahman teaches (“The multi-tier model decomposes the multiclass classification into binary and ternary classification sub-tasks. The ML models perform more optimally in case of binary and ternary classification. … … The performance metrics of the multi-tier model are compared with those of the single-tier model. Also, the performance metrics of the binary and ternary classifiers that constitute the multi-tier model are analyzed. …” page 5529, lines 1-10) (“The performance of the ML model is highly data-driven. A labelled dataset is required to perform supervised ML. Assigning 5 levels of sentiment to social media is a complex task. Labelling can either be done through human annotation or ML. Human annotation, although good in accuracy, is laborious. Thus, annotated Sentiment Analysis on Movie Reviews data set [12] tends to provide an optimal solution to this problem. In this study, data implementation has been performed on a publicly available movie review dataset. Table 1 shows the distribution of movie reviews among the classes. Most of the reviews belong to the neutral class. The extreme classes i.e., strong positive and string neutral contain minimum classes.” Page 5534, section 3.3.1 Dataset) by Rahman et al. (“Multi-Tier Sentiment Analysis of Social Media Text Using Supervised Machine Learning”) {IDS provided}
Claim 15 is a system claim with a limitation similar to the limitation of combination of method Claim 6 is rejected under similar rationale.
Claim 20 is a non-transitory computer-readable medium claim with a limitation similar to the limitation of combination of device Claim 5 and is rejected under similar rationale.
Claim 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rahman, Liu, Curtin, and further in view of Madduri et al. US 20210192540 A1.
Regarding Claim 7, the combination teaches the method of claim 1 as identified above.
The combination does not explicitly teach one or more services, or one or more features associated with a cloud service provider (CSP).
Madduri teaches:
7. The method of claim 1, wherein the training data includes instances of user feedback that relates to at least one of one or more products, one or more services, or one or more features associated with a cloud service provider (CSP). Madduri teaches (“[0094] FIGS. 8A-8F illustrate example UI views 800 of the mobile compliance application 112 on a mobile device 102 having a display device 302 for providing user feedback to the cloud based computer vision compliance system 170 by adding or modifying a product tag for a product that was not recognized in an example compliance audit.”) by Madduri et al. US 20210192540 A1
Madduri is considered to be analogous to the claimed invention because it relates to disclosed herein are system, method, and computer program product embodiments for compliance auditing using cloud-based computer vision.
Therefore, it would have been obvious for someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Rahman, Liu, and Curtin to incorporate the teachings of Madduri in order to include feature a cloud service provider (CSP).
One could have been motivated to do so because system can continuously improve the object recognition capabilities of one or more object recognition models through one or more compliance audits.. (“[0019] … This training via a mobile compliance application allows one or more users to continuously improve the object recognition capabilities of one or more object recognition models through one or more compliance audits. …”) by Madduri et al. US 20210192540 A1
Claim 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rahman, Liu, Curtin, Madduri and further in view of RAWAT et al. US 20220358529 A1.
Regarding Claim 8, the combination teaches the method of claim 7 as identified above.
The combination does not explicitly teach the user feedback is obtained directly from at least one of one or more computing devices of CSP.
RAWAT teaches:
8. The method of claim 7, wherein the user feedback is obtained directly from at least one of one or more computing devices of CSP. RAWAT teaches (“[0034] The online user feedback management application 205 includes hardware and software logic for implementing partial or full functions of the online user feedback management application 105. For example, the online user feedback management application 205 may implement certain topic identification, identifying a priority topic, and/or prioritizing actions for the priority topic, as will be described in detail in FIG. 2B. In some implementations, the online user feedback management application 205 may be a light version of the online user feedback management application 105, and thus include some simple processing, but leave some complex computations to the online user feedback management application 105, which be located on a cloud-based online user feedback management server 101 that has more computation power.”) by RAWAT et al. US 20220358529 A1
RAWAT is considered to be analogous to the claimed invention because it relates to disclosed herein are system, method, and computer program product embodiments for compliance auditing using cloud-based computer vision.
Therefore, it would have been obvious for someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Rahman, Liu, Curtin and Madduri to incorporate the teachings of RAWAT in order to include feature the user feedback is obtained directly from at least one of one or more computing devices of CSP.
One could have been motivated to do so because system can have user feedback platform tool that allows users or customers to provide online user feedback towards a product, service, website, etc. and user experience (UX) designers to address issues related to the product, service, or website, so as to improve the relevant product, service, or website for better user experience. (“[0001] An online user feedback platform is a tool that allows users or customers to provide online user feedback towards a product, service, website, etc. Online user feedbacks provide valuable information from users or customers about their reactions to a product, service, or website experience, which can be used by marketers, researchers, and user experience (UX) designers to address issues related to the product, service, or website, so as to improve the relevant product, service, or website for better user experience. …”) by RAWAT et al. US 20220358529 A1
Claim 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over RAHMAN, Liu, Curtin, Madduri, RAWAT and further in view of Stewart et al. US 12147763.
Regarding Claim 9, the combination teaches the method of claim 7 as identified above.
Stewart further teaches:
9. The method of claim 7, wherein the user feedback is obtained from one or more external computing devices associated with at least one of a social network, a messaging service, an Internet forum, or a chat room that is separate from the CSP. Stewart teaches (“(2) With the advancement of the computer and network technologies, various operations performed by users from different applications lead to extensive use of web services. This proliferation of the Internet and Internet based user activity continues to create a vast amount of digital content. For example, multiple users may provide feedback (e.g., responses to surveys, social media reviews, and opinions otherwise expressed during an interaction) about a business entity (e.g., a hotel or a restaurant) via different applications, such as mobile applications running on different platforms, as well as web-interfaces running on different browsers in different operating systems. Furthermore, users may also use different social media outlets to post their feedback about the business entity.”) (“(3) An application server for the business entity may store the feedback in a local storage device. Machine learning techniques can be used on the feedback to determine useful properties of the feedback, e.g. its sentiment. Sentiment analysis involves determining whether the text in feedback expresses positive, negative, neutral, or mixed sentiments. Such sentiment analysis typically uses a historic data set for training a sentiment analysis model. For example, a sentiment analysis model can be trained using a training data set that has been labeled by a user (e.g., the sentiments have been identified by the user). The trained model learns the associations between various language patterns and the corresponding sentiments in the training data set. The trained model is then used to analyze subsequent new data sets. When the trained model is used to analyze new data sets similar to the training data set, the model can achieve high accuracy.”) by Stewart et al. US 12147763 B2
Stewart is considered to be analogous to the claimed invention because it relates identifying relevantly similar sentiment in text strings.
Therefore, it would have been obvious for someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Rahman, Liu Curtin, Madduri, and RAWAT to incorporate the teachings of Stewart in order to include social network, a messaging service, an Internet forum, or a chat room that is separate from the CSP.
One could have been motivated to do so because system will improve model on sentiment analysis. (“(18) … This way, each client can update its client specific posting list by populating it with corrected sentiments. This way, each client can benefit from using the data set used to power universal sentiment analysis model 102 to obtain an initial judgement sentiment, and based on a confidence of the initial judgement sentiment, RSA model 160 can be leveraged to confirm whether initial judgement sentiment is the correct resolved sentiment or whether a secondary sentiment—obtained from the client specific posting list—is the correct resolved sentiment.” Col. 5, lines 25-50) by Stewart et al. US 12147763 B2
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to FOUZIA HYE SOLAIMAN whose telephone number is (571)270-5656. The examiner can normally be reached M-F (8-5)AM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras D. Shah can be reached at (571) 270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/F.H.S./Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
10/01/2026
Prosecution Insights