DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on February 23, 2026 has been entered. Claims 1, 8, and 17 have been amended. Claims 2, 4-5, 9, 13, and 20 are canceled. Claims 21-24 have been added. Claims 1, 3, 6-8, 10-12, 14-19 and 21-24 are presented for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed February 23, 2026 have been considered but they are not fully persuasive.
Regarding the rejection under 35 U.S.C. § 101, Applicant’s claim amendments have overcome the rejection. The Examiner agrees that the claimed algorithm is intrinsic to technical operations and presents more than a significant link to technology, thereby rendering the claims patent-eligible under 35 U.S.C. § 101.
Regarding the art rejection, Applicant generally asserts what is and what is not taught individually by each of Jezewski, Biessmann, Huang, and Givental (pages 12-13 of Applicant’s response); however, the claims are rejected in view of the combination of these references. Applicant has not addressed the combination, including the teachings and suggestions contributed by each reference to the rejection as a whole. Also, the Edwards reference has been incorporated into the rejections of the claims that recite tokenization and the Chelba reference has been incorporated into the rejections of some of the new claims.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 3, 6-8, 10-12, 14-19 and 21-24 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Independent claims 1, 8, and 17 have been amended to incorporate various references to “feature vectors.” While Applicant’s original disclosure explains that text data may be collected “based on each business target feature” (Spec: ¶ 42), this is not necessarily the same as creating a vector of the features. Applicant’s original disclosure does not provide support for the claimed recitations of “feature vectors.”
Independent claim 1 has been amended to recite “entity relationships derived from tokenized text data.” Applicant’s original disclosure does not refer to any variation of the word “token” or to any concept that seems to be synonymous with tokenization (or even a discussion of the related field of natural language processing). The original disclosure talks about identifying a role for each word, such as verbal phrases and nouns (Spec: ¶ 30), and the Examiner finds that this concept supports the amended language “syntactic roles”; however, this is not necessarily the same as “tokenization.” Applicant’s original disclosure does not provide support for the claimed recitation of “entity relationships derived from tokenized text data.”
Independent claims 1, 8, and 17 have been amended to incorporate various references to “divergence.” For example, claim 1 recites “wherein the feedback mechanism modifies stored model parameters associated with at least one of the classification logic or the score logic based on divergence between model outputs.” Claim 8 recites “modifying stored model parameters based on divergence between outputs.” Claim 17 recites “modifying stored model parameters based on divergence between outputs.” Additionally, new claim 23 recites “wherein modifying stored model parameters includes iteratively adjusting model weights based on measured divergence between scoring outputs and intent identification outputs.” There is no mention of divergence or synonyms thereof in Applicant’s original disclosure. Applicant’s Specification discusses using two outputs together with weighted balance. For example, paragraph 49 of Applicant’s Specification states, “In the illustrated implementation of FIG. 1D, the feedback 132 to acquire accurate measure of audience interest level combines outputs 160, 164 from both the information extractor 146/intent identifier 148 and the trained classifier 142/scorer 144. As indicated above, the information extractor 146/intent identifier 148 combination finds the data 160 with clear intent, while the trained classifier 142/scorer 144 combination adds labels with probability to the data without clearly identified intent to produce output 164.In this case, the outputs 160, 164 from the two paths can be used together, with weighted balance depending on the contribution to the business strategy refinement. For example, the text 160 with clear intent can have higher significance than the text 164 identified by the second path.” However, it is not clear that the various outputs (e.g., from the classifier and scorer) are necessarily divergent. Applicant’s original disclosure does not provide support for the claimed recitations of “divergence.”
New claim 21 recites “wherein the feedback mechanism dynamically updates numerical coefficients stored in non-transitory memory that parameterize the trained statistical model of the score logic.” Applicant’s Specification refers to weighted balance being based on the contribution of two paths (Spec: ¶¶ 40, 49) and mentions that older intents may be updated (Spec: ¶ 25). However, it is not explained that the weights are necessarily numerical coefficients, much less that the numerical coefficients parameterize the trained statistical model of the score logic. Applicant’s Specification adds labels with probability (Spec: ¶¶ 34, 40, 43, 48, 49) and the probability may be interpreted as a statistic, but this level of disclosure alone is insufficient to fully provide support for the limitation “wherein the feedback mechanism dynamically updates numerical coefficients stored in non-transitory memory that parameterize the trained statistical model of the score logic.” Applicant’s original disclosure does not provide support for this limitation.
New claim 24 recites “wherein the stored instructions cause the hardware controller to perform weighting operations without permitting manual assignment of weights by a user.” While Applicant’s Specification refers to using a computer to score labeled data and assigned intent (Spec: ¶¶ 11, 36), Applicant’s Specification does not actively prohibit a human user from assigning weights. Therefore, this limitation is deemed to present new matter.
The respective dependent claims inherit the aforementioned issues.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 6, 7, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Jezewski (US 2019/0251626) in view of Biessmann et al. (US 9,892,133) in view of Huang et al. (US 2014/0250032) in view of Givental et al. (US 2021/0281592) in view of Edwards et al. (US 2016/0012038).
[Claim 1] Jezewski discloses a system implemented as a text analysis application executed by a hardware controller to analyze text data and social media posts to determine an audience interest level associated with business target features (¶¶ 11-12, 120), the system comprising:
a data aggregation logic executed by the text analysis application to collect text data associated with at least one of the business target feature through a hardware network (¶¶ 15-20 – Indications of user sentiment toward companies may be gathered; ¶ 12 – “As shown in FIG. 1A, example implementation 100 may include a client device, comment sources, and a sentiment analysis platform. Assume that a user utilizes the client device to access one or more applications provided by the comment sources. In some implementations, the comment sources may include sources that provide social media applications, blog applications, chat room applications, message board applications, ratings system applications, and/or the like. As further shown in FIG. 1A, the user may utilize the client device to provide complaint information, about an entity (e.g., company A), to the one or more applications provided by the comment sources. For example, the complaint information may include a complaint indicating that the user is switching from company A to company B, a complaint indicating a negative statement about company A, a complaint indicating invalidation of a product or service of company A, and/or the like.”; ¶ 120 – “Some implementations described herein may provide a sentiment analysis platform that utilizes artificial intelligence to make a prediction about an entity based on user sentiment and transaction history. For example, the sentiment analysis platform may consider sentiments of users who have opinions, complaints, and predictions about the entity, and transactions conducted by users with the entities in order to predict a future stock price of the entity. The sentiment analysis platform may receive the opinions, the complaints, and the predictions of the users, about the entity, from social media sources, and may receive transaction information associated with the users and the entity from financial institutions. The sentiment analysis platform may determine correlations between the transaction information and the opinions, the complaints, and the predictions, in order to apply weights to the opinions, the complaints, and the predictions. The sentiment analysis platform may generate a prediction about the future stock price of the entity based on the opinions, the complaints, the predictions, the transaction information, and the correlations between the transaction information and the opinions, the complaints, and the predictions.” Comments made on/via social media and/or in association with a social media source are examples of social media posts.);
an intent identification logic executed by the text analysis application (¶¶ 36, 39, 92, 113-118 – Information related to an intent is extracted and identified.; ¶ 120 – AI sentiment analysis platform.), the intent identification logic including:
an information extraction logic configured to extract information from the collected text data by generating machine-encoded feature vectors representing syntactic roles, metadata, actions, entities, and entity relationships derived from text data (¶¶ 43-45; ¶¶ 13, 116 – The nature of the opinions, which is related to sentiment analysis, can include an assessment of actions, such as a desire to switch from one company to another or an action to be taken in regard to an entity; Jezewski implements its sentiment analysis using machine learning and/or natural language processing (¶ 36), which means that text data and related scoring data will be classified (i.e., labeled) and trained; Jezewski provides relevant details of the disclosed classification and label processing in ¶¶ 38-47. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) Identifying noun phrases and verb phrases is an example of identifying syntactic roles.; The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40). The fact that certain news in one domain may be good news and that the same news may be bad news in another domain is an example of scoring (e.g., as “good” or “bad”) based on intent.; ¶¶ 36, 39, 92, 113-118 – Information related to an intent is extracted and identified.; ¶¶ 38-40 – Text, i.e., metadata, is classified to identify entities and sentiment based on opinion information and in light of the different domains and respective set of features for different domains.; ¶¶ 15-20 – Indications of user sentiment toward companies may be gathered. This demonstrates entity relationships.; ¶ 46 – “In some implementations, the sentiment analysis platform may utilize a maximum entropy classifier model. A maximum entropy classifier model may convert labeled feature sets to vectors using encoding. The encoded vector may then be used to calculate weights for each feature, which may then be combined to determine a most likely label for a feature set.”), and
wherein the intent identification logic is configured to identify intent actions based on the generated machine-encoded feature vectors including related entities (¶¶ 36, 39, 92, 113-118 – Information related to an intent is extracted and identified.; ¶ 120 – AI sentiment analysis platform.; ¶ 46 – “In some implementations, the sentiment analysis platform may utilize a maximum entropy classifier model. A maximum entropy classifier model may convert labeled feature sets to vectors using encoding. The encoded vector may then be used to calculate weights for each feature, which may then be combined to determine a most likely label for a feature set.”);
a classification logic configured to assign at least one label to portions of the collected text data, the classification logic including a trained statistical model stored in non-transitory memory (¶¶ 44-48; ¶¶ 59-60 – “[0059] As shown in FIG. 1F, and by reference number 155, the sentiment analysis platform may utilize the transaction information to determine correlations with the interim scores (e.g., the necessity score, the abstract score, the ethics score, the industry score, the demands for service score, the supply for service score, the vendor score, the innovation score, the adaptability score, the execution score, and/or the like). In some implementations, the sentiment analysis platform may utilize a correlation clustering method, a Pearson's product-moment coefficient method, an Anscombe's quartet method, a Spearman's rank-correlation coefficient method, and/or the like in order to determine the correlations between the transaction information and the interim scores. [0060] A clustering method may include partitioning data points into groups based on their similarity, and the correlation clustering method may include clustering a set of objects into an optimum number of clusters without specifying that number in advance. Given a collection of paired (x, y) variables, the Pearson's product-moment coefficient method produces a value, between −1 and +1, that quantifies a strength of dependence between the variables x and y. A value of +1 means that all of the (x, y) points lie exactly on a line with positive slope, a value of −1 means that all of the points lie exactly on a line with negative slope, and a value of 0 means that there is no relationship between the two variables. The Anscombe's quartet method utilizes four datasets that have nearly identical simple descriptive statistics, yet appear very different when graphed. The Spearman's rank-correlation coefficient method provides a nonparametric measure of rank correlation (e.g., a statistical dependence between a ranking of two variables), and assesses how well a relationship between two variables can be described using a monotonic function.“; ¶ 88 – “Device 300 may perform one or more processes described herein. Device 300 may perform these processes based on processor 320 executing software instructions stored by a non-transitory computer-readable medium, such as memory 330 and/or storage component 340. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.”);
a score logic configured to compute intent scores for labelled data using a trained statistical model stored in non-transitory memory (¶ 36 – “In some implementations, the artificial intelligence techniques may include a sentiment analysis model that utilizes multiple artificial analysis techniques. In some implementations, the sentiment analysis model may include a model that uses natural language processing, text analysis, and machine learning to systematically identify, extract, quantify, and study affective states and subjective information.”; ¶¶ 59-60 – “[0059] As shown in FIG. 1F, and by reference number 155, the sentiment analysis platform may utilize the transaction information to determine correlations with the interim scores (e.g., the necessity score, the abstract score, the ethics score, the industry score, the demands for service score, the supply for service score, the vendor score, the innovation score, the adaptability score, the execution score, and/or the like). In some implementations, the sentiment analysis platform may utilize a correlation clustering method, a Pearson's product-moment coefficient method, an Anscombe's quartet method, a Spearman's rank-correlation coefficient method, and/or the like in order to determine the correlations between the transaction information and the interim scores. [0060] A clustering method may include partitioning data points into groups based on their similarity, and the correlation clustering method may include clustering a set of objects into an optimum number of clusters without specifying that number in advance. Given a collection of paired (x, y) variables, the Pearson's product-moment coefficient method produces a value, between −1 and +1, that quantifies a strength of dependence between the variables x and y. A value of +1 means that all of the (x, y) points lie exactly on a line with positive slope, a value of −1 means that all of the points lie exactly on a line with negative slope, and a value of 0 means that there is no relationship between the two variables. The Anscombe's quartet method utilizes four datasets that have nearly identical simple descriptive statistics, yet appear very different when graphed. The Spearman's rank-correlation coefficient method provides a nonparametric measure of rank correlation (e.g., a statistical dependence between a ranking of two variables), and assesses how well a relationship between two variables can be described using a monotonic function.“; ¶¶ 43-45; ¶¶ 13, 116 – The nature of the opinions, which is related to sentiment analysis, can include an assessment of actions, such as a desire to switch from one company to another or an action to be taken in regard to an entity; Jezewski implements its sentiment analysis using machine learning and/or natural language processing (¶ 36), which means that text data and related scoring data will be classified (i.e., labeled) and trained; Jezewski provides relevant details of the disclosed classification and label processing in ¶¶ 38-47. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40). The fact that certain news in one domain may be good news and that the same news may be bad news in another domain is an example of scoring (e.g., as “good” or “bad”) based on intent.; ¶ 88 – “Device 300 may perform one or more processes described herein. Device 300 may perform these processes based on processor 320 executing software instructions stored by a non-transitory computer-readable medium, such as memory 330 and/or storage component 340. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.”).
While Jezewski strongly suggests that the disclosed operations may be performed substantially simultaneously (i.e., in parallel) since the various models may be applied in conjunction with one another, including to classify and extract data (as seen in ¶¶ 36-46 of Jezewski), Jezewski does not explicitly disclose:
wherein the data aggregation logic routes the collected text data in parallel to the classification logic and the information extraction logic;
wherein outputs of the score logic and the intent identification logic are supplied to a feedback mechanism.
Biessmann verifies item attributes in an artificial intelligence environment and explains how the extraction and classification may be performed in parallel, as seen in the following excerpt:
The classifier training module 150, the feature extractor module 152, and/or the classifier module 154 can operate in parallel and for multiple users at the same time. For example, the verification of attributes listed in an item description may be requested from unique user devices 102 for different listings and the components of the attribute verification system 104 can verify the listed attributes simultaneously or nearly simultaneously for the unique user devices 102 in “real-time,” where “real-time” may be based on the perspective of the user. The generation and verification of attributes may be considered to occur in real time if, for example, the delay is sufficiently short (e.g., less than a few seconds) such that the user typically would not notice a processing delay. Real-time may also be based on the following: the attribute verification system 104 can verify listed attributes for a first user at the same time or at nearly the same time (e.g., within a couple seconds) as a verification of listed attributes is performed for a second user; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously as instantaneously as possible, limited by processing resources, available memory, network bandwidth conditions, and/or the like; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously based on a time it takes the hardware components of the attribute verification system 104 to process data; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously immediately as data is received (instead of storing, buffering, caching, or persisting data as it is received and processing the data later on); the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously by transforming data without intentional delay, given the processing limitations of the attribute verification system 104 and other systems, like the user devices 102, and the time required to accurately receive and/or transmit the data; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously by processing or transforming data fast enough to keep up with an input data stream; etc. (Biessmann: col. 12: 13-51)
Furthermore, Biessmann uses feedback to update the training, thereby confirming which suggested attributes are correct and incorrect (Biessmann: col 3: 5-24; col. 5: 41-58; col. 9: 26 – col. 10: 3; col. 14: 7-14).
While Biessmann does not explicitly link the classification and extraction to an environment in which an audience sentiment is evaluated, Huang sheds some additional light on the benefits of performing operations related to extraction (like sentiment analysis) simultaneously with classification in the area of user sentiment analysis, as described in the following excerpts of Huang:
[0002] Sentiment and topic analysis have a wide application in business marketing and customer care applications to assist in evaluating and understanding brand perception and customer requirements based on, for example, data gathered from millions of online posts such as social media, forums, and blogs. For example, when promoting a new policy/product, a company may monitor electronically posted customer comments regarding a particular policy/product so that the company can respond properly and address criticisms and issues in a timely manner. Hence, online monitoring of current sentiment trend and topics related to, for example, a preset product and brand name is important for modern marketing…
[0011] The aforementioned aspects and other objectives and advantages can now be achieved as described herein. Methods, systems and processor-readable media for simultaneous sentiment analysis and topic classification with multiple labels are disclosed herein. A sentiment and topic associated with a post can be classified at similar time and a result can be incorporated to predict a feature so that a label of two tasks can promote and reinforce each other iteratively. A feature extraction and selection can be performed on both tasks of sentiment and topic classification. A multi-task multi-label classification model can be trained for each task with maximum entropy utilizing multiple labels to ascertain data indicative of and/or derived from an extra label and to manage with class ambiguities. Each task has a separate classification model with different predicting features and they can be trained collectively which allows flexibility in model construction. Such multi-task multi-label (MTML) classification model produces a probabilistic result and the classes can be ranked by the probabilistic result and the post can be classified with the multi-label.
As discussed above, Jezewski uses machine learning and semantic analysis to glean user sentiment. The Examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of Applicant’s invention to modify Jezewski:
wherein the data aggregation logic routes the collected text data in parallel to the classification logic and the information extraction logic;
wherein outputs of the score logic and the intent identification logic are supplied to a feedback mechanism
in order to minimize delay in the sentiment analysis (as suggested in Biessmann: col. 12: 13-51) and so that the various dimensions of gathered information may be used to reinforce each other (as suggested in ¶ 11 of Huang) and to improve the accuracy of the models (as suggested in Biessmann: col 3: 5-24; col. 5: 41-58; col. 9: 26 – col. 10: 3; col. 14: 7-14).
Jezewski, Biessmann, and Huang do not explicitly disclose:
wherein the feedback mechanism modifies stored model parameters associated with at least one of the classification logic or the score logic based on divergence between model outputs; and
wherein a weighted balance between outputs of the score logic and the intent identification logic is computed using a machine-generated weighting function derived from model confidence metrics.
Givental uses hybrid machine learning to detect anomalies using feedback from the outputs of various learning models of an ensemble to assign weights to the various models based on whether or not each respective model has outputted a correct result (Givental: ¶¶ 52, 68). As discussed above, Jezewski evaluates outputs related to intent identification logic and information extraction. The Examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of Applicant’s invention to modify the Jezewski-Biessmann-Huang combination:
wherein the feedback mechanism modifies stored model parameters associated with at least one of the classification logic or the score logic based on divergence between model outputs; and
wherein a weighted balance between outputs of the score logic and the intent identification logic is computed using a machine-generated weighting function derived from model confidence metrics
in order to improve the overall accuracy of the ensemble of learning models (as suggested in ¶ 24 of Givental).
Jezewski does not explicitly disclose that the feature vectors are derived specifically from tokenized text data. Edwards explains that a natural language processing program identifies a part of speech of each token (e.g., each word or phrase) on an n-gram and a corresponding confidence level and the natural language processing program can use deep parsers (Edwards: ¶ 30). Jezewski performs natural language processing using a deep parser (Jezewski: ¶ 37). The Examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of Applicant’s invention to modify Jezewski such that the feature vectors are derived specifically from tokenized text data because the substitution of Edward’s approach to deep parsing in the area of natural language processing through tokenization for Jezewski’s approach to deep parsing in the area of natural language processing would have been well within the technical capability of those skilled in the art prior to Applicant’s invention and such a substitution would have yielded predictable and expected results.
[Claim 3] Jezewski discloses wherein the score logic adds probability to the at least one assigned label, wherein the probability indicates how likely each labelled data belongs to the at least one assigned label (¶¶ 43-45).
[Claim 6] Jezewski discloses wherein output of the intent identification logic couples to input of the classification logic so that the extracted information without clearly identified intent is sent to the classification logic (¶¶ 36-46 – The various models may be applied in conjunction with one another, including to classify and extract data. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40)).
[Claim 7] Jezewski discloses wherein the intent identification logic couples to the feedback so that the extracted information with clearly identified intent is sent to the feedback (¶¶ 36-46 – The various models may be applied in conjunction with one another, including to classify and extract data; ¶¶ 43-63, 111-112, 120 – A probabilistic classifier model may be used for sentiment analysis. Weights may be applied as part of training models in regard to interim scores, opinions, complaints, predictions, labeling features, identifying correlations among the various data and scores, etc. Feedback may refer to the feedback from customers and/or to feedback used to train the models).
[Claim 22] Jezewski does not explicitly disclose wherein the machine-generated weighting function is computed using quantified confidence scores produced by both the classification logic and the intent identification logic. Edwards explains that a natural language processing program identifies a part of speech of each token (e.g., each word or phrase) on an n-gram and a corresponding confidence level and the natural language processing program can use deep parsers (Edwards: ¶ 30). Edwards further states, “In operation 212, NLP program 104 adjusts the confidence level of the expanded n-gram. In one embodiment, NLP program 104 adjusts the confidence level based on the semantic analysis and the identified parts of speech of the expanded n-gram. In one embodiment, NLP program 104 adjusts the confidence level of an expanded n-gram by combining (e.g., by an average or by a weighted average) the confidence level associated with the identification of the part of speech of each token of the expanded n-gram (see operation 210) with the confidence level of the expanded n-gram from NLP data 106 (see operation 206).” (Edwards: ¶ 31) Jezewski performs natural language processing using a deep parser (Jezewski: ¶ 37). Similar to Jezewski’s predictive probability that a given feature set belongs to a label (Jezewski: ¶ 44) (which reflects a level of confidence), Edwards explains that “the initial confidence level represents a probability that the unigram is of a semantic type identified by the initial confidence level” (Edwards: ¶ 5). The Examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of Applicant’s invention to modify Jezewski such that the feature vectors are derived specifically from tokenized text data because the substitution of Edward’s approach to deep parsing in the area of natural language processing through the use of weighted functions and confidence scores for Jezewski’s approach to deep parsing in the area of natural language processing would have been well within the technical capability of those skilled in the art prior to Applicant’s invention and such a substitution would have yielded predictable and expected results.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Jezewski (US 2019/0251626) in view of Biessmann et al. (US 9,892,133) in view of Huang et al. (US 2014/0250032) in view of Givental et al. (US 2021/0281592) in view of Edwards et al. (US 2016/0012038), as applied to claim 1 above, in view of Chelba et al. (US 2006/0074630).
[Claim 21] Jezewski does not explicitly disclose wherein the feedback mechanism dynamically updates numerical coefficients stored in non-transitory memory that parameterize the trained statistical model of the score logic. Like Jezewski, Chelba performs natural language processing with feature selection and a statistical classifier (Chelba: ¶ 62). Additionally, Chelba constructs a statistical classifier comprising parameterized conditional likelihood over training data (Chelba: claims 2, 11, 12) and model parameters are estimated using coefficients and weights and model parameters are iteratively re-estimated (Chelba: ¶¶ 43-47). Chelba constructs a statistical classifier or model to be used later and stores the classifier construction module (Chelba: ¶¶ 31-32). The Examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of Applicant’s invention to modify the Jezewski-Biessmann-Givental-Edwards-Huang combination wherein the feedback mechanism dynamically updates numerical coefficients stored in non-transitory memory that parameterize the trained statistical model of the score logic because “[o]ne advantage of the growth transform re-estimation procedure is that the present model parameters are re-normalized or tuned at each iteration, thus maintaining parameters in the parameter space X.” (Chelba: ¶ 45)
Claims 8, 10-12, 14-19, and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Jezewski (US 2019/0251626) in view of Biessmann et al. (US 9,892,133) in view of Huang et al. (US 2014/0250032) in view of Givental et al. (US 2021/0281592).
[Claim 8] A computer-implemented method executed by a hardware controller operating a text analysis application to analyze text data and social media posts (¶¶ 11-12, 120), the method comprising:
collecting text data associated with at least one business target feature (¶¶ 15-20 – Indications of user sentiment toward companies may be gathered; ¶ 12 – “As shown in FIG. 1A, example implementation 100 may include a client device, comment sources, and a sentiment analysis platform. Assume that a user utilizes the client device to access one or more applications provided by the comment sources. In some implementations, the comment sources may include sources that provide social media applications, blog applications, chat room applications, message board applications, ratings system applications, and/or the like. As further shown in FIG. 1A, the user may utilize the client device to provide complaint information, about an entity (e.g., company A), to the one or more applications provided by the comment sources. For example, the complaint information may include a complaint indicating that the user is switching from company A to company B, a complaint indicating a negative statement about company A, a complaint indicating invalidation of a product or service of company A, and/or the like.”; ¶ 120 – “Some implementations described herein may provide a sentiment analysis platform that utilizes artificial intelligence to make a prediction about an entity based on user sentiment and transaction history. For example, the sentiment analysis platform may consider sentiments of users who have opinions, complaints, and predictions about the entity, and transactions conducted by users with the entities in order to predict a future stock price of the entity. The sentiment analysis platform may receive the opinions, the complaints, and the predictions of the users, about the entity, from social media sources, and may receive transaction information associated with the users and the entity from financial institutions. The sentiment analysis platform may determine correlations between the transaction information and the opinions, the complaints, and the predictions, in order to apply weights to the opinions, the complaints, and the predictions. The sentiment analysis platform may generate a prediction about the future stock price of the entity based on the opinions, the complaints, the predictions, the transaction information, and the correlations between the transaction information and the opinions, the complaints, and the predictions.” Comments made on/via social media and/or in association with a social media source are examples of social media posts.);
extracting information from the text data by generating machine-encoded feature vectors representing syntactic roles, metadata, actions, entities, and entity relationships (¶¶ 43-45; ¶¶ 13, 116 – The nature of the opinions, which is related to sentiment analysis, can include an assessment of actions, such as a desire to switch from one company to another or an action to be taken in regard to an entity; Jezewski implements its sentiment analysis using machine learning and/or natural language processing (¶ 36), which means that text data and related scoring data will be classified (i.e., labeled) and trained; Jezewski provides relevant details of the disclosed classification and label processing in ¶¶ 38-47. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) Identifying noun phrases and verb phrases is an example of identifying syntactic roles.; The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40). The fact that certain news in one domain may be good news and that the same news may be bad news in another domain is an example of scoring (e.g., as “good” or “bad”) based on intent.; ¶¶ 36, 39, 92, 113-118 – Information related to an intent is extracted and identified.; ¶¶ 38-40 – Text, i.e., metadata, is classified to identify entities and sentiment based on opinion information and in light of the different domains and respective set of features for different domains.; ¶¶ 15-20 – Indications of user sentiment toward companies may be gathered. This demonstrates entity relationships.; ¶ 46 – “In some implementations, the sentiment analysis platform may utilize a maximum entropy classifier model. A maximum entropy classifier model may convert labeled feature sets to vectors using encoding. The encoded vector may then be used to calculate weights for each feature, which may then be combined to determine a most likely label for a feature set.”);
identifying intent actions using the generated machine-encoded feature vectors (¶¶ 36, 39, 92, 113-118 – Information related to an intent is extracted and identified.; ¶ 120 – AI sentiment analysis platform.; ¶ 46 – “In some implementations, the sentiment analysis platform may utilize a maximum entropy classifier model. A maximum entropy classifier model may convert labeled feature sets to vectors using encoding. The encoded vector may then be used to calculate weights for each feature, which may then be combined to determine a most likely label for a feature set.”);
a classification logic configured to assign at least one label to portions of the collected text data, the classification logic including a trained statistical model stored in non-transitory memory (¶¶ 44-48; ¶¶ 59-60 – “[0059] As shown in FIG. 1F, and by reference number 155, the sentiment analysis platform may utilize the transaction information to determine correlations with the interim scores (e.g., the necessity score, the abstract score, the ethics score, the industry score, the demands for service score, the supply for service score, the vendor score, the innovation score, the adaptability score, the execution score, and/or the like). In some implementations, the sentiment analysis platform may utilize a correlation clustering method, a Pearson's product-moment coefficient method, an Anscombe's quartet method, a Spearman's rank-correlation coefficient method, and/or the like in order to determine the correlations between the transaction information and the interim scores. [0060] A clustering method may include partitioning data points into groups based on their similarity, and the correlation clustering method may include clustering a set of objects into an optimum number of clusters without specifying that number in advance. Given a collection of paired (x, y) variables, the Pearson's product-moment coefficient method produces a value, between −1 and +1, that quantifies a strength of dependence between the variables x and y. A value of +1 means that all of the (x, y) points lie exactly on a line with positive slope, a value of −1 means that all of the points lie exactly on a line with negative slope, and a value of 0 means that there is no relationship between the two variables. The Anscombe's quartet method utilizes four datasets that have nearly identical simple descriptive statistics, yet appear very different when graphed. The Spearman's rank-correlation coefficient method provides a nonparametric measure of rank correlation (e.g., a statistical dependence between a ranking of two variables), and assesses how well a relationship between two variables can be described using a monotonic function.“; ¶ 88 – “Device 300 may perform one or more processes described herein. Device 300 may perform these processes based on processor 320 executing software instructions stored by a non-transitory computer-readable medium, such as memory 330 and/or storage component 340. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.”);
assigning at least one label using a trained classification model stored in non-transitory memory (¶ 36 – “In some implementations, the artificial intelligence techniques may include a sentiment analysis model that utilizes multiple artificial analysis techniques. In some implementations, the sentiment analysis model may include a model that uses natural language processing, text analysis, and machine learning to systematically identify, extract, quantify, and study affective states and subjective information.”; ¶¶ 59-60 – “[0059] As shown in FIG. 1F, and by reference number 155, the sentiment analysis platform may utilize the transaction information to determine correlations with the interim scores (e.g., the necessity score, the abstract score, the ethics score, the industry score, the demands for service score, the supply for service score, the vendor score, the innovation score, the adaptability score, the execution score, and/or the like). In some implementations, the sentiment analysis platform may utilize a correlation clustering method, a Pearson's product-moment coefficient method, an Anscombe's quartet method, a Spearman's rank-correlation coefficient method, and/or the like in order to determine the correlations between the transaction information and the interim scores. [0060] A clustering method may include partitioning data points into groups based on their similarity, and the correlation clustering method may include clustering a set of objects into an optimum number of clusters without specifying that number in advance. Given a collection of paired (x, y) variables, the Pearson's product-moment coefficient method produces a value, between −1 and +1, that quantifies a strength of dependence between the variables x and y. A value of +1 means that all of the (x, y) points lie exactly on a line with positive slope, a value of −1 means that all of the points lie exactly on a line with negative slope, and a value of 0 means that there is no relationship between the two variables. The Anscombe's quartet method utilizes four datasets that have nearly identical simple descriptive statistics, yet appear very different when graphed. The Spearman's rank-correlation coefficient method provides a nonparametric measure of rank correlation (e.g., a statistical dependence between a ranking of two variables), and assesses how well a relationship between two variables can be described using a monotonic function.“; ¶¶ 43-45; ¶¶ 13, 116 – The nature of the opinions, which is related to sentiment analysis, can include an assessment of actions, such as a desire to switch from one company to another or an action to be taken in regard to an entity; Jezewski implements its sentiment analysis using machine learning and/or natural language processing (¶ 36), which means that text data and related scoring data will be classified (i.e., labeled) and trained; Jezewski provides relevant details of the disclosed classification and label processing in ¶¶ 38-47. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40). The fact that certain news in one domain may be good news and that the same news may be bad news in another domain is an example of scoring (e.g., as “good” or “bad”) based on intent.; ¶ 88 – “Device 300 may perform one or more processes described herein. Device 300 may perform these processes based on processor 320 executing software instructions stored by a non-transitory computer-readable medium, such as memory 330 and/or storage component 340. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.”);
computing intent scores using a trained scoring model stored in non-transitory memory (¶ 36 – “In some implementations, the artificial intelligence techniques may include a sentiment analysis model that utilizes multiple artificial analysis techniques. In some implementations, the sentiment analysis model may include a model that uses natural language processing, text analysis, and machine learning to systematically identify, extract, quantify, and study affective states and subjective information.”; ¶¶ 59-60 – “[0059] As shown in FIG. 1F, and by reference number 155, the sentiment analysis platform may utilize the transaction information to determine correlations with the interim scores (e.g., the necessity score, the abstract score, the ethics score, the industry score, the demands for service score, the supply for service score, the vendor score, the innovation score, the adaptability score, the execution score, and/or the like). In some implementations, the sentiment analysis platform may utilize a correlation clustering method, a Pearson's product-moment coefficient method, an Anscombe's quartet method, a Spearman's rank-correlation coefficient method, and/or the like in order to determine the correlations between the transaction information and the interim scores. [0060] A clustering method may include partitioning data points into groups based on their similarity, and the correlation clustering method may include clustering a set of objects into an optimum number of clusters without specifying that number in advance. Given a collection of paired (x, y) variables, the Pearson's product-moment coefficient method produces a value, between −1 and +1, that quantifies a strength of dependence between the variables x and y. A value of +1 means that all of the (x, y) points lie exactly on a line with positive slope, a value of −1 means that all of the points lie exactly on a line with negative slope, and a value of 0 means that there is no relationship between the two variables. The Anscombe's quartet method utilizes four datasets that have nearly identical simple descriptive statistics, yet appear very different when graphed. The Spearman's rank-correlation coefficient method provides a nonparametric measure of rank correlation (e.g., a statistical dependence between a ranking of two variables), and assesses how well a relationship between two variables can be described using a monotonic function.“; ¶¶ 43-45; ¶¶ 13, 116 – The nature of the opinions, which is related to sentiment analysis, can include an assessment of actions, such as a desire to switch from one company to another or an action to be taken in regard to an entity; Jezewski implements its sentiment analysis using machine learning and/or natural language processing (¶ 36), which means that text data and related scoring data will be classified (i.e., labeled) and trained; Jezewski provides relevant details of the disclosed classification and label processing in ¶¶ 38-47. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40). The fact that certain news in one domain may be good news and that the same news may be bad news in another domain is an example of scoring (e.g., as “good” or “bad”) based on intent.; ¶ 88 – “Device 300 may perform one or more processes described herein. Device 300 may perform these processes based on processor 320 executing software instructions stored by a non-transitory computer-readable medium, such as memory 330 and/or storage component 340. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.”).
While Jezewski strongly suggests that the disclosed operations may be performed substantially simultaneously (i.e., in parallel) since the various models may be applied in conjunction with one another, including to classify and extract data (as seen in ¶¶ 36-46 of Jezewski), Jezewski does not explicitly disclose:
routing collected text data in parallel with the generated machine-encoded feature vectors;
supplying outputs of intent identification and scoring operations to a feedback mechanism.
Biessmann verifies item attributes in an artificial intelligence environment and explains how the extraction and classification may be performed in parallel, as seen in the following excerpt:
The classifier training module 150, the feature extractor module 152, and/or the classifier module 154 can operate in parallel and for multiple users at the same time. For example, the verification of attributes listed in an item description may be requested from unique user devices 102 for different listings and the components of the attribute verification system 104 can verify the listed attributes simultaneously or nearly simultaneously for the unique user devices 102 in “real-time,” where “real-time” may be based on the perspective of the user. The generation and verification of attributes may be considered to occur in real time if, for example, the delay is sufficiently short (e.g., less than a few seconds) such that the user typically would not notice a processing delay. Real-time may also be based on the following: the attribute verification system 104 can verify listed attributes for a first user at the same time or at nearly the same time (e.g., within a couple seconds) as a verification of listed attributes is performed for a second user; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously as instantaneously as possible, limited by processing resources, available memory, network bandwidth conditions, and/or the like; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously based on a time it takes the hardware components of the attribute verification system 104 to process data; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously immediately as data is received (instead of storing, buffering, caching, or persisting data as it is received and processing the data later on); the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously by transforming data without intentional delay, given the processing limitations of the attribute verification system 104 and other systems, like the user devices 102, and the time required to accurately receive and/or transmit the data; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously by processing or transforming data fast enough to keep up with an input data stream; etc. (Biessmann: col. 12: 13-51)
Furthermore, Biessmann uses feedback to update the training, thereby confirming which suggested attributes are correct and incorrect (Biessmann: col 3: 5-24; col. 5: 41-58; col. 9: 26 – col. 10: 3; col. 14: 7-14).
While Biessmann does not explicitly link the classification and extraction to an environment in which an audience sentiment is evaluated, Huang sheds some additional light on the benefits of performing operations related to extraction (like sentiment analysis) simultaneously with classification in the area of user sentiment analysis, as described in the following excerpts of Huang:
[0002] Sentiment and topic analysis have a wide application in business marketing and customer care applications to assist in evaluating and understanding brand perception and customer requirements based on, for example, data gathered from millions of online posts such as social media, forums, and blogs. For example, when promoting a new policy/product, a company may monitor electronically posted customer comments regarding a particular policy/product so that the company can respond properly and address criticisms and issues in a timely manner. Hence, online monitoring of current sentiment trend and topics related to, for example, a preset product and brand name is important for modern marketing…
[0011] The aforementioned aspects and other objectives and advantages can now be achieved as described herein. Methods, systems and processor-readable media for simultaneous sentiment analysis and topic classification with multiple labels are disclosed herein. A sentiment and topic associated with a post can be classified at similar time and a result can be incorporated to predict a feature so that a label of two tasks can promote and reinforce each other iteratively. A feature extraction and selection can be performed on both tasks of sentiment and topic classification. A multi-task multi-label classification model can be trained for each task with maximum entropy utilizing multiple labels to ascertain data indicative of and/or derived from an extra label and to manage with class ambiguities. Each task has a separate classification model with different predicting features and they can be trained collectively which allows flexibility in model construction. Such multi-task multi-label (MTML) classification model produces a probabilistic result and the classes can be ranked by the probabilistic result and the post can be classified with the multi-label.
As discussed above, Jezewski uses machine learning and semantic analysis to glean user sentiment. The Examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of Applicant’s invention to modify Jezewski to perform the steps of:
routing collected text data in parallel with the generated machine-encoded feature vectors;
supplying outputs of intent identification and scoring operations to a feedback mechanism
in order to minimize delay in the sentiment analysis (as suggested in Biessmann: col. 12: 13-51) and so that the various dimensions of gathered information may be used to reinforce each other (as suggested in ¶ 11 of Huang) and to improve the accuracy of the models (as suggested in Biessmann: col 3: 5-24; col. 5: 41-58; col. 9: 26 – col. 10: 3; col. 14: 7-14).
Jezewski, Biessmann, and Huang do not explicitly disclose:
modifying stored model parameters based on divergence between outputs; and
computing a weighted balance between scoring outputs and intent identification outputs using a machine-generated weighting function derived from model confidence metrics.
Givental uses hybrid machine learning to detect anomalies using feedback from the outputs of various learning models of an ensemble to assign weights to the various models based on whether or not each respective model has outputted a correct result (Givental: ¶¶ 52, 68). As discussed above, Jezewski evaluates outputs related to intent identification logic and information extraction. The Examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of Applicant’s invention to modify the Jezewski-Biessmann-Huang combination to perform the steps of:
modifying stored model parameters based on divergence between outputs; and
computing a weighted balance between scoring outputs and intent identification outputs using a machine-generated weighting function derived from model confidence metrics
in order to improve the overall accuracy of the ensemble of learning models (as suggested in ¶ 24 of Givental).
[Claim 10] Jezewski discloses wherein intent is identified by aggregating general idea or action toward an object (¶¶ 13, 116 – The nature of the opinions, which is related to sentiment analysis, can include an assessment of actions, such as a desire to switch from one company to another or an action to be taken in regard to an entity; ¶¶ 54, 57 – Complaint and opinion information may be aggregated).
[Claim 11] Jezewski discloses assigning at least one label to each data of the collected text data using a trained classification logic (¶¶ 44-48).
[Claim 12] Jezewski discloses scoring each labelled data based on training and assign intent based on the at least one assigned label using a score logic (¶¶ 43-45; ¶¶ 13, 116 – The nature of the opinions, which is related to sentiment analysis, can include an assessment of actions, such as a desire to switch from one company to another or an action to be taken in regard to an entity; Jezewski provides relevant details of the disclosed classification and label processing in ¶¶ 38-47. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40). The fact that certain news in one domain may be good news and that the same news may be bad news in another domain is an example of scoring (e.g., as “good” or “bad”) based on intent.).
[Claim 14] Jezewski discloses wherein extracting information is performed by an information extraction logic (¶¶ 15-20).
[Claim 15] While Jezewski strongly suggests that the disclosed operations may be performed substantially simultaneously (i.e., in parallel) since the various models may be applied in conjunction with one another, including to classify and extract data (as seen in ¶¶ 36-46 of Jezewski), Jezewski does not explicitly perform the step of applying the collected text data in parallel to both the classification logic and the information extraction logic. Biessmann verifies item attributes in an artificial intelligence environment and explains how the extraction and classification may be performed in parallel, as seen in the following excerpt:
The classifier training module 150, the feature extractor module 152, and/or the classifier module 154 can operate in parallel and for multiple users at the same time. For example, the verification of attributes listed in an item description may be requested from unique user devices 102 for different listings and the components of the attribute verification system 104 can verify the listed attributes simultaneously or nearly simultaneously for the unique user devices 102 in “real-time,” where “real-time” may be based on the perspective of the user. The generation and verification of attributes may be considered to occur in real time if, for example, the delay is sufficiently short (e.g., less than a few seconds) such that the user typically would not notice a processing delay. Real-time may also be based on the following: the attribute verification system 104 can verify listed attributes for a first user at the same time or at nearly the same time (e.g., within a couple seconds) as a verification of listed attributes is performed for a second user; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously as instantaneously as possible, limited by processing resources, available memory, network bandwidth conditions, and/or the like; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously based on a time it takes the hardware components of the attribute verification system 104 to process data; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously immediately as data is received (instead of storing, buffering, caching, or persisting data as it is received and processing the data later on); the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously by transforming data without intentional delay, given the processing limitations of the attribute verification system 104 and other systems, like the user devices 102, and the time required to accurately receive and/or transmit the data; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously by processing or transforming data fast enough to keep up with an input data stream; etc. (Biessmann: col. 12: 13-51)
While Biessmann does not explicitly link the classification and extraction to an environment in which an audience sentiment is evaluated, Huang sheds some additional light on the benefits of performing operations related to extraction (like sentiment analysis) simultaneously with classification in the area of user sentiment analysis, as described in the following excerpts of Huang:
[0002] Sentiment and topic analysis have a wide application in business marketing and customer care applications to assist in evaluating and understanding brand perception and customer requirements based on, for example, data gathered from millions of online posts such as social media, forums, and blogs. For example, when promoting a new policy/product, a company may monitor electronically posted customer comments regarding a particular policy/product so that the company can respond properly and address criticisms and issues in a timely manner. Hence, online monitoring of current sentiment trend and topics related to, for example, a preset product and brand name is important for modern marketing…
[0011] The aforementioned aspects and other objectives and advantages can now be achieved as described herein. Methods, systems and processor-readable media for simultaneous sentiment analysis and topic classification with multiple labels are disclosed herein. A sentiment and topic associated with a post can be classified at similar time and a result can be incorporated to predict a feature so that a label of two tasks can promote and reinforce each other iteratively. A feature extraction and selection can be performed on both tasks of sentiment and topic classification. A multi-task multi-label classification model can be trained for each task with maximum entropy utilizing multiple labels to ascertain data indicative of and/or derived from an extra label and to manage with class ambiguities. Each task has a separate classification model with different predicting features and they can be trained collectively which allows flexibility in model construction. Such multi-task multi-label (MTML) classification model produces a probabilistic result and the classes can be ranked by the probabilistic result and the post can be classified with the multi-label.
As discussed above, Jezewski uses machine learning and semantic analysis to glean user sentiment. The Examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of Applicant’s invention to modify Jezewski to perform the step of applying the collected text data in parallel to both the classification logic and the information extraction logic in order to minimize delay in the sentiment analysis (as suggested in Biessmann: col. 12: 13-51) and so that the various dimensions of gathered information may be used to reinforce each other (as suggested in ¶ 11 of Huang).
[Claim 16] Jezewski discloses sending the extracted information with clearly identified intent to the feedback (¶¶ 36-46 – The various models may be applied in conjunction with one another, including to classify and extract data; ¶¶ 43-63, 111-112, 120 – A probabilistic classifier model may be used for sentiment analysis. Weights may be applied as part of training models in regard to interim scores, opinions, complaints, predictions, labeling features, identifying correlations among the various data and scores, etc. Feedback may refer to the feedback from customers and/or to feedback used to train the models; Jezewski provides relevant details of the disclosed classification and label processing in ¶¶ 38-47. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40). The fact that certain news in one domain may be good news and that the same news may be bad news in another domain is an example of scoring (e.g., as “good” or “bad”) based on intent.); and
sending the extracted information without clearly identified intent is sent to the classification logic (¶¶ 36-46 – The various models may be applied in conjunction with one another, including to classify and extract data. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40)).
[Claim 23] Jezewski, Biessmann, and Huang do not explicitly disclose wherein modifying stored model parameters includes iteratively adjusting model weights based on measured divergence between scoring outputs and intent identification outputs.
Givental uses hybrid machine learning to detect anomalies using feedback from the outputs of various learning models of an ensemble to assign weights to the various models based on whether or not each respective model has outputted a correct result (Givental: ¶¶ 52, 68). Givental repeats its operations until all input data has been processed, including for the modification of the ensemble of machine learning models and to generate a fully labeled dataset (Givental: ¶¶ 75, 78). As discussed above, Jezewski evaluates outputs related to intent identification logic and information extraction. The Examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of Applicant’s invention to modify the Jezewski-Biessmann-Huang combination wherein modifying stored model parameters includes iteratively adjusting model weights based on measured divergence between scoring outputs and intent identification outputs in order to improve the overall accuracy of the ensemble of learning models (as suggested in ¶ 24 of Givental).
[Claim 17] A non-transitory computer-readable storage medium storing instructions which, when executed by a hardware controller operating a text analysis applications cause the hardware controller (¶¶ 11-12, 88; ¶ 120 – AI sentiment analysis platform) to perform operations comprising:
collecting text data associated with business target features (¶¶ 15-20 – Indications of user sentiment toward companies may be gathered; ¶ 12 – “As shown in FIG. 1A, example implementation 100 may include a client device, comment sources, and a sentiment analysis platform. Assume that a user utilizes the client device to access one or more applications provided by the comment sources. In some implementations, the comment sources may include sources that provide social media applications, blog applications, chat room applications, message board applications, ratings system applications, and/or the like. As further shown in FIG. 1A, the user may utilize the client device to provide complaint information, about an entity (e.g., company A), to the one or more applications provided by the comment sources. For example, the complaint information may include a complaint indicating that the user is switching from company A to company B, a complaint indicating a negative statement about company A, a complaint indicating invalidation of a product or service of company A, and/or the like.”; ¶ 120 – “Some implementations described herein may provide a sentiment analysis platform that utilizes artificial intelligence to make a prediction about an entity based on user sentiment and transaction history. For example, the sentiment analysis platform may consider sentiments of users who have opinions, complaints, and predictions about the entity, and transactions conducted by users with the entities in order to predict a future stock price of the entity. The sentiment analysis platform may receive the opinions, the complaints, and the predictions of the users, about the entity, from social media sources, and may receive transaction information associated with the users and the entity from financial institutions. The sentiment analysis platform may determine correlations between the transaction information and the opinions, the complaints, and the predictions, in order to apply weights to the opinions, the complaints, and the predictions. The sentiment analysis platform may generate a prediction about the future stock price of the entity based on the opinions, the complaints, the predictions, the transaction information, and the correlations between the transaction information and the opinions, the complaints, and the predictions.” Comments made on/via social media and/or in association with a social media source are examples of social media posts.);
extracting information from the text data by generating machine-encoded feature vectors representing syntactic roles, metadata, actions, entities, and entity relationships (¶¶ 43-45; ¶¶ 13, 116 – The nature of the opinions, which is related to sentiment analysis, can include an assessment of actions, such as a desire to switch from one company to another or an action to be taken in regard to an entity; Jezewski implements its sentiment analysis using machine learning and/or natural language processing (¶ 36), which means that text data and related scoring data will be classified (i.e., labeled) and trained; Jezewski provides relevant details of the disclosed classification and label processing in ¶¶ 38-47. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) Identifying noun phrases and verb phrases is an example of identifying syntactic roles.; The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40). The fact that certain news in one domain may be good news and that the same news may be bad news in another domain is an example of scoring (e.g., as “good” or “bad”) based on intent.; ¶¶ 36, 39, 92, 113-118 – Information related to an intent is extracted and identified.; ¶¶ 38-40 – Text, i.e., metadata, is classified to identify entities and sentiment based on opinion information and in light of the different domains and respective set of features for different domains.; ¶¶ 15-20 – Indications of user sentiment toward companies may be gathered. This demonstrates entity relationships.; ¶ 46 – “In some implementations, the sentiment analysis platform may utilize a maximum entropy classifier model. A maximum entropy classifier model may convert labeled feature sets to vectors using encoding. The encoded vector may then be used to calculate weights for each feature, which may then be combined to determine a most likely label for a feature set.”);
generating machine-encoded feature vectors representing syntactic roles, metadata, actions, entities, and entity relationships (¶¶ 43-45; ¶¶ 13, 116 – The nature of the opinions, which is related to sentiment analysis, can include an assessment of actions, such as a desire to switch from one company to another or an action to be taken in regard to an entity; Jezewski implements its sentiment analysis using machine learning and/or natural language processing (¶ 36), which means that text data and related scoring data will be classified (i.e., labeled) and trained; Jezewski provides relevant details of the disclosed classification and label processing in ¶¶ 38-47. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) Identifying noun phrases and verb phrases is an example of identifying syntactic roles.; The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40). The fact that certain news in one domain may be good news and that the same news may be bad news in another domain is an example of scoring (e.g., as “good” or “bad”) based on intent.; ¶¶ 36, 39, 92, 113-118 – Information related to an intent is extracted and identified.; ¶¶ 38-40 – Text, i.e., metadata, is classified to identify entities and sentiment based on opinion information and in light of the different domains and respective set of features for different domains.; ¶¶ 15-20 – Indications of user sentiment toward companies may be gathered. This demonstrates entity relationships.; ¶ 46 – “In some implementations, the sentiment analysis platform may utilize a maximum entropy classifier model. A maximum entropy classifier model may convert labeled feature sets to vectors using encoding. The encoded vector may then be used to calculate weights for each feature, which may then be combined to determine a most likely label for a feature set.”);
identifying intent actions using the generated feature vectors (¶¶ 36, 39, 92, 113-118 – Information related to an intent is extracted and identified.; ¶ 120 – AI sentiment analysis platform.; ¶ 46 – “In some implementations, the sentiment analysis platform may utilize a maximum entropy classifier model. A maximum entropy classifier model may convert labeled feature sets to vectors using encoding. The encoded vector may then be used to calculate weights for each feature, which may then be combined to determine a most likely label for a feature set.”);
assigning labels using a trained classification model stored in non-transitory memory (¶ 36 – “In some implementations, the artificial intelligence techniques may include a sentiment analysis model that utilizes multiple artificial analysis techniques. In some implementations, the sentiment analysis model may include a model that uses natural language processing, text analysis, and machine learning to systematically identify, extract, quantify, and study affective states and subjective information.”; ¶¶ 59-60 – “[0059] As shown in FIG. 1F, and by reference number 155, the sentiment analysis platform may utilize the transaction information to determine correlations with the interim scores (e.g., the necessity score, the abstract score, the ethics score, the industry score, the demands for service score, the supply for service score, the vendor score, the innovation score, the adaptability score, the execution score, and/or the like). In some implementations, the sentiment analysis platform may utilize a correlation clustering method, a Pearson's product-moment coefficient method, an Anscombe's quartet method, a Spearman's rank-correlation coefficient method, and/or the like in order to determine the correlations between the transaction information and the interim scores. [0060] A clustering method may include partitioning data points into groups based on their similarity, and the correlation clustering method may include clustering a set of objects into an optimum number of clusters without specifying that number in advance. Given a collection of paired (x, y) variables, the Pearson's product-moment coefficient method produces a value, between −1 and +1, that quantifies a strength of dependence between the variables x and y. A value of +1 means that all of the (x, y) points lie exactly on a line with positive slope, a value of −1 means that all of the points lie exactly on a line with negative slope, and a value of 0 means that there is no relationship between the two variables. The Anscombe's quartet method utilizes four datasets that have nearly identical simple descriptive statistics, yet appear very different when graphed. The Spearman's rank-correlation coefficient method provides a nonparametric measure of rank correlation (e.g., a statistical dependence between a ranking of two variables), and assesses how well a relationship between two variables can be described using a monotonic function.“; ¶¶ 43-45; ¶¶ 13, 116 – The nature of the opinions, which is related to sentiment analysis, can include an assessment of actions, such as a desire to switch from one company to another or an action to be taken in regard to an entity; Jezewski implements its sentiment analysis using machine learning and/or natural language processing (¶ 36), which means that text data and related scoring data will be classified (i.e., labeled) and trained; Jezewski provides relevant details of the disclosed classification and label processing in ¶¶ 38-47. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40). The fact that certain news in one domain may be good news and that the same news may be bad news in another domain is an example of scoring (e.g., as “good” or “bad”) based on intent.; ¶ 88 – “Device 300 may perform one or more processes described herein. Device 300 may perform these processes based on processor 320 executing software instructions stored by a non-transitory computer-readable medium, such as memory 330 and/or storage component 340. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.”);
computing intent scores using a trained scoring model stored in non-transitory memory (¶ 36 – “In some implementations, the artificial intelligence techniques may include a sentiment analysis model that utilizes multiple artificial analysis techniques. In some implementations, the sentiment analysis model may include a model that uses natural language processing, text analysis, and machine learning to systematically identify, extract, quantify, and study affective states and subjective information.”; ¶¶ 59-60 – “[0059] As shown in FIG. 1F, and by reference number 155, the sentiment analysis platform may utilize the transaction information to determine correlations with the interim scores (e.g., the necessity score, the abstract score, the ethics score, the industry score, the demands for service score, the supply for service score, the vendor score, the innovation score, the adaptability score, the execution score, and/or the like). In some implementations, the sentiment analysis platform may utilize a correlation clustering method, a Pearson's product-moment coefficient method, an Anscombe's quartet method, a Spearman's rank-correlation coefficient method, and/or the like in order to determine the correlations between the transaction information and the interim scores. [0060] A clustering method may include partitioning data points into groups based on their similarity, and the correlation clustering method may include clustering a set of objects into an optimum number of clusters without specifying that number in advance. Given a collection of paired (x, y) variables, the Pearson's product-moment coefficient method produces a value, between −1 and +1, that quantifies a strength of dependence between the variables x and y. A value of +1 means that all of the (x, y) points lie exactly on a line with positive slope, a value of −1 means that all of the points lie exactly on a line with negative slope, and a value of 0 means that there is no relationship between the two variables. The Anscombe's quartet method utilizes four datasets that have nearly identical simple descriptive statistics, yet appear very different when graphed. The Spearman's rank-correlation coefficient method provides a nonparametric measure of rank correlation (e.g., a statistical dependence between a ranking of two variables), and assesses how well a relationship between two variables can be described using a monotonic function.“; ¶¶ 43-45; ¶¶ 13, 116 – The nature of the opinions, which is related to sentiment analysis, can include an assessment of actions, such as a desire to switch from one company to another or an action to be taken in regard to an entity; Jezewski implements its sentiment analysis using machine learning and/or natural language processing (¶ 36), which means that text data and related scoring data will be classified (i.e., labeled) and trained; Jezewski provides relevant details of the disclosed classification and label processing in ¶¶ 38-47. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40). The fact that certain news in one domain may be good news and that the same news may be bad news in another domain is an example of scoring (e.g., as “good” or “bad”) based on intent.; ¶ 88 – “Device 300 may perform one or more processes described herein. Device 300 may perform these processes based on processor 320 executing software instructions stored by a non-transitory computer-readable medium, such as memory 330 and/or storage component 340. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.”).
While Jezewski strongly suggests that the disclosed operations may be performed substantially simultaneously (i.e., in parallel) since the various models may be applied in conjunction with one another, including to classify and extract data (as seen in ¶¶ 36-46 of Jezewski), Jezewski does not explicitly disclose:
routing collected text data in parallel with generated feature vectors;
supplying outputs to a feedback mechanism.
Biessmann verifies item attributes in an artificial intelligence environment and explains how the extraction and classification may be performed in parallel, as seen in the following excerpt:
The classifier training module 150, the feature extractor module 152, and/or the classifier module 154 can operate in parallel and for multiple users at the same time. For example, the verification of attributes listed in an item description may be requested from unique user devices 102 for different listings and the components of the attribute verification system 104 can verify the listed attributes simultaneously or nearly simultaneously for the unique user devices 102 in “real-time,” where “real-time” may be based on the perspective of the user. The generation and verification of attributes may be considered to occur in real time if, for example, the delay is sufficiently short (e.g., less than a few seconds) such that the user typically would not notice a processing delay. Real-time may also be based on the following: the attribute verification system 104 can verify listed attributes for a first user at the same time or at nearly the same time (e.g., within a couple seconds) as a verification of listed attributes is performed for a second user; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously as instantaneously as possible, limited by processing resources, available memory, network bandwidth conditions, and/or the like; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously based on a time it takes the hardware components of the attribute verification system 104 to process data; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously immediately as data is received (instead of storing, buffering, caching, or persisting data as it is received and processing the data later on); the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously by transforming data without intentional delay, given the processing limitations of the attribute verification system 104 and other systems, like the user devices 102, and the time required to accurately receive and/or transmit the data; the attribute verification system 104 can verify listed attributes simultaneously or nearly simultaneously by processing or transforming data fast enough to keep up with an input data stream; etc. (Biessmann: col. 12: 13-51)
Furthermore, Biessmann uses feedback to update the training, thereby confirming which suggested attributes are correct and incorrect (Biessmann: col 3: 5-24; col. 5: 41-58; col. 9: 26 – col. 10: 3; col. 14: 7-14).
While Biessmann does not explicitly link the classification and extraction to an environment in which an audience sentiment is evaluated, Huang sheds some additional light on the benefits of performing operations related to extraction (like sentiment analysis) simultaneously with classification in the area of user sentiment analysis, as described in the following excerpts of Huang:
[0002] Sentiment and topic analysis have a wide application in business marketing and customer care applications to assist in evaluating and understanding brand perception and customer requirements based on, for example, data gathered from millions of online posts such as social media, forums, and blogs. For example, when promoting a new policy/product, a company may monitor electronically posted customer comments regarding a particular policy/product so that the company can respond properly and address criticisms and issues in a timely manner. Hence, online monitoring of current sentiment trend and topics related to, for example, a preset product and brand name is important for modern marketing…
[0011] The aforementioned aspects and other objectives and advantages can now be achieved as described herein. Methods, systems and processor-readable media for simultaneous sentiment analysis and topic classification with multiple labels are disclosed herein. A sentiment and topic associated with a post can be classified at similar time and a result can be incorporated to predict a feature so that a label of two tasks can promote and reinforce each other iteratively. A feature extraction and selection can be performed on both tasks of sentiment and topic classification. A multi-task multi-label classification model can be trained for each task with maximum entropy utilizing multiple labels to ascertain data indicative of and/or derived from an extra label and to manage with class ambiguities. Each task has a separate classification model with different predicting features and they can be trained collectively which allows flexibility in model construction. Such multi-task multi-label (MTML) classification model produces a probabilistic result and the classes can be ranked by the probabilistic result and the post can be classified with the multi-label.
As discussed above, Jezewski uses machine learning and semantic analysis to glean user sentiment. The Examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of Applicant’s invention to modify Jezewski to perform the steps of:
routing collected text data in parallel with generated feature vectors;
supplying outputs to a feedback mechanism
in order to minimize delay in the sentiment analysis (as suggested in Biessmann: col. 12: 13-51) and so that the various dimensions of gathered information may be used to reinforce each other (as suggested in ¶ 11 of Huang) and to improve the accuracy of the models (as suggested in Biessmann: col 3: 5-24; col. 5: 41-58; col. 9: 26 – col. 10: 3; col. 14: 7-14).
Jezewski, Biessmann, and Huang do not explicitly disclose:
modifying stored model parameters based on divergence between outputs; and
computing a weighted balance between intent identification outputs and scoring outputs using a machine-generated weighting function derived from model confidence metrics.
Givental uses hybrid machine learning to detect anomalies using feedback from the outputs of various learning models of an ensemble to assign weights to the various models based on whether or not each respective model has outputted a correct result (Givental: ¶¶ 52, 68). As discussed above, Jezewski evaluates outputs related to intent identification logic and information extraction. The Examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of Applicant’s invention to modify the Jezewski-Biessmann-Huang combination to perform the steps of:
modifying stored model parameters based on divergence between outputs; and
computing a weighted balance between intent identification outputs and scoring outputs using a machine-generated weighting function derived from model confidence metrics
in order to improve the overall accuracy of the ensemble of learning models (as suggested in ¶ 24 of Givental).
[Claim 18] Jezewski discloses executable instructions that cause the computer to assign at least one label to each data of the collected text data (¶¶ 44-48).
[Claim 19] Jezewski discloses executable instructions that cause the computer to score each labelled data based on training and assign intent based on the at least one assigned label (¶¶ 43-45; ¶¶ 13, 116 – The nature of the opinions, which is related to sentiment analysis, can include an assessment of actions, such as a desire to switch from one company to another or an action to be taken in regard to an entity; Jezewski provides relevant details of the disclosed classification and label processing in ¶¶ 38-47. For example, Jezewski explains, “In some implementations, the sentiment analysis model may utilize natural language processing methods with the historical information, the complaint information, the opinion information, and the prediction information in order to make the historical information, the complaint information, the opinion information, and the prediction information analyzable. For example, the sentiment analysis model may use natural language processing to derive meaning from natural language input stemming from the historical information, the complaint information, the opinion information, and the prediction information. For example, the sentiment analysis model may utilize deep parsing to break sentences down into noun phrases and verb phrases and then determine associated prepositional phrases. In this way, the sentiment analysis model may determine how entities relate to each other and navigate through unstructured text.” (Jezewski: ¶ 37) The classifier may provide additional context for the sentiment analysis, as explained in the following excerpt: “In some implementations, the classification method may categorize the historical information, the complaint information, the opinion information, and the prediction information into different domains (e.g., markets, economy, industry, technology, and/or the like). The classification method may be used since there may be a different set of features for different domains and thus, each domain may have a different classifier. For example, a news article in the technology domain may be positive news for company A but may be negative news for company B, great news about company A may be slightly bad news for company B, who is a competitor (e.g., an vice versa), and/or the like. Thus, if the sentiment analysis model knows competitor information associated with competitors for each entity, then when the sentiment analysis model identifies information that is good (or bad) for an entity, the sentiment analysis model may determine that the information is bad (or good) for the competitors of the entity.” (Jezewski: ¶ 40). The fact that certain news in one domain may be good news and that the same news may be bad news in another domain is an example of scoring (e.g., as “good” or “bad”) based on intent.).
Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over Jezewski (US 2019/0251626) in view of Biessmann et al. (US 9,892,133) in view of Huang et al. (US 2014/0250032) in view of Givental et al. (US 2021/0281592), as applied to claim 17 above, in view of Chelba et al. (US 2006/0074630).
[Claim 24] Jezewski does not explicitly disclose wherein the stored instructions cause the hardware controller to perform weighting operations without permitting manual assignment of weights by a user. Like Jezewski, Chelba performs natural language processing with feature selection and a statistical classifier (Chelba: ¶ 62). Additionally, Chelba constructs a statistical classifier comprising parameterized conditional likelihood over training data (Chelba: claims 2, 11, 12) and model parameters are estimated using coefficients and weights and model parameters are iteratively re-estimated (Chelba: ¶¶ 43-47). Chelba constructs a statistical classifier or model to be used later and stores the classifier construction module (Chelba: ¶¶ 31-32). Chelba states, “A statistical classification approach (e.g. n-gram, Naive Bayes, and maximum entropy) to the problem is common and the most successful one used so far since it deals gracefully with the irregularities of natural language and can usually be estimated in a data-driven fashion, without expensive human authoring.” (Chelba: ¶ 5) The Examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of Applicant’s invention to modify the Jezewski-Biessmann-Huang-Givental-Edwards combination wherein the stored instructions cause the hardware controller to perform weighting operations without permitting manual assignment of weights by a user because “[o]ne advantage of the growth transform re-estimation procedure is that the present model parameters are re-normalized or tuned at each iteration, thus maintaining parameters in the parameter space X.” (Chelba: ¶ 45) Additionally, preventing manual assignment of weights by a user would have reduced expenses otherwise associated with human authoring (as suggested in ¶ 5 of Chelba).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SUSANNA M DIAZ whose telephone number is (571)272-6733. The examiner can normally be reached M-F, 8 am-4:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian Epstein can be reached at (571) 270-5389. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SUSANNA M. DIAZ/
Primary Examiner
Art Unit 3625A
Prosecution Insights