Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 8 is objected to because of the following informalities:
Claim 8 states “further comprising extracting actional items”. This should be written as “further comprising extracting actionable items”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second
paragraph, as being indefinite for failing to particularly po int out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 4, the phrase " overall mismatch first performance review " is indefinite because it is unclear what constitutes the claimed “overall mismatch first performance review.” The specification does not describe an “overall mismatch first performance review” and the phrase “overall mismatch first performance review” does not have an established meaning in the art, thereby leaving the scope of the claim uncertain.
Therefore, claim 4 is unclear and is rejected.
Regarding claim 20, claim 20 incorporates all limitations thereof of claim 4.
Therefore, claim 20 is rejected for at least the same reasons as described above with reference to claim 4.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 9-16 are rejected under 35 U.S.C. 101 because the claimed invention is directed to nonstatutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because they recite a computer readable medium comprising having instructions stored thereon. The applicant's specification provides a definition for computer-readable medium which in its plain meaning includes data signals per se as one potential form of the medium. Data signals per se do not faII into one of the four statutory categories of invention. As such, they are non-statutory subject matter. In contrast, a claimed non-statutory computer-readable storage medium excludes data signals from its scope, and does fall into one of the four statutory categories of invention.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim [ 1, 2, 5, 6, 9, 10, 13, 14, 16, 17, 18 ] are rejected under 35 U.S.C. 103 as being unpatentable over Kottha (US 2017/0193397 A1, hereinafter Kottha) in view of Kanagovi (US11675823B2, hereinafter Kanagovi) and in further view of Yan (Yan, Hang, et al. "A unified generative framework for aspect-based sentiment analysis." Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: Long papers). 2021, hereinafter Yan).
Regarding claim 1, Kottha teaches
A method of evaluating performance, the method comprising: receiving a plurality of training performance reviews;
[0016 "..an example system 100 that performs natural language processing of unstructured text. Briefly, and as described in further detail below, the system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment. The system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment" where employee feedback provided to an employer regarding work-related experiences constitutes performance review"]; [0026” The trend analyzer 145 provides the review data 105 and 110 along with the identified topics, sentiments, and trends to the internal data analyzer 155. The internal data analyzer 155 analyzes internal data 160 that may only be accessible to an entity that is being reviewed. The internal data 160 may include demographic and employment data.
For example, the internal data 160 may be human resources data that include for each employee, gender information, race information, employment dates, performance review
information, income information, age, and tenure”];
[0017 " The review data 105 and 110 may each be associated with a particular employee with the system relating each of the review data 105 and 110 with an employee identifier and may not be easily parsable" where "each" of the "review data 105 and 110" implies there are at least two distinct performance records];
[0023 "The system 100 may train the machine learning system 135 using the training
data 140. The training data 140 may include various entries of review data and assigned topics/aspects and sentiments" ].
extracting from the training performance reviews, using a first machine learning model, a plurality of features comprising a training aspect, a training sentiment,
[0016 "..an example system 100 that performs natural language processing of unstructured text. Briefly, and as described in further detail below, the system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment. The system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment" where employee feedback provided to an employer regarding work-related experiences constitutes performance feedback"];
[0032 “The system 205 provides the data from the data sources to the machine learning/AI/analytics engine 210. The machine learning/AI/analytics engine 210 analyzes thedata from the data sources 205 using machine learning and natural language processing to identify topics/aspects and corresponding sentiments. The machine learning/AI/analytics
engine 210 may also identify trends and correlate the topics/aspects and sentiments to groups of employees and generate user interfaces based on the topics, sentiments, and
groups”].
[0033 “The machine learning/AI/analytics engine 210 includes a natural language information extractor 220 that processes the data received from the data sources 205 based
on natural processing techniques. The natural language information extractor 220 includes a data extraction engine 230. The data extraction engine 230 is configured to process
unstructured data such as text reviews that employees provide to review websites or directly to their employers"];
[0035 “The data analysis engine 225 may extract sentiments from the data entries using recursive neural tensor networks or other deep learning algorithms.”].
[0023 "The system 100 may train the machine learning system 135 using the training
data 140. The training data 140 may include various entries of review data and assigned topics/aspects and sentiments" ]
receiving a first performance review;
[0016 "The system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment" where employee feedback provided to an employer regarding work-related experiences constitutes performance review"].
extracting from the first performance review one or more first aspects,
[0016 "..an example system 100 that performs natural language processing of unstructured text. Briefly, and as described in further detail below, the system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment. The system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment" where employee feedback provided to an employer regarding work-related experiences constitutes performance feedback"];
[0032 “The system 205 provides the data from the data sources to the machine learning/AI/analytics engine 210. The machine learning/AI/analytics engine 210 analyzes the data from the data sources 205 using machine learning and natural language processing to identify topics/aspects and
corresponding sentiments. The machine learning/AI/analytics engine 210 may also identify trends and correlate the topics/aspects and sentiments to groups of employees and generate user interfaces based on the topics, sentiments, and groups”]
[0033 “The machine learning/AI/analytics engine 210 includes a natural language information extractor 220 that processes the data received from the data sources 205 based
on natural processing techniques. The natural language information extractor 220 includes a data extraction engine 230. The data extraction engine 230 is configured to process
unstructured data such as text reviews that employees provide to review websites or directly to their employers"];
[0035 “The data analysis engine 225 may extract sentiments from the data entries using recursive neural tensor networks or other deep learning algorithms.”].
However, Kottha does not expressly recite
using the extracted plurality of features to train a second machine learning model;
and using the trained second machine learning model, predicting first sentiment scores for each of the first aspects.
But Kanagovi teaches
using the extracted plurality of features to train a second machine learning model;
[Column 18, lines 6-17 "The output of the context retainer block 830 is provided to a downstream sentiment classifier, shown in FIG. 8C as a feed forward neural network block 835. The feed forward neural network block 835 performs sentiment prediction (e.g., negative, neutral, positive) at the aspect term level. The sentiments are advantageously trained for each aspect term at a time (e.g., by activating and averaging out the encodings of the tokens for one aspect term only). For example, an article or other document having four aspect terms will separately train the BERT Sentiments model 815 for each of the four aspect terms, one at a time" where the sentiment classifier is the second model]; [Column 16, lines 3-8 "In some embodiments, both tasks performed by the aspect term sentiment analysis logic 116—aspect term extract and prediction of sentiments for extracted aspect terms, are performed using BERT models that are trained in parallel by averaging out the BERT encodings of the aspect terms from both of the models"].
using the trained second machine learning model, predicting first sentiment scores for each of the first aspects.
[Column 18, lines "6-10 The output of the context retainer block 830 is provided
to a downstream sentiment classifier, shown in FIG. SC as a feed forward neural network block 835. The feed forward neural network block 835 performs sentiment prediction ( e.g., negative, neutral, positive) at the aspect term level." where sentiment prediction of negative, neutral or positive correspond to sentiment scores].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Kottha into the teachings of Kanagovi because Kottha teaches extracting aspects and sentiments from performance reviews while Kanagovi teaches using extracted aspect terms to train a downstream sentiment prediction model. Both of these references process textual review documents to extract aspect-level sentiment information using machine learning models. Combining them would enable extracted review features to train a second sentiment model, thereby improving granularity and accuracy of sentiment analysis.
Kottha in view of Kanagovi do not teach
a corresponding training evidence;
one or more corresponding first evidences,
However, Yan teaches
a corresponding training evidence;
[Introduction, lines 1-4 "Aspect-based Sentiment Analysis (ABSA) is the fine-grained Sentiment Analysis (SA) task, which aims to identify the aspect term (a), its corresponding sentiment polarity (s), and the opinion term (o)" wherein, in the context of the employee performance reviews described by Kottha, the opinion terms extracted by the machine learning model (BART Page 3) Yan correspond to the claimed training evidence associated with the extracted aspects and sentiments]. ].
one or more corresponding first evidences,
[Introduction, lines 1-4 "Aspect-based Sentiment Analysis (ABSA) is the fine-grained Sentiment Analysis (SA) task, which aims to identify the aspect term (a), its corresponding sentiment polarity (s), and the opinion term (o)" wherein, in the context of the employee performance reviews described by Kottha, the opinion terms extracted by the machine learning model of Yan(BART Page 3) correspond to the claimed first evidence associated with the extracted aspects and sentiments].].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Kottha in view Kanagovi with the teachings of Yan because combining Yan’s teaching of extracting opinion terms associated with aspect and sentiment labels, wherein the opinion terms provide textual support for the sentiment determination, would thereby provide a more granular, accurate, and explainable evolution of employee performance.
Regarding claim 2 the rejection of claim 1 is incorporated. Kottha, Kanagovi, and Yan teach all limitations of the current invention as stated above.
Kottha further teaches
comprising extracting from the first performance review
[0016 "..an example system 100 that performs natural language processing of unstructured text. Briefly, and as described in further detail below, the system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment. The system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment" where employee feedback provided to an employer regarding work-related experiences constitutes performance feedback"];
[0032] The system 205 provides the data from the data sources to the machine learning/AI/analytics engine 210. The machine learning/AI/analytics engine 210 analyzes the
data from the data sources 205 using machine learning and natural language processing to identify topics/aspects and corresponding sentiments. The machine learning/AI/analytics
engine 210 may also identify trends and correlate the topics/aspects and sentiments to groups of employees and generate user interfaces based on the topics, sentiments, and
groups”].
[0033 “The machine learning/AI/analytics engine 210 includes a natural language information extractor 220 that processes the data received from the data sources 205 based
on natural processing techniques. The natural language information extractor 220 includes a data extraction engine 230. The data extraction engine 230 is configured to process
unstructured data such as text reviews that employees provide to review websites or directly to their employers"];
[0035 “The data analysis engine 225 may extract sentiments from the data entries using recursive neural tensor networks or other deep learning algorithms.”].
[0023 "The system 100 may train the machine learning system 135 using the training
data 140. The training data 140 may include various entries of review data and assigned topics/aspects and sentiments" ].
But Kottha does not expressly recite
However, Kanagovi recites
or more first aspect weights.
[Column 16, lines 11-19 " To ensure that the right attributing words are accurately identified in a given sentence or other portion of a document that are acting upon an aspect term to accurately predict sentiments, the adjusted weight variant of the self-attention model gives a higher weightage to words which are closer to an aspect term and lower weightage to words that are farther from the aspect term" wherein, in the context of the employee performance reviews described by Kotta, the dynamic weighted approach taught by Kanagovi generates different weights for respective extracted aspects, such as first aspect, a second aspect, etc and these aspect specific weights correspond to the one or more first aspects].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Kottha in view Kanagovi because applying the weighting techniques of Kanagovi to the performance reviews in order extract aspect weights indicative of the relative importance of the identified aspects, would thereby provide a more granular, and accurate, and explainable evaluation of the employee performance.
Regarding claim 5, the method of claim 1 is incorporated. Kottha, Kanagovi and Yan teach all the limitations of the current invention as stated above.
Kottha further teaches
wherein the first machine learning model comprises an large language learning model,.
[0033 “The machine learning/AI/analytics engine 210 includes a natural language information extractor 220 that processes the data received from the data sources 205 based
on natural processing techniques”];
But Kottha does not expressly recite the second machine learning model comprises a deep learning model
However, Kanagovi teaches
and the second machine learning model comprises a deep learning model
[Column 18, lines 6-11 " The output of the context retainer block 830 is provided
to a downstream sentiment classifier, shown in FIG. SC as a feed forward neural network block 835. The feed forward neural network block 835 performs sentiment prediction ( e.g., negative, neutral, positive) at the aspect term level. The sentiments are advantageously trained for each aspect term" where a feed forward neural network is a deep learning model and the sentiment classifier is the second model].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Kottha in view Kanagovi because applying the machine learning aspect extraction techniques with the deep learning sentiment prediction techniques to the performance reviews, would thereby improve the graduality , accuracy, and explainability of employee performance evaluations.
Regarding claim 6, the method of claim 1 is incorporated. Kottha, Kanagovi and Yan teach all the limitations of the current invention as stated above.
Kottha does teach
wherein the first performance review consists of textual data.
[0016 “FIG. 1 illustrates an example system 100 that performs natural language processing of unstructured text. Briefly, and as described in further detail below, the system
100 receives review data that is from employees who are providing feedback to their employers related to their work environment.” ]
Regarding claim 9, Kottha teaches
A computer readable medium having instructions stored thereon that, when executed by one or more processors, cause the processor to evaluate performance, the evaluating performance comprising: receiving a plurality of training performance reviews;
[0064 “These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object oriented programming language, and/or in assembly/machine language. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine readable medium that receives machine instructions as machine-readable signal”];
[0017 " The review data 105 and 110 may each be associated with a particular employee with the system relating each of the review data 105 and 110
with an employee identifier and may not be easily parsable" where "each" of the "review data 105 and 110" implies there are at least two distinct performance records].
extracting from the training performance reviews, using a first machine learning model, a plurality of features comprising a training aspect, a training sentiment,
[0016 "..an example system 100 that performs natural language processing of unstructured text. Briefly, and as described in further detail below, the system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment. The system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment" where employee feedback provided to an employer regarding work-related experiences constitutes performance review"];
[0032 “The system 205 provides the data from the data sources to the machine learning/AI/analytics engine 210. The machine learning/AI/analytics engine 210 analyzes the
data from the data sources 205 using machine learning and natural language processing to identify topics/aspects and corresponding sentiments. The machine learning/AI/analytics
engine 210 may also identify trends and correlate the topics/aspects and sentiments to groups of employees and generate user interfaces based on the topics, sentiments, and
groups”].
[0033 “The machine learning/AI/analytics engine 210 includes a natural language information extractor 220 that processes the data received from the data sources 205 based
on natural processing techniques. The natural language information extractor 220 includes a data extraction engine 230. The data extraction engine 230 is configured to process
unstructured data such as text reviews that employees provide to review websites or directly to their employers"];
[0035 “The data analysis engine 225 may extract sentiments from the data entries using recursive neural tensor networks or other deep learning algorithms.”].
[0023 "The system 100 may train the machine learning system 135 using the training
data 140. The training data 140 may include various entries of review data and assigned topics/aspects and sentiments" ]
receiving a first performance review;
[0016 "The system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment" where employee feedback provided to an employer regarding work-related experiences constitutes performance review"].
extracting from the first performance review one or more first aspects,
[0016 "..an example system 100 that performs natural language processing of unstructured text. Briefly, and as described in further detail below, the system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment. The system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment" where employee feedback provided to an employer regarding work-related experiences constitutes performance feedback"];
[0032 “The system 205 provides the data from the data sources to the machine learning/AI/analytics engine 210. The machine learning/AI/analytics engine 210 analyzes the data from the data sources 205 using machine learning and natural language processing to identify topics/aspects and
corresponding sentiments. The machine learning/AI/analytics engine 210 may also identify trends and correlate the topics/aspects and sentiments to groups of employees and generate user interfaces based on the topics, sentiments, and groups”]
[0033 “The machine learning/AI/analytics engine 210 includes a natural language information extractor 220 that processes the data received from the data sources 205 based
on natural processing techniques. The natural language information extractor 220 includes a data extraction engine 230. The data extraction engine 230 is configured to process
unstructured data such as text reviews that employees provide to review websites or directly to their employers"];
[0035 “The data analysis engine 225 may extract sentiments from the data entries using recursive neural tensor networks or other deep learning algorithms.”].
However, Kottha does not expressly recite
using the extracted plurality of features to train a second machine learning model;
and using the trained second machine learning model, predicting first sentiment scores for each of the first aspects.
But Kanagovi teaches
using the extracted plurality of features to train a second machine learning model;
[Column 18, lines 6-17 "The output of the context retainer block 830 is provided to a downstream sentiment classifier, shown in FIG. 8C as a feed forward neural network block 835. The feed forward neural network block 835 performs sentiment prediction (e.g., negative, neutral, positive) at the aspect term level. The sentiments are advantageously trained for each aspect term at a time (e.g., by activating and averaging out the encodings of the tokens for one aspect term only). For example, an article or other document having four aspect terms will separately train the BERT Sentiments model 815 for each of the four aspect terms, one at a time" where the sentiment classifier is the second model]; [Column 16, lines 3-8 "In some embodiments, both tasks performed by the aspect term sentiment analysis logic 116—aspect term extract and prediction of sentiments for extracted aspect terms, are performed using BERT models that are trained in parallel by averaging out the BERT encodings of the aspect terms from both of the models"].
using the trained second machine learning model, predicting first sentiment scores for each of the first aspects.
[Column 18, lines "6-10 The output of the context retainer block 830 is provided
to a downstream sentiment classifier, shown in FIG. SC as a feed forward neural network block 835. The feed forward neural network block 835 performs sentiment prediction ( e.g., negative, neutral, positive) at the aspect term level." where sentiment prediction of negative, neutral or positive correspond to sentiment scores].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Kottha into the teachings of Kanagovi because Kottha teaches extracting aspects and sentiments from performance reviews while Kanagovi teaches using extracted aspect terms to train a downstream sentiment prediction model. Both of these references process textual review documents to extract aspect-level sentiment information using machine learning models. Combining them would enable extracted review features to train a second sentiment model, thereby improving granularity and accuracy of sentiment analysis.
Kottha in view of Kanagovi do not expressly recite
a corresponding training evidence;
one or more corresponding first evidences,
However, Yan teaches
a corresponding training evidence;
[Introduction, lines 1-4 "Aspect-based Sentiment Analysis (ABSA) is the fine-grained Sentiment Analysis (SA) task, which aims to identify the aspect term (a), its corresponding sentiment polarity (s), and the opinion term (o)" wherein, in the context of the employee performance reviews described by Kottha, the opinion terms extracted by the machine learning model of Yan (BART Page 3) correspond to the claimed training evidence associated with the extracted aspects and sentiments].
one or more corresponding first evidences,
[Introduction, lines 1-4 "Aspect-based Sentiment Analysis (ABSA) is the fine-grained Sentiment Analysis (SA) task, which aims to identify the aspect term (a), its corresponding sentiment polarity (s), and the opinion term (o)" wherein, in the context of the employee performance reviews described by Kottha, the opinion terms extracted by the machine learning model of Yan (BART Page 3) correspond to the claimed first evidence associated with the extracted aspects and sentiments].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Kottha in view Kanagovi with the teachings of Yan because combining Yan’s teaching of extracting opinion terms associated with aspect and sentiment labels, wherein the opinion terms provide textual support for the sentiment determination, would thereby provide a more granular, accurate, and explainable evolution of employee performance.
Regarding claim 10, the rejection of claim 2 is incorporated. Claim 10 is substantially the same as claim 2 and is therefore rejected under the same rationale as above.
Regarding claim 13, the rejection of claim 5 is incorporated. Claim 13 is substantially the same as claim 5 and is therefore rejected under the same rationale as above.
Regarding claim 14, the rejection of claim 6 is incorporated. Claim 14 is substantially the same as claim 6 and is therefore rejected under the same rationale as above.
Regarding claim 16, the rejection of claim 8 is incorporated. Claim 16 is substantially the same as claim 8 and is therefore rejected under the same rationale as above.
Regarding claim 17, Kottha teaches
A performance evaluation system comprising:
[0016 “FIG. 1 illustrates an example system 100 that performs natural language processing of unstructured text. Briefly, and as described in further detail below, the system
100 receives review data that is from employees who are providing feedback to their employers related to their work environment. The system 100 processes the review data and
identifies different topics or aspects or both (topics/aspects) for each review and sentiments that corresponds to each of the different topics/aspects. The system 100 then correlates
the sentiments and topics/aspects to the employer's data to identify groups of employees whose reviews are related to similar topics/aspects and sentiments”];
[0026” The trend analyzer 145 provides the review data 105 and 110 along with the identified topics, sentiments, and trends to the internal data analyzer 155. The internal data analyzer 155 analyzes internal data 160 that may only be accessible to an entity that is being reviewed. The internal data 160 may include demographic and employment data.
For example, the internal data 160 may be human resources data that include for each employee, gender information, race information, employment dates, performance review
information, income information, age, and tenure”];
a first machine learning model;
[0032 “The system 205 provides the data from the data sources to the machine learning/AI/analytics engine 210. The machine learning/AI/analytics engine 210 analyzes the
data from the data sources 205 using machine learning and natural language processing to identify topics/aspects and corresponding sentiments”]
one or more processors configured to: receive a plurality of training performance reviews;
[0017 " The review data 105 and 110 may each be associated with a particular employee with the system relating each of the review data 105 and 110 with an employee identifier and may not be easily parsable" where "each" of the "review data 105 and 110" implies there are at least two distinct performance records].
extract from the training performance reviews, using a first machine learning model, a plurality of features comprising a training aspect, a training sentiment,
[0016 "..an example system 100 that performs natural language processing of unstructured text. Briefly, and as described in further detail below, the system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment. The system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment" where employee feedback provided to an employer regarding work-related experiences constitutes performance feedback"];
[0032 “The system 205 provides the data from the data sources to the machine learning/AI/analytics engine 210. The machine learning/AI/analytics engine 210 analyzes the
data from the data sources 205 using machine learning and natural language processing to identify topics/aspects and corresponding sentiments. The machine learning/AI/analytics
engine 210 may also identify trends and correlate the topics/aspects and sentiments to groups of employees and generate user interfaces based on the topics, sentiments, and
groups”].
[0033 “The machine learning/AI/analytics engine 210 includes a natural language information extractor 220 that processes the data received from the data sources 205 based
on natural processing techniques. The natural language information extractor 220 includes a data extraction engine 230. The data extraction engine 230 is configured to process
unstructured data such as text reviews that employees provide to review websites or directly to their employers"];
[0035 “The data analysis engine 225 may extract sentiments from the data entries using recursive neural tensor networks or other deep learning algorithms.”].
[0023 "The system 100 may train the machine learning system 135 using the training
data 140. The training data 140 may include various entries of review data and assigned topics/aspects and sentiments" ]
receiving a first performance review;
[0016 "The system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment" where employee feedback provided to an employer regarding work-related experiences constitutes performance review"].
extract from the first performance review one or more first aspects,
[0016 "..an example system 100 that performs natural language processing of unstructured text. Briefly, and as described in further detail below, the system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment. The system 100 receives review data that is from employees who are providing feedback to their employers related to their work environment" where employee feedback provided to an employer regarding work-related experiences constitutes performance review"];
[0032 “The system 205 provides the data from the data sources to the machine learning/AI/analytics engine 210. The machine learning/AI/analytics engine 210 analyzes the data from the data sources 205 using machine learning and natural language processing to identify topics/aspects and
corresponding sentiments. The machine learning/AI/analytics engine 210 may also identify trends and correlate the topics/aspects and sentiments to groups of employees and generate user interfaces based on the topics, sentiments, and groups”]
[0033 “The machine learning/AI/analytics engine 210 includes a natural language information extractor 220 that processes the data received from the data sources 205 based
on natural processing techniques. The natural language information extractor 220 includes a data extraction engine 230. The data extraction engine 230 is configured to process
unstructured data such as text reviews that employees provide to review websites or directly to their employers"];
[0035 “The data analysis engine 225 may extract sentiments from the data entries using recursive neural tensor networks or other deep learning algorithms.”].
However, Kottha does not expressly recite
a second machine learning model;
use the extracted plurality of features to train a second machine learning model;
and using the trained second machine learning model, predicting first sentiment scores for each of the first aspects.
But Kanagovi teaches
a second machine learning model;
[Column 18, lines 6-17 "The output of the context retainer block 830 is provided to a downstream sentiment classifier, shown in FIG. 8C as a feed forward neural network block 835. The feed forward neural network block 835 performs sentiment prediction (e.g., negative, neutral, positive) at the aspect term level. The sentiments are advantageously trained for each aspect term at a time (e.g., by activating and averaging out the encodings of the tokens for one aspect term only). For example, an article or other document having four aspect terms will separately train the BERT Sentiments model 815 for each of the four aspect terms, one at a time." where the sentiment classifier is the second model];
use the extracted plurality of features to train a second machine learning model;
[Column 18, lines 6-17 "The output of the context retainer block 830 is provided to a downstream sentiment classifier, shown in FIG. 8C as a feed forward neural network block 835. The feed forward neural network block 835 performs sentiment prediction (e.g., negative, neutral, positive) at the aspect term level. The sentiments are advantageously trained for each aspect term at a time (e.g., by activating and averaging out the encodings of the tokens for one aspect term only). For example, an article or other document having four aspect terms will separately train the BERT Sentiments model 815 for each of the four aspect terms, one at a time" where the sentiment classifier is the second model]; [Column 16, lines 3-8 "In some embodiments, both tasks performed by the aspect term sentiment analysis logic 116—aspect term extract and prediction of sentiments for extracted aspect terms, are performed using BERT models that are trained in parallel by averaging out the BERT encodings of the aspect terms from both of the models"].
using the trained second machine learning model, predicting first sentiment scores for each of the first aspects.
[Column 18, lines "6-10 The output of the context retainer block 830 is provided
to a downstream sentiment classifier, shown in FIG. SC as a feed forward neural network block 835. The feed forward neural network block 835 performs sentiment prediction ( e.g., negative, neutral, positive) at the aspect term level." where sentiment prediction of negative, neutral or positive correspond to sentiment scores].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Kottha into the teachings of Kanagovi because Kottha teaches extracting aspects and sentiments from performance reviews while Kanagovi teaches using extracted aspect terms to train a downstream sentiment prediction model. Both of these references process textual review documents to extract aspect-level sentiment information using machine learning models. Combining them would enable extracted review features to train a second sentiment model, thereby improving granularity and accuracy of sentiment analysis.
Kottha in view of Kanagovi do not expressly recite
a corresponding training evidence;
one or more corresponding first evidences,
However, Yan teaches
a corresponding training evidence;
[Introduction, lines 1-4 "Aspect-based Sentiment Analysis (ABSA) is the fine-grained Sentiment Analysis (SA) task, which aims to identify the aspect term (a), its corresponding sentiment polarity (s), and the opinion term (o)" wherein, in the context of the employee performance reviews described by Kottha, the opinion terms extracted by the machine learning model (BART Page 3) of Yan correspond to the claimed training evidence associated with the extracted aspects and sentiments].
one or more corresponding first evidences,
[Introduction, lines 1-4 "Aspect-based Sentiment Analysis (ABSA) is the fine-grained Sentiment Analysis (SA) task, which aims to identify the aspect term (a), its corresponding sentiment polarity (s), and the opinion term (o)" wherein, in the context of the employee performance reviews described by Kottha, the opinion terms extracted by the machine learning model (BART Page 3) of Yan correspond to the claimed first evidence associated with the extracted aspects and sentiments].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Kottha in view Kanagovi with the teachings of Yan because combining Yan’s teaching of extracting opinion terms associated with aspect and sentiment labels, wherein the opinion terms provide textual support for the sentiment determination, would thereby provide a more granular, accurate, and explainable evolution of employee performance.
Regarding claim 18, the rejection of claim 2 is incorporated. Claim 18 is substantially the same as claim 2 and is therefore rejected under the same rationale as above.
Claim(s) [ 3, 4, 8, 11, 12, 19, 20] is/are rejected under 35 U.S.C. 103 as being unpatentable over Kottha, Kanagovi, and Yan as applied to claims 1, 9, and 17 above, and further in view of Muthuswamy (US 10,796,265 B2) and in further view of Locke (Locke J, Violante S, Pullmann MD, Kerns SEU, Jungbluth N, Dorsey S. Agreement and Discrepancy Between Supervisor and Clinician Alliance: Associations with Clinicians' Perceptions of Psychological Climate and Emotional Exhaustion. Adm Policy Ment Health. 2018 May;45(3):505-517. doi: 10.1007/s10488-017-0841-y. PMID: 29230606; PMCID: PMC6040831).
Regarding claim 3, the rejection of claim 1 is incorporated. Kottha in view of Kanagovi and in further view of Yan do not expressly recite
wherein the first performance review comprises a manager portion and an employee portion, further comprising: determining a mismatch score for each of one or more first aspects that are common to the manager portion and the employee portion.
However, Muthuswamy teaches
wherein the first performance review comprises a manager portion and an employee portion [Column 4, 60-66 "Further, the performance evaluating system may receive a feedback in the review matrix for each of the one or more employees from the one or more recommenders. The feedback comprises a recommender's review score and review comments. In some embodiments, the one or more recommenders may in turn receive a feedback from the one or more employees" where feedback from recommenders is the manager portion and feedback from the employees is the employee portion];
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Kottha in view Kanagovi and in view of Yan with the teachings of Muthuswamy because doing so would enable performance reviews to include both a manager portion and an employee portion, thereby providing a more comprehensive evaluation of performance.
However, Kottha in view of Kanagovi and in further view of Yan and in further view of Muthuswamy do not expressly recite
determining a mismatch score for each of one or more first aspects that are common to the manager portion and the employee portion
But Locke teaches
determining a mismatch score for each of one or more first aspects that are common to the manager portion and the employee portion.
[Data Analysis section, Page 6 “We first inspected the data to ensure adequate frequency of discrepancies between supervisor and clinician ratings for supervisory relationship and working alliance (Shanock et al. 2010), as this was a prerequisite for any further analyses examining discrepancy and agreement. Standardized scores were calculated for each predictor variable. As suggested in Shanock et al. (2010) and used in Aarons et al. (2017), participant pairs with a standardized score for supervisor-rated alliance that was half a standard deviation above or below the standardized score on clinician-rated alliance were considered discrepant.” where supervisor is the manager and clinician ratings is coming from clinicians(employees). The discrepancies(different scores) serve as mismatch scores ];
[Section Measures, Page 5, The supervisory relationship was assessed using the Supervisor/Trainee Personal Reaction Scale-Revised (SPRS-R/TPRS-R; Holloway and Wampold 1984). The 12-item SPRS-R and TPRS-R measure critical factors in the supervisory interpersonal relationship including feelings and reactions toward a supervisee/ supervisor, respectively. The measure includes items that refer to the self (e.g., “Sometimes after the supervisor said something I just couldn’t think of any response”; “I felt pretty ineffective with this supervisee”), other (e.g., “I disagree with this supervisee about some basic matters”) and comfort (e.g., “I got irritated at some of my supervisor’s remarks”). The items are rated on a 5-point Likert-type scale ranging from 1 (not characteristic of my feelings) to 5 (highly characteristic of my feelings)” where the interpersonal factors measured by SPRS-R outline here show the aspects that are common to both supervisor(manager) and the clinicians(employee)].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Kottha in view of Kanagovi and in further view of Yan and in further view of Muthuswamy with the teachings of Locke because extracting common aspects from the manager and employee review portions and determining discrepancies between the respective assessments for those aspects in order to identify areas of alignment and disagreement would thereby improve the granularity, explainability, and usefulness of employee performance evaluations.
Regarding claim 4 the rejection of claim 3 is incorporated. Kottha, Kanagovi, Yan, Muthuswamy teach all limitations of the current invention as stated above.
However, Kottha fails to expressly recite further comprising determining an overall mismatch first performance review based on the first aspect weights and the mismatch score
But, Locke teaches determining an overall mismatch based on the aspect weights and the mismatch score
[Data Analysis section, Page 6 “We first inspected the data to ensure adequate frequency of discrepancies between supervisor and clinician ratings for supervisory relationship and working alliance (Shanock et al. 2010), as this was a prerequisite for any further analyses examining discrepancy and agreement. Standardized scores were calculated for each predictor variable. As suggested in Shanock et al. (2010) and used in Aarons et al. (2017), participant pairs with a standardized score for supervisor-rated alliance that was half a standard deviation above or below the standardized score on clinician-rated alliance were considered discrepant.” where supervisor is the manager and clinician ratings is coming from clinicians(employees). The discrepancies(different scores) serve as mismatch scores ];
[Page 7 “We first inspected the data to ensure adequate frequency of discrepancies between supervisor and clinician ratings for supervisory relationship and working alliance (Shanock et al. 2010), as this was a prerequisite for any further analyses examining discrepancy and agreement. Standardized scores were calculated for each predictor variable (Fleenor et al. 1996). As suggested in Shanock et al. (2010) and used in Aarons et al. (2017), participant pairs with a standardized score for supervisor-rated alliance that was half a standard deviation above or below the standardized score on clinician-rated alliance were considered discrepant. We then centered the predictors (supervisor and clinician ratings of supervisory relationship and working alliance) around the midpoint for each scale (3, 4, and 2, respectively; Atwater et al. 1998; Shanock et al. 2010). Subsequently, we created three new variables for each alliance measure: (a) the square of the centered variable for the supervisor-rated alliance; (b) the cross-product of the centered supervisor- and clinician-rated alliance measure; and (c) the square of the centered clinician-rated alliance. To examine model significance, the proportion of the variance explained in outcome (R2) was evaluated” wherein the supervisory working alliance dimensions measured by the SWAI correspond to the one of more first aspects, the discrepancies between supervisor and clinician ratings for the SWAI dimensions correspond to one or more mismatch scores, the centered predictor variables associated with the SWAI dimensions, which adjust the relative contribution of each dimension correspond to first aspect weights, such that the frequency and magnitude of the discrepancies correspond to the overall mismatch score].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Kottha in view of Kanagovi and in further view of Yan and in further view of Muthuswamy with the teachings of Locke because incorporating Locke’s teachings of examining agreement and discrepancy between supervisor and clinician ratings and treating the discrepancy between the two predictor variables as a critical factor in the analysis would improve the usefulness and interpretability of performance reviews.
Regarding claim 8, the rejection of claim 4 is incorporated. Kottha, Kanagovi, Yan, Muthuswamy and Locke teach all limitations of the current invention as state above.
Kanagovi teaches extracting actional items
[Column 19, lines 63-67 ,Column 20 , lines 1-3 "Sentiment analysis methods may be used to analyze customer reviews (e.g., from various online stores), social media comments, critic reviews, etc. This is particularly useful for various groups or teams within an entity, including
for tasks such as product engineering and for product group teams to draw necessary actionable insights to decide on product price moves, product quality, engineering, validating
marketing efforts, etc”]; [Column 20, lines 17-18 “Survey analysis 1005 may include drawing accurate and actionable insights from customer feedback surveys”].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Kottha with the teachings of Kanagovi
because applying the actionable insight generation techniques of Kanagovi to the performance reviews would enable extraction of actionable insights indicative of employee strengths, weakness, and opportunities for improvement, thereby improving the usefulness of employee performance evaluations.
Regarding claim 11, the rejection of claim 3 is incorporated. Claim 11 is substantially the same as claim 3 and is therefore rejected under the same rationale as above.
Regarding claim 12, the rejection of claim 4 is incorporated. Claim 12 is substantially the same as claim 4 and is therefore rejected under the same rationale as above.
Regarding claim 19, the rejection of claim 3 is incorporated. Claim 19 is substantially the same as claim 3 and is therefore rejected under the same rationale as above.
Regarding claim 20, the rejection of claim 4 is incorporated. Claim 20 is substantially the same as claim 4 and is therefore rejected under the same rationale as above.
Claim [7, 15 ] are rejected under 35 U.S.C. 103 as being unpatentable over Kottha (US 2017/0193397 A1) in view of Kanagovi (US11675823B2) and in further view of Yan (Yan, Hang, et al. "A unified generative framework for aspect-based sentiment analysis." Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: Long papers). 2021) and in further view of Rehling (US8463595B1).
Regarding claim 7, the rejection of claim 1 is incorporated. Kottha, Kanagovi, and Yan teach all limitations of the current invention as stated above.
Kottha further teaches
wherein the first sentiment comprises a negative, neutral or positive label
[0022 “The sentiment may be negative, neutral, or positive.”]
However, Kottha does not expressly recite the first sentiment score comprises a probability of belonging to one of the first sentiment.
Rehling teaches
the first sentiment score comprises a probability of belonging to one of the first sentiment.
[Column 3, lines "45-48 As will be described in more detail below, detailed sentiment
analysis system 102 is configured to determine a set of sentiment
scores (used to assign "sentiment labels") for the documents it receives, across a plurality of dimensions"];
[Column 5, lines 35-40 "As one example, if the Naive Bayes classifier variant of machine learning is used, the model is a table of probabilities mapping each feature (short
sequences of stemmed words) to the probability of a sentiment label (i.e., "positive" or "negative" sentiment) being appropriate for the text."].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine teachings of Kottha in view Kanagovi and in view of Yan with the teachings of Rehling because incorporating the probabilistic sentiment classification techniques in the system resulting from the combination of Kottha, Kanagovi and Yan in order to provide confidence measures for predicted sentiments would thereby improve the reliability, interpretability, and usefulness of employee performance evaluations.
Regarding claim 15, the rejection of claim 7 is incorporated. Claim 15 is substantially the same as claim 7 and is therefore rejected under the same rationale as above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHEZA ABDUL AZIZ whose telephone number is (571)272-9610. The examiner can normally be reached Monday-Friday 7:30am-5pm Alternate Fridays off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHEZA ABDUL AZIZ/Examiner, Art Unit 2657
/Sean E Serraguard/Primary Examiner, Art Unit 2657
Prosecution Insights