DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 08/05/26 have been fully considered but they are not persuasive.
Applicant argues that Janz does not disclose "rater electronic device" that supplies rating data obtained from humans for quality-assurance comparison; generate quality assurance data that identifies a rating as an "outlier" flagged "for re-review." (Amendment, pages 6 – 8).
The examiner disagrees, since Janz discloses “Outliers can unduly influence the clusters, especially when a squared error criterion is used. However, in some clustering applications, the outliers should not be eliminated or discounted, as their appropriate inclusion may lead to important insights… A plurality of human users may each classify the same exemplar response, further comprising determining an inter-user agreement rating for the classification. A plurality of human users may each classify the same exemplar response, the method further comprising determining a discrimination factor for the classification with respect to at least one of the at least two non-binary ratings of independent characteristics… receiving feedback information reflecting a performance of the interviewee after the evaluation of the interviewee, and updating the natural language speech analyzer based on at least the feedback.” (paragraphs 127, 171 – 174).
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1 – 14, 16 – 21, 25 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Janz (US PAP 2018/0025303).
As per claims 1, 13, Janz discloses a computer system for natural language processing for quality assurance for constructed- response tests to identify rating outliers for re-review, the system comprising:
a memory (paragraph 180);
a processor coupled to the memory programmed with executable instructions, the instructions including an interface for obtaining rating data and response data for a constructed-response test, wherein the processor executes the instructions to provide a natural language processing engine to generate predicted rating data for the response data for the constructed- response test (paragraphs 64 – 66, 180),
wherein the processor compares the predicted rating data to the rating data, generates quality assurance data that evaluates and validates the rating data obtained from a human rater based on results of the comparison using one or more models to accurately assess whether the rater data is accurate and identify a rating outlier for re-review and transmits the quality assurance data that evaluates the rating data to provide feedback on the rating data and identifies the rating outlier for re-review, the quality assurance data for display at interfaces of electronic devices (“determining an inter-user agreement rating for the classification…receiving feedback information reflecting a performance of the interviewee after the evaluation of the interviewee, and updating a statistical model for the statistical processing based on at least the feedback.”; paragraphs 127, 170 - 174);
an examinee electronic device for collecting the response data for the constructed-response test, the device having a transceiver for transmitting the collected response data to the interface (“determine a correspondence of the analyzed human communications to a set of classified human communications comprising at least one of exemplar responses collected from prior interviewees' respective responses to corresponding respective interview prompts in a context”; paragraphs 96, 153, 158, 159); and
a rater electronic device for collecting the rating data for the response data for the constructed-response test, the electronic device having a transceiver for transmitting the collected rating data to the interface (“A plurality of human users may each classify the same exemplar response, further comprising determining an inter-user agreement rating for the classification. A plurality of human users may each classify the same exemplar response, the method further comprising determining a discrimination factor for the classification with respect to at least one of the at least two non-binary ratings of independent characteristics…in computer-mediated communications of authors who transmit or receive computer-mediated communications, such as, but not limited to, email, chat, website content, etc.”; paragraphs 81, 87, 153,170, 171).
As per claim 18, Janz teaches a computer process for natural language processing for quality assurance of rating data for constructed-response tests to identify rating outliers for re-review, the process comprising:
by a processor coupled to memory programmed with executable instructions, the instructions including an interface for obtaining rating data and response data for a constructed-response test, wherein the rating data comprises ratings or scores of the response data for the constructed-response test, wherein the processor executes the instructions to provide a natural language processing engine(“determining an inter-user agreement rating for the classification…receiving feedback information reflecting a performance of the interviewee after the evaluation of the interviewee, and updating a statistical model for the statistical processing based on at least the feedback.”; paragraphs 170 - 173);
generating, using the response data, combined response data for multiple questions for a scenario of the constructed-response test (paragraphs 100, 155, 156).
extracting features from the combined response data; generating training data set and testing data set from the features; developing, training and validating one or more models using the training data (“the cue generator includes an audio cue generator that collects audio data from the training data set (or current data) and identifies utterances in the audio signal of a digital interview by a candidate.” paragraphs 96, 97);
selecting a model from the one or more models; generating predicted ratings using the testing data set and the selected model(“The voice data is mined by applying a predetermined linguistic-based psychological behavioral model, and behavioral assessment data is generated that corresponds to the analyzed voice data. Certain psychological behavioral models have been developed as tools to evaluate and understand how and/or why one person or a group of people interacts with another person or group of people.”; paragraphs 7, 8);
generating quality assurance data that evaluates and validates the rating data obtained from a human rater by comparing the predicted ratings to the rating data to identify a rating outlier for re-review (“determining an inter-user agreement rating for the classification…receiving feedback information reflecting a performance of the interviewee after the evaluation of the interviewee, and updating a statistical model for the statistical processing based on at least the feedback.”; paragraphs 170 - 173); and
transmitting the quality assurance output data that evaluates the rating data to an electronic device and identifies the rating outlier for re-review for display or storing the quality assurance output data that evaluates the rating data and identifies the rating outlier for re-review in the memory to provide feedback on the rating data (paragraphs 97, 127, 153, 206, 207).
As per claim 2, Janz further discloses the processor generates, using the response data, combined response data for multiple questions for a scenario of the constructed-response test (paragraphs 100, 155, 156).
As per claim 3, Janz further discloses the processor extracts features from the combined response data, and generates training data set and testing data set from the features for training and testing the one or more models (“the cue generator includes an audio cue generator that collects audio data from the training data set (or current data) and identifies utterances in the audio signal of a digital interview by a candidate.” paragraphs 96, 97).
As per claim 4, Janz further discloses the processor develops, trains and validates the one or more models using the training data and the extracted features and rating data (paragraphs 96, 97).
As per claim 5, Janz further discloses the processor selects a model from the one or more models with hyperparameters that produce an optimized average error measurement, and generates the predicted ratings using the testing data set and the selected model (“The voice data is mined by applying a predetermined linguistic-based psychological behavioral model, and behavioral assessment data is generated that corresponds to the analyzed voice data... the distance between centroids or medioids of the clusters, the closest points in two clusters, the further points in two clusters, or some average distance metric. Ward's method measures the proximity between two clusters in terms of the increase in the sum of the squares of the errors that results from merging the two clusters.”; paragraphs 7, 8, 123 - 128).
As per claim 6, Janz further discloses the processor uses the natural language processing engine for unsupervised text classification to categorize the response data based on similarities between the response data and one or more topics for the constructed response test (“Requests for narrative information on customary topics will generally be regarded as a normal and expected part of a candidate-matching system, and customary topics may therefore be preferable.”; paragraph 5).
As per claim 7, Janz further discloses the processor determines alignment between the one or more topics for the constructed response test, the response data, and the rating data to compute precision and recall data for the one or more topics for the constructed response test (“A plurality of human users may each classify the same exemplar response, further comprising determining an inter-user agreement rating for the classification. A plurality of human users may each classify the same exemplar response, the method further comprising determining a discrimination factor for the classification with respect to at least one of the at least two non-binary ratings of independent characteristics.”; paragraphs 5, 81, 87, 170, 171).
As per claims 8, 16, Janz further discloses the processor uses the natural language processing engine for unsupervised text classification to assign labels or categories to the response data and evaluate alignment between the assigned labels or categories and one or more topics or aspects of the constructed response test (“the mapping from a set of input data vectors to a finite set of discrete class labels is modeled in terms of some mathematical function including a vector of adjustable parameters.”; paragraph 110, 157 - 159).
As per claims 9, 17, Janz further discloses the processor uses the natural language processing engine for sentiment analysis to compute sentiment and subjectivity scores for the response data, wherein the processor compares sentiment and subjectivity scores for the response data to the rating data to generate the quality assurance data that evaluates the rating data (paragraph 97).
As per claim 10, Janz further discloses the processor transmits the quality assurance output data to the rater electronic device for display using one or more visual elements to provide real-time or near real-time feedback on the rating data and stores the quality assurance output data in the memory(paragraphs 97, 206, 207).
As per claim 11, Janz further discloses the processor extracts features from the response data, generates one or more models using the extracted features, and generates the predicted ratings using a selected model of the one or more models (paragraphs 96- 100).
As per claims 12, 20 Janz further discloses one or more cameras or sensors to generate the response data (paragraphs 97- 100, 158 - 178).
As per claim 14 Janz further discloses the processor uses the natural language processing engine for the unsupervised text classification to categorize the response data based on similarities between the response data and one or more topics for the constructed response test, and determines alignment between the one or more topics for the constructed response test, the response data, and the rating data to compute precision and recall for the one or more topics for the constructed response test (paragraphs 5, 81, 87, 170, 171).
As per claim 19 Janz further discloses extracting features from the response data, generating one or more models using the extracted features, selecting a model of the one or more models with hyperparameters that produce an optimized average error measurement and generating the predicted ratings using the selected model of the one or more models (“The voice data is mined by applying a predetermined linguistic-based psychological behavioral model, and behavioral assessment data is generated that corresponds to the analyzed voice data... the distance between centroids or medioids of the clusters, the closest points in two clusters, the further points in two clusters, or some average distance metric. Ward's method measures the proximity between two clusters in terms of the increase in the sum of the squares of the errors that results from merging the two clusters.”; paragraphs 7, 8, 123 - 128).
As per claim 21 Janz further discloses implementing sentiment analysis and unsupervised text classification to generate predicted rating data for the response data for the constructed- response test (paragraph 97).
As per claim 25 Janz further discloses the processor tests the one or more models with validation and hyperparameter tuning to optimize an average cross-validated error measurement between predicted rating data and human rating data (paragraphs 123 – 128).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEONARD SAINT-CYR whose telephone number is (571)272-4247. The examiner can normally be reached Monday- Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richemond Dorvil can be reached at (571)272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LEONARD SAINT-CYR/ Primary Examiner, Art Unit 2658
Prosecution Insights