DETAILED ACTION
Claims 1-20 of the instant application are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. More specifically directed to the abstract idea grouping of: mental process and/or certain methods of organizing human activity .
The independent claim(s) recite(s):
1. A system comprising:
one or more computer readable storage media;
one or more processors operatively coupled with the one or more computer readable storage media; and
an application comprising program instructions stored on the one or more computer readable storage media that, when executed by the one or more processors, direct a computing system to at least:
determine, by a formative feedback engine, a written assessment associated with a first client device;
determine, by the formative feedback engine, first content of the written assessment;
vectorize, by the formative feedback engine, the first content to form a semantic representation of the first content;
identify, by the formative feedback engine, semantically equivalent content to the semantic representation of the first content;
determine, by the formative feedback engine, textual feedback associated with the semantically equivalent content;
generate, by the formative feedback engine, a first commentary insight for the first content based on the textual feedback; and
provide, by the formative feedback engine, the first commentary insight for the written assessment to a client device.
This reads on a human (e.g., mentally and/or using pen and paper):
Determining a written assessment (e.g., text) is associated with a second human (and/or their device);
Determining or identifying content from said written assessment/text;
Applying a predetermined set of rules to write the content in a vectorized form associated with semantic representation;
Identifying content that is semantically equivalent to the content;
Determine feedback associated with the semantically equivalent content;
Generating a comment based on the feedback and
Writing down and providing said comment to the second human.
7. A method comprising:
receiving, from a client device, a written assessment comprising first content;
generating, by a formative feedback engine, a semantic representation of the first content;
querying, by the formative feedback engine, a semantic store based on the semantic representation of the first content, wherein the semantic store comprises a plurality of commentary insights associated with one or more previously evaluated written assessments;
determining, by the formative feedback engine, textual feedback from the semantic store based on the semantic representation of the first content;
generating, by the formative feedback engine, a commentary insight for the first content based on the textual feedback and the first content; and
providing, by the formative feedback engine, the commentary insight to the client device.
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving a written assessment (e.g., text) associated with a second human (and/or their device), comprising content;
Applying a predetermined set of rules to generate a semantic representation of the content (e.g., vectorized form);
Identifying content that is semantically equivalent to the content in a data store (e.g., book);
Determine feedback associated with the semantically equivalent content;
Generating a comment based on the feedback and
Writing down and providing said comment to the second human.
15. A computer readable storage media comprising processor-executable instructions configured to cause one or more processors to:
determine, by a formative feedback engine, a written assessment comprising first content;
vectorize, by a formative feedback engine, the first content to form a first semantic representation of the first content;
compare, by the formative feedback engine, the first semantic representation to a semantic store;
identify, by the formative feedback engine, a first textual feedback based on the comparison of the first semantic representation to the semantic store;
generate, by the formative feedback engine, a first commentary insight for the first content based on the first textual feedback; and
provide, by the formative feedback engine, the first commentary insight as associated with the first content within the written assessment.
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving a written assessment (e.g., text) associated with a second human (and/or their device), comprising content;
Applying a predetermined set of rules to write the content in a vectorized form associated with semantic representation;
Identifying content that is semantically equivalent to the content;
Determine feedback associated with the semantically equivalent content;
Generating a comment based on the feedback and
Writing down and providing said comment to the second human.
This judicial exception is not integrated into a practical application because for example: claim 1 recites “a system,” “one or more computer readable storage media,” “one or more processors operatively coupled with the one or more computer readable storage media,” “an application comprising program instructions stored on the one or more computer readable storage media that, when executed by the one or more processors, direct a computing system,” “a formative feedback engine,” and “a client device,” similarly, claim 7 recites “a client device,” “a formative feedback engine,” and “a semantic store,” and lastly, claim 15, recites “A computer readable storage media,” “one or more processors,” “a formative feedback engine,” and “a semantic store.” As an example, in [0075] of the as filed specification, it is disclosed: In general, the software 895 may, when loaded into the processing system 892 and executed, transform a suitable apparatus, system, or device (of which computing system 891 is representative) overall from a general-purpose computing system into a special-purpose computing system customized to generate features, functionality, and user experiences provided by the formative feedback engine. Therefore, a general-purpose computer or computing device is described and mainly used as an application thereof. Accordingly, these additional elements do not integrate the abstract idea into a practical idea because it does not impose any meaningful limits on practicing the abstract idea.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of using a computer is listed as a general computing device as noted. The claim is not patent eligible.
With respect to claims 2 and 8, the claim(s) recite:
2. The system of claim 1, wherein the program instructions to generate, by the formative feedback engine, the commentary insight for the first content based on the textual feedback, when executed by the one or more processors, further direct the computing system to:
generate, by the formative feedback engine, a feedback input comprising the textual feedback and the first content;
transmit, by the formative feedback engine, the feedback input to a content generator; and
receive, by the formative feedback engine, the commentary insight from the content generator, wherein the commentary insight rephrases the textual feedback based on the first content.
8. The method of claim 7, wherein generating, by the formative feedback engine, the commentary insight for the first content based on the textual feedback and the first content comprises:
generating, by the formative feedback engine, a feedback input comprising the textual feedback and the first content;
providing, by the formative feedback engine, the feedback input to a content generator, wherein the content generator generates the commentary insight responsive to receiving the feedback input; and
receiving, by the formative feedback engine, the commentary insight from the content generator.
This reads on a human (e.g., mentally and/or using pen and paper):
Writing down feedback along with the content/text;
Showing said feedback/content to the second human;
The second human rephrasing said feedback (only claim 2).
Additional limitations are present: content generator. Same analysis applied to independent claims, above, apply.
With respect to claims 3, 9, and 17, the claim(s) recite:
3. The system of claim 1, wherein the program instructions, when executed by the one or more processors, further direct the computing system to:
parse, by the formative feedback engine, the written assessment to identify one or more content issues; and
generate, by the formative feedback engine, a second commentary insight based on the one or more content issues.
9. The method of claim 7, wherein the method further comprises:
analyzing, by the formative feedback engine, the written assessment for a content issue, wherein the content issue comprises one or more of a factual error, a wording error, or a concept contradiction;
determining, by the formative feedback engine, that second content within the written assessment comprises the content issue;
generating, by the formative feedback engine, a second commentary insight identifying the content issue; and
providing, by the formative feedback engine, the second commentary insight as associated with the second content within the written assessment.
17. The computer readable storage media of claim 15, wherein the written assessment comprises second content, and the processor-executable instructions cause the one or more processors to further execute processor-executable instructions stored in the computer readable storage media to:
determine, by the formative feedback engine, that the second content comprises a content issue; and
generate, by the formative feedback engine, a second commentary insight based on the content issue, wherein the second commentary insight identifies the content issue within the second content.
This reads on a human (e.g., mentally and/or using pen and paper):
Parse/segmenting/analyzing the content/text to identify any issues (e.g., factual/wording errors);
Determining if there are any issues in the content/text (only claims 9 and 17);
Generating a second comment or feedback based on the issues;
Writing down or providing said second comment or feedback (claim 9).
No additional limitations are present.
With respect to claim 4, the claim(s) recite:
4. The system of claim 1, wherein the program instructions to provide, by the formative feedback engine, the first commentary insight for the written assessment to the client device, when executed by the one or more processors, further direct the computing system to:
display, by the formative feedback engine, the first commentary insight on the written assessment within visual proximity to the first content.
This reads on a human (e.g., mentally and/or using pen and paper):
Writing down the feedback close to the content for display purposes.
No additional limitations are present.
With respect to claim 5, the claim(s) recite:
5. The system of claim 1, wherein the program instructions, when executed by the one or more processors, further direct the computing system to:
associate, by the formative feedback engine, the semantic representation of the first content with the first commentary insight; and
index, by the formative feedback engine, the semantic representation and the first commentary insight in a semantic store.
This reads on a human (e.g., mentally and/or using pen and paper):
Associate semantic representation of the content/text to the feedback; and
Use a predetermined set of rules to index or assign a value to the semantic representation and the feedback.
No additional limitations are present.
With respect to claims 6 and 14, the claim(s) recite:
6 and 14. The system of claim 1, wherein the program instructions, when executed by the one or more processors, further direct the computing system to: / The method of claim 7, the method further comprising:
receive/receiving, by the formative feedback engine, a modification to the first commentary insight from the client device;
generate/generating, by the formative feedback engine, a modified commentary insight for the first content based on the modification; and
provide/providing, by the formative feedback engine, the modified commentary insight as associated with the first content within the written assessment.
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving from another human a modification to the feedback;
Generating or writing down a modified feedback/comment for the content based on the modification; and
Showing said written modified feedback with the content/text.
No additional limitations are present.
With respect to claims 10 and 19, the claim(s) recite:
10. The method of claim 7, wherein determining, by the formative feedback engine, the textual feedback from the semantic store based on the semantic representation of the first content comprises:
determining, by the formative feedback engine, a plurality of semantically equivalent content based on the semantic representation;
determining, by the formative feedback engine, a subset of semantically equivalent content from the plurality of semantically equivalent content based on the client device;
ranking, by the formative feedback engine, the subset of semantically equivalent content by degree of similarity to the semantic representation of the first content;
determining, by the formative feedback engine, a first semantically equivalent content based on the ranking; and
determining, by the formative feedback engine, the textual feedback associated with the first semantically equivalent content.
19. The computer readable storage media of claim 15, wherein the processor-executable instructions to identify, by the formative feedback engine, the first textual feedback based on the comparison of the first semantic representation to the semantic store cause the one or more processors to further execute processor-executable instructions stored in the computer readable storage media to:
determine, by the formative feedback engine, a client device associated with the writing assessment;
determine, by the formative feedback engine, a plurality of semantically equivalent content to the first semantic representation within the semantic store;
determine, by the formative feedback engine, a plurality of textual feedback based on the plurality of semantically equivalent content, wherein each of the textual feedback within the plurality of textual feedback corresponds to a respective semantically equivalent content of the plurality of semantically equivalent content;
determine, by the formative feedback engine, a subset of textual feedback from the plurality of textual feedback based on the client device;
rank, by the formative feedback engine, the subset of textual feedback by degree of similarity between a subset of semantically equivalent content to the first semantic representation, wherein the subset of semantically equivalent content corresponds to the subset of textual feedback;
determine, by the formative feedback engine, a first semantically equivalent content based on the ranking; and
determine, by the formative feedback engine, the textual feedback as associated with the first semantically equivalent content.
This reads on a human (e.g., mentally and/or using pen and paper):
Determining where/who the text is coming from (only claim 19);
Determining semantically equivalent content/text within a data store (e.g., book) (only claim 19);
Determining a plurality of semantically equivalent (e.g., synonyms) content/text based on the semantic representation;
Determining a plurality of semantically equivalent (e.g., synonyms) content/text based on who it is coming from;
Ranking the semantically equivalent content/text with respect to the semantic representation of content/text;
Choosing the semantically equivalent content/text based on the ranking;
Determining the textual feedback associated with the content/text(s).
No additional limitations are present.
With respect to claim 11, the claim(s) recite:
11. The method of claim 7, wherein generating, by the formative feedback engine, a commentary insight for the first content based on the textual feedback and the first content further comprises:
determining, by the formative feedback engine, a writing style associated with the client device; and
generating, by the formative feedback engine, the commentary insight in the writing style of the client device.
This reads on a human (e.g., mentally and/or using pen and paper):
Determining a writing style associated to another human; and
Generating feedback using said writing style.
No additional limitations are present.
With respect to claims 12 and 20, the claim(s) recite:
12. The method of claim 7, wherein querying, by the formative feedback engine, the semantic store based on the semantic representation of the first content and determining, by the formative feedback engine, the textual feedback from the semantic store based on the semantic representation of the first content comprises:
performing, by the formative feedback engine, a Retrieval-Augmented Generation (RAG) operation for the semantic representation of the first content using the semantic store; and
determining, by the formative feedback engine, the textual feedback from the RAG operation.
20. The computer readable storage media of claim 15, wherein the processor-executable instructions to identify, by the formative feedback engine, the first textual feedback based on the comparison of the first semantic representation to the semantic store cause the one or more processors to further execute processor-executable instructions stored in the computer readable storage media to:
perform, by the formative feedback engine, a Retrieval-Augmented Generation (RAG) operation for the first semantic representation using the semantic store; and
determine, by the formative feedback engine, the first textual feedback from the RAG operation.
This reads on a human (e.g., mentally and/or using pen and paper):
Using a predetermined set of rules to perform the semantic representation of the content/text using the data store (e.g., book); and
Determine feedback.
No additional limitations are present.
With respect to claim 13, the claim(s) recite:
13. The method of claim 7, wherein the method further comprises:
determining, by the formative feedback engine, a plurality of commentary insights associated with the written assessment, wherein each of the commentary insights corresponds to respective content within the written assessment;
associating, by the formative feedback engine, each of the commentary insights with the respective content; and
indexing, by the formative feedback engine, the plurality of commentary insights and the respective content in the semantic store.
This reads on a human (e.g., mentally and/or using pen and paper):
Determining comments/feedback insights associated with text;
Associating said comments/feedback insights with the content in the data store (e.g., book).
No additional limitations are present.
With respect to claim 16, the claim(s) recite:
16. The computer readable storage media of claim 15, wherein the processor-executable instructions to identify, by the formative feedback engine, the first textual feedback based on the comparison of the first semantic representation to the semantic store cause the one or more processors to further execute processor-executable instructions stored in the computer readable storage media to:
determine, by the formative feedback engine, semantically equivalent content to the first content within the semantic store; and
determine, by the formative feedback engine, the first textual feedback as associated with the semantically equivalent content.
This reads on a human (e.g., mentally and/or using pen and paper):
Determining semantically equivalent content within a data store (e.g., book); and
Determining the feedback is associated with the semantically equivalent content/text.
No additional limitations are present.
With respect to claim 18, the claim(s) recite:
18. The computer readable storage media of claim 15, wherein the processor-executable instructions cause the one or more processors to further execute processor-executable instructions stored in the computer readable storage media to:
determine, by the formative feedback engine, completion of the written assessment, wherein upon completion the written assessment comprises a plurality of commentary insights; and
generate, by the formative feedback engine, a summary of the commentary insights for the written assessment.
This reads on a human (e.g., mentally and/or using pen and paper):
Determining completion of text, wherein the completion comprises plurality of comments/feedback; and
Generating a summary of the comments/feedback.
No additional limitations are present.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 7, and 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Walsh et al. (US 20250036880 A1) and further in view of Chorakhalikar et al. (US 20250053818 A1).
As to independent claim 1, Walsh et al. teaches:
1. A system (see ¶ [0004]: “In an aspect, embodiments described herein provide a system for natural language processing for quality assurance and automatic response assessment for constructed-response tests…”) comprising:
one or more computer readable storage media (see ¶ [0103]: “… It should be appreciated that the use of such terms is deemed to represent one or more computing devices having at least one processor configured to execute software instructions stored on a computer readable tangible, non-transitory medium…”);
one or more processors operatively coupled with the one or more computer readable storage media (see ¶ [0103] citation as in limitation above and further ¶ [0057]: “An electronic device 160 can have memory storing instructions for applications and one or more processors that execute the instructions and applications….”); and
an application comprising program instructions stored on the one or more computer readable storage media that, when executed by the one or more processors, direct a computing system (see ¶ [0103] citation as in limitation above and further ¶ [0057]: “An electronic device 160 can have memory storing instructions for applications and one or more processors that execute the instructions and applications….”) to at least:
determine, by a formative feedback engine, a written assessment associated with a first client device (see ¶ [0004]: “…The system has: a memory storing one or more generative models; a processor coupled to the memory programmed with executable instructions, the instructions including an interface for obtaining response data for a constructed-response test from a plurality of examinee electronic devices,…” and ¶ [0044]: “…Embodiments described herein use LLMs with rubric, automated scoring, narrative analysis, feedback, formative assessment, and so on…” );
determine, by the formative feedback engine, first content of the written assessment (see ¶ [0061]: “To ensure the correct interpretation and use of scores, system 100 implements a QA process using an NLP engine 122 for automatically evaluating the content, response data, internal structure, and response processes…” and ¶ [0074]: “At 306, system 100 extracts features from the combined response data. The system 100 can use multiple features, different types of features, different combinations of features, different semantic relationships, and so on. As an illustrative example, different features can include lemma type-token ratios, lexical token and type density, lexical overlap between adjacent sentences, semantic overlap between adjacent sentences, keyword similarity with scenario prompts, questions, rater guiding questions, and rubrics, semantic similarity with scenario prompts, questions, rater guiding questions, rubrics, pre-trained document embeddings, and response sentiment and subjectivity…”);
determine, by the formative feedback engine, textual feedback (see ¶ [0004]: “… wherein the processor compares the predicted rating data to the response data and generates quality assurance data based on results of the comparison using one or more models, wherein processor uses the one or more generative models to generate feedback data about the response data, wherein the quality assurance data comprises the feedback data, wherein the processor uses the predicted rating data and the feedback data for one or more of identification of improvement areas,…” and ¶ [0007]: “…generating feedback data for the response data using the predicted ratings; and transmitting the feedback data to an electronic device for display or storing the feedback data in the memory.”);
generate, by the formative feedback engine, a first commentary insight for the first content based on the textual feedback (see ¶ [0004 and 0007] citations as in limitation(s) above and further ¶ [0048]: “… System 100 can automatically generate feedback data about the response data using one or more generative models to generate text output that describes the evaluation of the response data to generate feedback content for display at an examinee device. The quality assurance data can include feedback data about the response data. For example, the feedback data can provide feedback content that identifies areas of improvement or provides recommendations for improvement or knowledge development…”); and
provide, by the formative feedback engine, the first commentary insight for the written assessment to a client device (see ¶ [0004 and 0007] citations as in limitation(s) above and further ¶ [0048]: “… System 100 can automatically generate feedback data about the response data using one or more generative models to generate text output that describes the evaluation of the response data to generate feedback content for display at an examinee device. The quality assurance data can include feedback data about the response data…”).
However, Walsh et al. does not explicitly teach, but Chorakhalikar et al. does teach:
vectorize, by the formative feedback engine, the first content to form a semantic representation of the first content (see ¶ [0067]: “In at least one embodiment, NLP clustering component 206 receives preprocessed text data from data preprocessing component 204. In at least one embodiment, NLP clustering component 206 generates vector representations of each string of text data 202, where vectorized representations may be used to identify semantic similarity between strings of text data…”);
identify, by the formative feedback engine, semantically equivalent content to the semantic representation of the first content (see ¶ [0067] citation as in limitation above and further ¶ [0067]: “…In at least one embodiment, a BERT NLP model is used to generate vectorized representations of text data 202 for identifying semantic similarity. In at least one embodiment, based on identified semantic similarity using vectorized representations of text data, said text data is grouped into clusters of semantically similar strings of text data…”);
determine, by the formative feedback engine, textual feedback associated with the semantically equivalent content (see ¶ [0067] citation as in limitation above and further ¶ [0067]: “[0067] In at least one embodiment, NLP clustering component 206 receives preprocessed text data from data preprocessing component 204. In at least one embodiment, NLP clustering component 206 generates vector representations of each string of text data 202, where vectorized representations may be used to identify semantic similarity between strings of text data. In at least one embodiment, a BERT NLP model is used to generate vectorized representations of text data 202 for identifying semantic similarity. In at least one embodiment, based on identified semantic similarity using vectorized representations of text data, said text data is grouped into clusters of semantically similar strings of text data. In at least one embodiment, where text data is of feedback related to an application, feedback containing “I cannot hear any sound” and “please fix headphone output issue” would be labeled by NLP clustering component to be semantically similar and identified as being part of a cluster, whereas “my game is lagging when I press fire” would be identified as not being semantically similar and not identified as part of a cluster related to sound issues.” and ¶ [0080]: “In at least one embodiment, NLP clustering component 706 receives preprocessed text data from data preprocessing component 704. In at least one embodiment, NLP clustering component 706 generates vector representations of each string of text data 702, where vectorized representations may be used to identify semantic similarity between strings of text data. In at least one embodiment, vectorized representation of text data by NLP clustering component 706 is proved to a dimensionality reduction component 708 to generate visualization plots. In at least one embodiment, dimensionality reduction component 708 uses t-Distributed Stochastic Neighbor Embedding (“t-SNE”) as an algorithm for dimensionality reduction and data visualization of text data 702 so to generate a cluster data visualization plot 710. In at least one embodiment, dimensionality reduction component 708 has a parameter toggling option between three levels to define data density of plot.”);
Walsh et al. and Chorakhalikar et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. to incorporate the teachings of Chorakhalikar et al. of vectorize, by the formative feedback engine, the first content to form a semantic representation of the first content; identify, by the formative feedback engine, semantically equivalent content to the semantic representation of the first content; and determine, by the formative feedback engine, textual feedback associated with the semantically equivalent content which provides the benefit of aiding in monitoring accuracy and performance of clustering of user feedback ([0062] of Chorakhalikar et al.).
As to independent claim 7, Walsh et al. teaches:
7. A method (see ¶ [0043]: “Embodiments described herein relate to systems and methods for natural language processing for quality assurance of rating data (i.e. data assessing how raters themselves are reviewing test-taker's responses) and assessing response data (i.e. the test-taker's input in response to questions) for constructed-response tests that can involve different types of assessments to measure non-cognitive skills (including but not limited to professionalism, situational awareness, social and emotional intelligence) using constructive open responses…”) comprising:
receiving, from a client device, a written assessment comprising first content (see ¶ [0004]: “…The system has: a memory storing one or more generative models; a processor coupled to the memory programmed with executable instructions, the instructions including an interface for obtaining response data for a constructed-response test from a plurality of examinee electronic devices,…” );
determining, by the formative feedback engine, textual feedback (see ¶ [0004]: “… wherein the processor compares the predicted rating data to the response data and generates quality assurance data based on results of the comparison using one or more models, wherein processor uses the one or more generative models to generate feedback data about the response data, wherein the quality assurance data comprises the feedback data, wherein the processor uses the predicted rating data and the feedback data for one or more of identification of improvement areas,…” and ¶ [0007]: “…generating feedback data for the response data using the predicted ratings; and transmitting the feedback data to an electronic device for display or storing the feedback data in the memory.”);
generating, by the formative feedback engine, a commentary insight for the first content based on the textual feedback and the first content (see ¶ [0004 and 0007] citations as in limitation(s) above and further ¶ [0048]: “… System 100 can automatically generate feedback data about the response data using one or more generative models to generate text output that describes the evaluation of the response data to generate feedback content for display at an examinee device. The quality assurance data can include feedback data about the response data. For example, the feedback data can provide feedback content that identifies areas of improvement or provides recommendations for improvement or knowledge development…”); and
providing, by the formative feedback engine, the commentary insight to the client device (see ¶ [0004 and 0007] citations as in limitation(s) above and further ¶ [0048]: “… System 100 can automatically generate feedback data about the response data using one or more generative models to generate text output that describes the evaluation of the response data to generate feedback content for display at an examinee device. The quality assurance data can include feedback data about the response data…”).
However, Walsh et al. does not explicitly teach, but Chorakhalikar et al. does teach:
generating, by a formative feedback engine, a semantic representation of the first content (see ¶ [0067]: “In at least one embodiment, NLP clustering component 206 receives preprocessed text data from data preprocessing component 204. In at least one embodiment, NLP clustering component 206 generates vector representations of each string of text data 202, where vectorized representations may be used to identify semantic similarity between strings of text data…”);
querying, by the formative feedback engine, a semantic store based on the semantic representation of the first content (see ¶ [0067] citation as in limitation above and further ¶ [0064]: “In at least one embodiment, clustering component 106 may include one or more NLP neural networks to cause text data stored in date store 104 to be clustered into semantically similar groupings. In at least one embodiment, clustering component 106 may contain Bidirectional Encoder Representations from Transformers (“BERT”) NLP model to cause text data to be clustered into semantically similar groups.”),
wherein the semantic store comprises a plurality of commentary insights associated with one or more previously evaluated written assessments (see ¶ [0064 and 0067] citations as in limitations above and further ¶ [0062]: “[0062] In at least one embodiment, raw text data is textual data provided by a user. In at least one embodiment, user feedback is entered through a software application in which a user may submit issues related to the software application. In at least one embodiment, raw text data may be free from semantic constraints and may be entered in a user's language of choice. In at least one embodiment, text submitted by a user via an application client or through a webpage forum may be passes through and accumulated into a user issue feedback data store. In at least one embodiment, to analyze raw textual unstructured and ungrouped data point, a clustering algorithm may be used to help identify underlying patterns in user feedback and identify common and new issues faced by users in a given time period. In at least one embodiment, a system may generate metrics to aid in monitoring accuracy and performance of clustering of user feedback.”);
determining, by the formative feedback engine, textual feedback from the semantic store based on the semantic representation of the first content (see ¶ [0062, 0064, and 0067] citation as in limitation above and further ¶ [0067]: “[0067] In at least one embodiment, NLP clustering component 206 receives preprocessed text data from data preprocessing component 204. In at least one embodiment, NLP clustering component 206 generates vector representations of each string of text data 202, where vectorized representations may be used to identify semantic similarity between strings of text data. In at least one embodiment, a BERT NLP model is used to generate vectorized representations of text data 202 for identifying semantic similarity. In at least one embodiment, based on identified semantic similarity using vectorized representations of text data, said text data is grouped into clusters of semantically similar strings of text data. In at least one embodiment, where text data is of feedback related to an application, feedback containing “I cannot hear any sound” and “please fix headphone output issue” would be labeled by NLP clustering component to be semantically similar and identified as being part of a cluster, whereas “my game is lagging when I press fire” would be identified as not being semantically similar and not identified as part of a cluster related to sound issues.” and ¶ [0080]: “In at least one embodiment, NLP clustering component 706 receives preprocessed text data from data preprocessing component 704. In at least one embodiment, NLP clustering component 706 generates vector representations of each string of text data 702, where vectorized representations may be used to identify semantic similarity between strings of text data. In at least one embodiment, vectorized representation of text data by NLP clustering component 706 is proved to a dimensionality reduction component 708 to generate visualization plots. In at least one embodiment, dimensionality reduction component 708 uses t-Distributed Stochastic Neighbor Embedding (“t-SNE”) as an algorithm for dimensionality reduction and data visualization of text data 702 so to generate a cluster data visualization plot 710. In at least one embodiment, dimensionality reduction component 708 has a parameter toggling option between three levels to define data density of plot.”);
Walsh et al. and Chorakhalikar et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. to incorporate the teachings of Chorakhalikar et al. of generating, by a formative feedback engine, a semantic representation of the first content; querying, by the formative feedback engine, a semantic store based on the semantic representation of the first content, wherein the semantic store comprises a plurality of commentary insights associated with one or more previously evaluated written assessments; determining, by the formative feedback engine, textual feedback from the semantic store based on the semantic representation of the first content which provides the benefit of aiding in monitoring accuracy and performance of clustering of user feedback ([0062] of Chorakhalikar et al.).
As to independent claim 15, Walsh et al. teaches:
15. A computer readable storage media comprising processor-executable instructions configured to cause one or more processors (see ¶ [0103]: “… It should be appreciated that the use of such terms is deemed to represent one or more computing devices having at least one processor configured to execute software instructions stored on a computer readable tangible, non-transitory medium…”) to:
determine, by a formative feedback engine, a written assessment comprising first content (see ¶ [0061]: “To ensure the correct interpretation and use of scores, system 100 implements a QA process using an NLP engine 122 for automatically evaluating the content, response data, internal structure, and response processes…” and ¶ [0074]: “At 306, system 100 extracts features from the combined response data. The system 100 can use multiple features, different types of features, different combinations of features, different semantic relationships, and so on. As an illustrative example, different features can include lemma type-token ratios, lexical token and type density, lexical overlap between adjacent sentences, semantic overlap between adjacent sentences, keyword similarity with scenario prompts, questions, rater guiding questions, and rubrics, semantic similarity with scenario prompts, questions, rater guiding questions, rubrics, pre-trained document embeddings, and response sentiment and subjectivity…”);
identify, by the formative feedback engine, a first textual feedback (see ¶ [0004]: “… wherein the processor compares the predicted rating data to the response data and generates quality assurance data based on results of the comparison using one or more models, wherein processor uses the one or more generative models to generate feedback data about the response data, wherein the quality assurance data comprises the feedback data, wherein the processor uses the predicted rating data and the feedback data for one or more of identification of improvement areas,…” and ¶ [0007]: “…generating feedback data for the response data using the predicted ratings; and transmitting the feedback data to an electronic device for display or storing the feedback data in the memory.”);
generate, by the formative feedback engine, a first commentary insight for the first content based on the first textual feedback (see ¶ [0004 and 0007] citations as in limitation(s) above and further ¶ [0048]: “… System 100 can automatically generate feedback data about the response data using one or more generative models to generate text output that describes the evaluation of the response data to generate feedback content for display at an examinee device. The quality assurance data can include feedback data about the response data. For example, the feedback data can provide feedback content that identifies areas of improvement or provides recommendations for improvement or knowledge development…”); and
provide, by the formative feedback engine, the first commentary insight as associated with the first content within the written assessment (see ¶ [0004 and 0007] citations as in limitation(s) above and further ¶ [0048]: “… System 100 can automatically generate feedback data about the response data using one or more generative models to generate text output that describes the evaluation of the response data to generate feedback content for display at an examinee device. The quality assurance data can include feedback data about the response data…”).
However, Walsh et al. does not explicitly teach, but Chorakhalikar et al. does teach:
vectorize, by a formative feedback engine, the first content to form a first semantic representation of the first content (see ¶ [0067]: “In at least one embodiment, NLP clustering component 206 receives preprocessed text data from data preprocessing component 204. In at least one embodiment, NLP clustering component 206 generates vector representations of each string of text data 202, where vectorized representations may be used to identify semantic similarity between strings of text data…”);
compare, by the formative feedback engine, the first semantic representation to a semantic store (see ¶ [0067] citation as in limitation above and further ¶ [0067]: “…In at least one embodiment, a BERT NLP model is used to generate vectorized representations of text data 202 for identifying semantic similarity. In at least one embodiment, based on identified semantic similarity using vectorized representations of text data, said text data is grouped into clusters of semantically similar strings of text data…”);
identify, by the formative feedback engine, a first textual feedback based on the comparison of the first semantic representation to the semantic store (see ¶ [0067] citation as in limitation above and further ¶ [0067]: “[0067] In at least one embodiment, NLP clustering component 206 receives preprocessed text data from data preprocessing component 204. In at least one embodiment, NLP clustering component 206 generates vector representations of each string of text data 202, where vectorized representations may be used to identify semantic similarity between strings of text data. In at least one embodiment, a BERT NLP model is used to generate vectorized representations of text data 202 for identifying semantic similarity. In at least one embodiment, based on identified semantic similarity using vectorized representations of text data, said text data is grouped into clusters of semantically similar strings of text data. In at least one embodiment, where text data is of feedback related to an application, feedback containing “I cannot hear any sound” and “please fix headphone output issue” would be labeled by NLP clustering component to be semantically similar and identified as being part of a cluster, whereas “my game is lagging when I press fire” would be identified as not being semantically similar and not identified as part of a cluster related to sound issues.” and ¶ [0080]: “In at least one embodiment, NLP clustering component 706 receives preprocessed text data from data preprocessing component 704. In at least one embodiment, NLP clustering component 706 generates vector representations of each string of text data 702, where vectorized representations may be used to identify semantic similarity between strings of text data. In at least one embodiment, vectorized representation of text data by NLP clustering component 706 is proved to a dimensionality reduction component 708 to generate visualization plots. In at least one embodiment, dimensionality reduction component 708 uses t-Distributed Stochastic Neighbor Embedding (“t-SNE”) as an algorithm for dimensionality reduction and data visualization of text data 702 so to generate a cluster data visualization plot 710. In at least one embodiment, dimensionality reduction component 708 has a parameter toggling option between three levels to define data density of plot.”);
Walsh et al. and Chorakhalikar et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. to incorporate the teachings of Chorakhalikar et al. of vectorize, by a formative feedback engine, the first content to form a first semantic representation of the first content; compare, by the formative feedback engine, the first semantic representation to a semantic store; identify, by the formative feedback engine, a first textual feedback based on the comparison of the first semantic representation to the semantic store which provides the benefit of aiding in monitoring accuracy and performance of clustering of user feedback ([0062] of Chorakhalikar et al.).
Regarding claim 16, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claims 1 and 7, above.
Chorakhalikar et al. further teaches:
16. The computer readable storage media of claim 15, wherein the processor-executable instructions to identify, by the formative feedback engine, the first textual feedback based on the comparison of the first semantic representation to the semantic store (see ¶ [0067 and 0080] citation as in claim 15 above.) cause the one or more processors to further execute processor-executable instructions stored in the computer readable storage media to:
determine, by the formative feedback engine, semantically equivalent content to the first content within the semantic store (see ¶ [0068]: “The system 100 may also split the extracted text content into individual sentences. For each sentence, the system 100 may generate an embedding (a numerical representation that captures the semantic essence of the sentence). The system 100 may store such embeddings in an indexed vector database, such as pg_vector, which facilitates efficient and precise future searches and retrieval of data. This indexed storage of sentence embeddings allows for quick and accurate comparisons and searches within the text content, enhancing the system 100's ability to retrieve specific information and analyze the relevance of artifacts in future queries.”); and
determine, by the formative feedback engine, the first textual feedback as associated with the semantically equivalent content (see ¶ [0067]: “[0067] In at least one embodiment, NLP clustering component 206 receives preprocessed text data from data preprocessing component 204. In at least one embodiment, NLP clustering component 206 generates vector representations of each string of text data 202, where vectorized representations may be used to identify semantic similarity between strings of text data. In at least one embodiment, a BERT NLP model is used to generate vectorized representations of text data 202 for identifying semantic similarity. In at least one embodiment, based on identified semantic similarity using vectorized representations of text data, said text data is grouped into clusters of semantically similar strings of text data. In at least one embodiment, where text data is of feedback related to an application, feedback containing “I cannot hear any sound” and “please fix headphone output issue” would be labeled by NLP clustering component to be semantically similar and identified as being part of a cluster, whereas “my game is lagging when I press fire” would be identified as not being semantically similar and not identified as part of a cluster related to sound issues.” and ¶ [0080]: “In at least one embodiment, NLP clustering component 706 receives preprocessed text data from data preprocessing component 704. In at least one embodiment, NLP clustering component 706 generates vector representations of each string of text data 702, where vectorized representations may be used to identify semantic similarity between strings of text data. In at least one embodiment, vectorized representation of text data by NLP clustering component 706 is proved to a dimensionality reduction component 708 to generate visualization plots. In at least one embodiment, dimensionality reduction component 708 uses t-Distributed Stochastic Neighbor Embedding (“t-SNE”) as an algorithm for dimensionality reduction and data visualization of text data 702 so to generate a cluster data visualization plot 710. In at least one embodiment, dimensionality reduction component 708 has a parameter toggling option between three levels to define data density of plot.”).
Walsh et al. and Chorakhalikar et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. to incorporate the teachings of Chorakhalikar et al. of determine, by the formative feedback engine, semantically equivalent content to the first content within the semantic store; and determine, by the formative feedback engine, the first textual feedback as associated with the semantically equivalent content which provides the benefit of aiding in monitoring accuracy and performance of clustering of user feedback ([0062] of Chorakhalikar et al.).
Claims 2 and 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Walsh et al. (US 20250036880 A1) in view of Chorakhalikar et al. (US 20250053818 A1) as applied to claim 1 above, and further in view of Zhang et al. ("eRevise: Using natural language processing to provide formative feedback on text evidence usage in student writing." Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.).
Regarding claim 2, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claim 1, above.
Walsh et al. further teaches:
2. The system of claim 1, wherein the program instructions to generate, by the formative feedback engine, the commentary insight for the first content based on the textual feedback (see ¶ [0004, 0007, and 0048] citations as in claim 1, above.), when executed by the one or more processors, further direct the computing system (see ¶ [0057 and 0103] citations as in claim 1, above.) to:
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but Zhang et al. does teach:
generate, by the formative feedback engine, a feedback input comprising the textual feedback and the first content (see Figure 2: A formative feedback screenshot of the eRevise system and ¶ 3 under section System Usage and Architecture (page 9620): “During the second class period, students login to eRevise and now revise their first drafts using eRevise. Figure 2 shows a screenshot illustrating revision guided by formative feedback. The left top box shows a student’s first draft. This helps students to recall their first drafts and eases revising (e.g., by allowing cutting and pasting). The right-hand side of the screen shows the feedback on the first draft that was automatically selected by the AWE system. The left bottom box shows where students create their second drafts, hope fully guided by the feedback displayed on the right.3”);
transmit, by the formative feedback engine, the feedback input to a content generator (see Figure 2: A formative feedback screenshot of the eRevise system and ¶ 3 under section System Usage and Architecture (pages 9620-9621) citations as in limitation above. More specifically see display feedback messages shown in Figure 2.); and
receive, by the formative feedback engine, the commentary insight from the content generator (see Figure 2: A formative feedback screenshot of the eRevise system and ¶ 3 under section System Usage and Architecture (page 9620-9621) citations as in limitation above. More specifically see display feedback messages shown in Figure 2.),
wherein the commentary insight rephrases the textual feedback based on the first content (see Figure 2: A formative feedback screenshot of the eRevise system and ¶ 3 under section System Usage and Architecture (page 9620-9621) citations as in limitation above. More specifically, see Figure 2: “For example, writing, “The school fee was a problem” is not specific enough. It is better to write, “Students could not attend school because they did not have enough money to pay the school fee.”).
PNG
media_image1.png
590
1186
media_image1.png
Greyscale
Walsh et al., Chorakhalikar et al., and Zhang et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of Zhang et al. of generate, by the formative feedback engine, a feedback input comprising the textual feedback and the first content; transmit, by the formative feedback engine, the feedback input to a content generator; and receive, by the formative feedback engine, the commentary insight from the content generator, wherein the commentary insight rephrases the textual feedback based on the first content which provides the benefit of improving the quality of text evidence usage in writing after students received formative feedback ([abstract] of Zhang et al.).
Regarding claim 8, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claim 7, above.
Walsh et al. further teaches:
8. The method of claim 7, wherein generating, by the formative feedback engine, the commentary insight for the first content based on the textual feedback and the first content (see ¶ [0004, 0007, and 0048] citations as in claim 1, above.) comprises:
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but Zhang et al. does teach:
generating, by the formative feedback engine, a feedback input comprising the textual feedback and the first content (see Figure 2: A formative feedback screenshot of the eRevise system and ¶ 3 under section System Usage and Architecture (pages 9620-9621) citations as in claim 2, above.);
providing, by the formative feedback engine, the feedback input to a content generator, wherein the content generator generates the commentary insight responsive to receiving the feedback input (see Figure 2: A formative feedback screenshot of the eRevise system and ¶ 3 under section System Usage and Architecture (pages 9620-9621) citations as in claim 2, above. More specifically, see Figure 2: “For example, writing, “The school fee was a problem” is not specific enough. It is better to write, “Students could not attend school because they did not have enough money to pay the school fee.”); and
receiving, by the formative feedback engine, the commentary insight from the content generator (see Figure 2: A formative feedback screenshot of the eRevise system and ¶ 3 under section System Usage and Architecture (pages 9620-9621) citations as in claim 2, above. More specifically see display feedback messages shown in Figure 2.).
Walsh et al., Chorakhalikar et al., and Zhang et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of Zhang et al. of generating, by the formative feedback engine, a feedback input comprising the textual feedback and the first content; providing, by the formative feedback engine, the feedback input to a content generator, wherein the content generator generates the commentary insight responsive to receiving the feedback input; and receiving, by the formative feedback engine, the commentary insight from the content generator which provides the benefit of improving the quality of text evidence usage in writing after students received formative feedback ([abstract] of Zhang et al.).
Claims 3-4, 9, and 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Walsh et al. (US 20250036880 A1) in view of Chorakhalikar et al. (US 20250053818 A1) as applied to claim 1 above, and further in view of Escalante et al. (Escalante, Juan, Austin Pack, and Alex Barrett. "AI-generated feedback on writing: Insights into efficacy and ENL student preference." International Journal of Educational Technology in Higher Education 20.1 (2023): 57.)
Regarding claim 3, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claim 1, above.
Walsh et al. further teaches:
3. The system of claim 1, wherein the program instructions, when executed by the one or more processors (see ¶ [0057 and 0103] citation as in claim 1, above.), further direct the computing system to:
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but Escalante et al. does teach:
parse, by the formative feedback engine, the written assessment to identify one or more content issues (see Feedback on the grammatical accuracy table under Appendix C (page 18): Sentence row with sentences parsed from the paragraph under ¶ 1 under Appendix C (Original student writing)); and
generate, by the formative feedback engine, a second commentary insight based on the one or more content issues (see Feedback on the grammatical accuracy table under Appendix C (page 18): Error type, Description and Suggestion rows associated to the paragraph under ¶ 1 under Appendix C (Original student writing)).
PNG
media_image2.png
534
624
media_image2.png
Greyscale
Walsh et al., Chorakhalikar et al., and Escalante et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of Escalante et al. of parse, by the formative feedback engine, the written assessment to identify one or more content issues; and generate, by the formative feedback engine, a second commentary insight based on the one or more content issues which provides the benefit of being useful for improving student’s writing ([conclusion] of Escalante et al.).
Regarding claim 9, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claim 7, above.
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but Escalante et al. does teach:
9. The method of claim 7, wherein the method further comprises:
analyzing, by the formative feedback engine, the written assessment for a content issue (see Feedback on the grammatical accuracy table under Appendix C (page 18): Sentence row with sentences parsed from the paragraph under ¶ 1 under Appendix C (Original student writing) and the Description and Suggestion rows),
wherein the content issue comprises one or more of a factual error, a wording error, or a concept contradiction (see Feedback on the grammatical accuracy table under Appendix C (page 18): Sentence row with sentences parsed from the paragraph under ¶ 1 under Appendix C (Original student writing) and the Error type: word form, awkward phrasing, verb tense, etc.);
determining, by the formative feedback engine, that second content within the written assessment comprises the content issue (see Feedback on the grammatical accuracy table under Appendix C (page 18): Sentence row with sentences parsed from the paragraph under ¶ 1 under Appendix C (Original student writing) and the Error type / Description rows);
generating, by the formative feedback engine, a second commentary insight identifying the content issue (see Feedback on the grammatical accuracy table under Appendix C (page 18): Sentence row with sentences parsed from the paragraph under ¶ 1 under Appendix C (Original student writing) and the Error type / Description rows); and
providing, by the formative feedback engine, the second commentary insight as associated with the second content within the written assessment (see Feedback on the grammatical accuracy table under Appendix C (page 18): Sentence row with sentences parsed from the paragraph under ¶ 1 under Appendix C (Original student writing) and the Error type / Description rows).
Walsh et al., Chorakhalikar et al., and Escalante et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of Escalante et al. of analyzing, by the formative feedback engine, the written assessment for a content issue, wherein the content issue comprises one or more of a factual error, a wording error, or a concept contradiction; determining, by the formative feedback engine, that second content within the written assessment comprises the content issue; generating, by the formative feedback engine, a second commentary insight identifying the content issue; and providing, by the formative feedback engine, the second commentary insight as associated with the second content within the written assessment which provides the benefit of being useful for improving student’s writing ([conclusion] of Escalante et al.).
Regarding claim 4, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claim 1, above.
Walsh et al. further teaches:
4. The system of claim 1, wherein the program instructions to provide, by the formative feedback engine, the first commentary insight for the written assessment to the client device (see ¶ [0004 and 0007] citations as in claim 1, above.), when executed by the one or more processors, further direct the computing system (see ¶ [0103] citation as in claim 1, above.) to:
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but Escalante et al. does teach:
display, by the formative feedback engine, the first commentary insight on the written assessment within visual proximity to the first content (see Feedback on the grammatical accuracy table under Appendix C (page 18) and ¶ 4 of Instruments sub-section under Method section (page 7): “…The AI was instructed to put the feedback on grammatical accuracy into a table that organized the following elements: the sentence where the error in the writing is found, the error type, a description of what this kind of error is, and suggestions as to how to address the error…” and ¶ 7 of Appendix B: “Using simple language, comment on the grammatical accuracy of the language of the writing, such as spelling, capitalization, punctuation, singular and plural nouns, verb tense, subject verb agreement, word form, awkward phrasing, prepositions, articles, and sentence fragments and run-ons. Put the feedback on grammatical accuracy in a table with four columns. The first column includes the sentence where the error in the student’s writing is found. The second column is the error type. The third column is a description of what this kind of error is. The fourth column is a suggestion on how to address the error”).
Walsh et al., Chorakhalikar et al., and Escalante et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of Escalante et al. of display, by the formative feedback engine, the first commentary insight on the written assessment within visual proximity to the first content which provides the benefit of being useful for improving student’s writing ([conclusion] of Escalante et al.).
Regarding claim 17, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claim 15, above.
Walsh et al. further teaches:
17. The computer readable storage media of claim 15, wherein the written assessment comprises second content (see ¶ [0061 and 0074] citations as in claim 1, above.), and the processor-executable instructions cause the one or more processors to further execute processor-executable instructions stored in the computer readable storage media (see ¶ [0057 and 0103] citations as in claim 1, above) to:
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but Escalante et al. does teach:
determine, by the formative feedback engine, that the second content comprises a content issue (see Feedback on the grammatical accuracy table under Appendix C (page 18): Sentence row with sentences parsed from the paragraph under ¶ 1 under Appendix C (Original student writing)); and
generate, by the formative feedback engine, a second commentary insight based on the content issue, wherein the second commentary insight identifies the content issue within the second content (see Feedback on the grammatical accuracy table under Appendix C (page 18): Error type, Description and Suggestion rows associated to the paragraph under ¶ 1 under Appendix C (Original student writing)).
Walsh et al., Chorakhalikar et al., and Escalante et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of Escalante et al. of determine, by the formative feedback engine, that the second content comprises a content issue; and generate, by the formative feedback engine, a second commentary insight based on the content issue, wherein the second commentary insight identifies the content issue within the second content which provides the benefit of being useful for improving student’s writing ([conclusion] of Escalante et al.).
Regarding claim 18, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claim 15, above.
Walsh et al. further teaches:
18. The computer readable storage media of claim 15, wherein the processor-executable instructions cause the one or more processors to further execute processor-executable instructions stored in the computer readable storage media (see ¶ [0057 and 0103] citations as in claim 1, above) to:
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but Escalante et al. does teach:
determine, by the formative feedback engine, completion of the written assessment, wherein upon completion the written assessment comprises a plurality of commentary insights (formative feedback engine, that the second content comprises a content issue (see Feedback on the grammatical accuracy table under Appendix C (page 18): Sentence row with sentences parsed from the paragraph under ¶ 1 under Appendix C (Original student writing)); and
generate, by the formative feedback engine, a summary of the commentary insights for the written assessment (see Feedback on the grammatical accuracy table under Appendix C (page 18): Error type, Description and Suggestion rows associated to the paragraph under ¶ 1 under Appendix C (Original student writing)).
Walsh et al., Chorakhalikar et al., and Escalante et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of Escalante et al. of determine, by the formative feedback engine, completion of the written assessment, wherein upon completion the written assessment comprises a plurality of commentary insights formative feedback engine, that the second content comprises a content issue; and generate, by the formative feedback engine, a summary of the commentary insights for the written assessment which provides the benefit of being useful for improving student’s writing ([conclusion] of Escalante et al.).
Claim 5 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Walsh et al. (US 20250036880 A1) in view of Chorakhalikar et al. (US 20250053818 A1) as applied to claims 1 and 7 above, and further in view of Nichols (US 20260010528 A1).
Regarding claim 5, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claim 1, above.
Walsh et al. further teaches:
5. The system of claim 1, wherein the program instructions, when executed by the one or more processors, further direct the computing system (see ¶ [0057 and 0103] citations as in claim 1, above) to:
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but Nichols does teach:
associate, by the formative feedback engine, the semantic representation of the first content with the first commentary insight (see ¶ [0068]: “The system 100 may also split the extracted text content into individual sentences. For each sentence, the system 100 may generate an embedding (a numerical representation that captures the semantic essence of the sentence). The system 100 may store such embeddings in an indexed vector database, such as pg_vector, which facilitates efficient and precise future searches and retrieval of data. This indexed storage of sentence embeddings allows for quick and accurate comparisons and searches within the text content, enhancing the system 100's ability to retrieve specific information and analyze the relevance of artifacts in future queries.”); and
index, by the formative feedback engine, the semantic representation and the first commentary insight in a semantic store (see ¶ [0068] citation as in limitation above: “…The system 100 may store such embeddings in an indexed vector database, such as pg_vector, which facilitates efficient and precise future searches and retrieval of data…”).
Walsh et al., Chorakhalikar et al., and Nichols are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of Nichols of associate, by the formative feedback engine, the semantic representation of the first content with the first commentary insight; and index, by the formative feedback engine, the semantic representation and the first commentary insight in a semantic store which provides the benefit of allowing for quick and accurate comparisons and searches within the text content, enhancing the system’s ability to retrieve specific information and analyze the relevance of artifacts in future queries ([0068] of Nichols).
Regarding claim 13, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claim 7, above.
Walsh et al. further teaches:
13. The method of claim 7, wherein the method further comprises:
determining, by the formative feedback engine, a plurality of commentary insights associated with the written assessment, wherein each of the commentary insights corresponds to respective content within the written assessment (see ¶ [0004 and 0007] citations as in limitation(s) above and further ¶ [0048]: “… System 100 can automatically generate feedback data about the response data using one or more generative models to generate text output that describes the evaluation of the response data to generate feedback content for display at an examinee device. The quality assurance data can include feedback data about the response data. For example, the feedback data can provide feedback content that identifies areas of improvement or provides recommendations for improvement or knowledge development…”);
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but Nichols does teach:
associating, by the formative feedback engine, each of the commentary insights with the respective content (see ¶ [0068]: “The system 100 may also split the extracted text content into individual sentences. For each sentence, the system 100 may generate an embedding (a numerical representation that captures the semantic essence of the sentence). The system 100 may store such embeddings in an indexed vector database, such as pg_vector, which facilitates efficient and precise future searches and retrieval of data. This indexed storage of sentence embeddings allows for quick and accurate comparisons and searches within the text content, enhancing the system 100's ability to retrieve specific information and analyze the relevance of artifacts in future queries.”); and
indexing, by the formative feedback engine, the plurality of commentary insights and the respective content in the semantic store (see ¶ [0068] citation as in limitation above: “…The system 100 may store such embeddings in an indexed vector database, such as pg_vector, which facilitates efficient and precise future searches and retrieval of data…”).
Walsh et al., Chorakhalikar et al., and Nichols are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of Nichols of associating, by the formative feedback engine, each of the commentary insights with the respective content; and indexing, by the formative feedback engine, the plurality of commentary insights and the respective content in the semantic store which provides the benefit of allowing for quick and accurate comparisons and searches within the text content, enhancing the system’s ability to retrieve specific information and analyze the relevance of artifacts in future queries ([0068] of Nichols).
Claims 6 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Walsh et al. (US 20250036880 A1) in view of Chorakhalikar et al. (US 20250053818 A1) as applied to claims 1 and 7 above, and further in view of Hantz et al. (US 20250252863 A1).
Regarding claims 6 and 14, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claims 1 and 7, above.
Walsh et al. further teaches:
6 and 14. The system of claim 1, wherein the program instructions, when executed by the one or more processors, further direct the computing system (see ¶ [0057 and 0103] citations as in claim 1, above.) to: / The method of claim 7, the method (see ¶ [0043] citation as in claim 7, above.) further comprising:
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but Hantz et al. does teach:
receive/receiving, by the formative feedback engine, a modification to the first commentary insight from the client device (see ¶ [0018]: “ In the computer system 100 shown by way of example in FIG. 1, the human tutor 102 interacts with the tutor computing device 104 to view the essay, to review AI-generated feedback and to accept, reject or edit (modify) the AI-generated feedback, or to add a new comment, as will be explained in greater detail below...”);
generate/generating, by the formative feedback engine, a modified commentary insight for the first content based on the modification (see ¶ [0018] citation as in limitation above and further ¶ [0020]: “…The tutor computing device 104 receives input from the human tutor 102 via a user interface to accept, reject or edit the AI-generated suggested written corrective feedback. The human-reviewed and/or human-modified AI-generated feedback thus constitutes HITL-AI written corrective feedback, i.e. AI-generated feedback that is refined by human-in-the-loop oversight or supervision. Optionally, the HITL-AI written corrective feedback may be supplemented by or include solely human comments from the human tutor. The tutor computing device 104, in response to a command or instruction from the human tutor 102 via the user interface, communicates the HITL-AI written corrective feedback to the one or more tutoring platform servers 150. The one or more tutoring platform servers then communicates the HITL-AI written corrective feedback to the student computing device 114 for viewing by the student 112.”); and
provide/providing, by the formative feedback engine, the modified commentary insight as associated with the first content within the written assessment (see ¶ [0018 and 0020] citations as in limitation above and further ¶ [0032]: “…Finally, at step 340 of the method, the HITL-AI written corrective feedback is communicated to the student.”).
Walsh et al., Chorakhalikar et al., and Hantz et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of Hantz et al. of receive/receiving, by the formative feedback engine, a modification to the first commentary insight from the client device; generate/generating, by the formative feedback engine, a modified commentary insight for the first content based on the modification; and provide/providing, by the formative feedback engine, the modified commentary insight as associated with the first content within the written assessment which provides the benefit of significantly improves the overall quality of solely human-written comments ([0034] of Hantz et al.).
Claims 10 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Walsh et al. (US 20250036880 A1) in view of Chorakhalikar et al. (US 20250053818 A1) as applied to claims 1 and 7 above, and further in view of Fink et al. (US 20150269153 A1).
Regarding claim 10, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claims 1 and 7, above.
Chorakhalikar et al. further teaches:
10. The method of claim 7, wherein determining, by the formative feedback engine, the textual feedback from the semantic store based on the semantic representation of the first content (see ¶ [0062, 0064, and 0067] citations as in claims 1 and 7 above.) comprises:
determining, by the formative feedback engine, a plurality of semantically equivalent content based on the semantic representation (see ¶ [0061 and 0065]: “[0061] In at least one embodiment, an output of a data clustering algorithm is examined and a score to indicate how well a clustering algorithm performed is generated. In at least one embodiment, an evaluation metric is a score generated that identifies how well clusters are grouped together based on semantic similarity within and between said clusters. In at least one embodiment, an evaluation metric is a score is generated by passing output clusters, which consist of raw text and inferenced cluster grouping, to a neural network, which then transforms raw text into vectors that represent semantic meaning of each string of text… [0065] …In at least one embodiment, clustering performance evaluator 108 may output performance metrics to administrator client 114. In at least one embodiment, administrator 116 may update parameters of clustering component 106 through administrator client 114 based on performance metrics generated by clustering performance evaluator 108 and provided to administration client 114.”);
determining, by the formative feedback engine, a subset of semantically equivalent content from the plurality of semantically equivalent content based on the client device (see ¶ [0061 and 0065] citations as in limitation above: “[0065] …In at least one embodiment, clustering performance evaluator 108 may output performance metrics to administrator client 114. In at least one embodiment, administrator 116 may update parameters of clustering component 106 through administrator client 114 based on performance metrics generated by clustering performance evaluator 108 and provided to administration client 114.””); and
determining, by the formative feedback engine, the textual feedback associated with the first semantically equivalent content (see ¶ [0067]: “[0067] In at least one embodiment, NLP clustering component 206 receives preprocessed text data from data preprocessing component 204. In at least one embodiment, NLP clustering component 206 generates vector representations of each string of text data 202, where vectorized representations may be used to identify semantic similarity between strings of text data. In at least one embodiment, a BERT NLP model is used to generate vectorized representations of text data 202 for identifying semantic similarity. In at least one embodiment, based on identified semantic similarity using vectorized representations of text data, said text data is grouped into clusters of semantically similar strings of text data. In at least one embodiment, where text data is of feedback related to an application, feedback containing “I cannot hear any sound” and “please fix headphone output issue” would be labeled by NLP clustering component to be semantically similar and identified as being part of a cluster, whereas “my game is lagging when I press fire” would be identified as not being semantically similar and not identified as part of a cluster related to sound issues.” and ¶ [0080]: “In at least one embodiment, NLP clustering component 706 receives preprocessed text data from data preprocessing component 704. In at least one embodiment, NLP clustering component 706 generates vector representations of each string of text data 702, where vectorized representations may be used to identify semantic similarity between strings of text data. In at least one embodiment, vectorized representation of text data by NLP clustering component 706 is proved to a dimensionality reduction component 708 to generate visualization plots. In at least one embodiment, dimensionality reduction component 708 uses t-Distributed Stochastic Neighbor Embedding (“t-SNE”) as an algorithm for dimensionality reduction and data visualization of text data 702 so to generate a cluster data visualization plot 710. In at least one embodiment, dimensionality reduction component 708 has a parameter toggling option between three levels to define data density of plot.”).
Walsh et al. and Chorakhalikar et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. to incorporate the teachings of Chorakhalikar et al. determining, by the formative feedback engine, a plurality of semantically equivalent content based on the semantic representation; determining, by the formative feedback engine, a subset of semantically equivalent content from the plurality of semantically equivalent content based on the client device; and determining, by the formative feedback engine, the textual feedback associated with the first semantically equivalent content which provides the benefit of aiding in monitoring accuracy and performance of clustering of user feedback ([0062] of Chorakhalikar et al.).
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but Fink et al. does teach:
ranking, by the formative feedback engine, the subset of semantically equivalent content by degree of similarity to the semantic representation of the first content (see ¶ [0014, 0018, and 0027]: “[0014] In one embodiment, the summary application may identify related entities and associated text phrases for the summary by removing filler or statistically common words such as "the," "and," "of," etc. from sentences which include the entity identified from the user-selected text. The summary application may then take remaining clauses which are subjects (or targets) to be the related entities. Other text which is relevant to the identified entity itself and/or the related entities may be taken as the associated text phrases. After identifying the related entities and associated text phrases, the summary application may further determine semantically important text phrases so that important text phrases are presented to the user. In one embodiment, the summary application may determine semantically important text phrases using text analytics, with text phrases that are repeated often considered more important, and vice versa. Using this approach, the summary application may rank the text phrases. The summary application may then include in the summary frequently correlated related entities along and associated text phrases that are semantically important (i.e., highly ranked). Due to limited screen space, semantically important text phrases may be ranked, with less important text phrases being culled from display. Such a summary may be presented to the user via, e.g., a pop-up window. [0018] After identifying the related entities and associated text phrases, the summary application may further determine most semantically important text phrases so that only important text phrases are presented in the summary 130. More specifically, the summary application may rank the text phrases according to semantic importance, and select one or more of the highly ranked text phrases to present in the summary 130. In one embodiment, the summary application may determine semantically important text phrases based on the frequency of occurrence of the text phrases. For example, if "Peter" or "Peter Thompson" is repeatedly mentioned as having "Harold Thompson" as a parent, then the text phrases which include this mention of his parents may be considered more semantically important. [0027] At step 240, the summary application determines the most semantically important text phrases using text analytics. For example, the summary application may rank the text phrases according to semantic importance, and select one or more of the highly ranked text phrases as the semantically important text phrases. The summary application may identify a large number of text phrases associated with related entities at step 230. Step 240 would then be applied to reduce the number of text phrases so that a limited number of short text phrases may be are added to a summary, discussed below.”);
determining, by the formative feedback engine, a first semantically equivalent content based on the ranking (see ¶ [0014, 0018, and 0027] citations as in limitation above: “[0027] At step 240, the summary application determines the most semantically important text phrases using text analytics. For example, the summary application may rank the text phrases according to semantic importance, and select one or more of the highly ranked text phrases as the semantically important text phrases…”)
Walsh et al., Chorakhalikar et al., and Fink et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of Fink et al. of ranking, by the formative feedback engine, the subset of semantically equivalent content by degree of similarity to the semantic representation of the first content; determining, by the formative feedback engine, a first semantically equivalent content based on the ranking which provides the benefit of providing assistance to readers of electronic documents ([0068] of Fink et al.).
Regarding claim 19, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claim 15, above.
Walsh et al. further teaches:
19. The computer readable storage media of claim 15,
wherein the processor-executable instructions to identify, by the formative feedback engine, the first textual feedback (see ¶ [0004-0007] citation as in claim 15 above.) cause the one or more processors to further execute processor-executable instructions stored in the computer readable storage media (see ¶ [0057 and 0103] citations as in claim 1, above) to:
determine, by the formative feedback engine, a client device associated with the writing assessment (see ¶ [0004]: “…The system has: a memory storing one or more generative models; a processor coupled to the memory programmed with executable instructions, the instructions including an interface for obtaining response data for a constructed-response test from a plurality of examinee electronic devices,…” and ¶ [0044]: “…Embodiments described herein use LLMs with rubric, automated scoring, narrative analysis, feedback, formative assessment, and so on…”);
Chorakhalikar et al. further teaches:
identify, by the formative feedback engine, the first textual feedback based on the comparison of the first semantic representation to the semantic store (see ¶ [0067 and 0080] citation as in claim 15 above.)
determine, by the formative feedback engine, a plurality of semantically equivalent content to the first semantic representation within the semantic store (see ¶ [0061 and 0065]: “[0061] In at least one embodiment, an output of a data clustering algorithm is examined and a score to indicate how well a clustering algorithm performed is generated. In at least one embodiment, an evaluation metric is a score generated that identifies how well clusters are grouped together based on semantic similarity within and between said clusters. In at least one embodiment, an evaluation metric is a score is generated by passing output clusters, which consist of raw text and inferenced cluster grouping, to a neural network, which then transforms raw text into vectors that represent semantic meaning of each string of text… [0065] …In at least one embodiment, clustering performance evaluator 108 may output performance metrics to administrator client 114. In at least one embodiment, administrator 116 may update parameters of clustering component 106 through administrator client 114 based on performance metrics generated by clustering performance evaluator 108 and provided to administration client 114.”);
determine, by the formative feedback engine, a plurality of textual feedback based on the plurality of semantically equivalent content (see ¶ [0061 and 0065] citations as in limitation above: “…evaluation metric is a score generated that identifies how well clusters are grouped together based on semantic similarity within and between said clusters…”),
wherein each of the textual feedback within the plurality of textual feedback corresponds to a respective semantically equivalent content of the plurality of semantically equivalent content (see ¶ [0061 and 0065] citations as in limitation above: “…which then transforms raw text into vectors that represent semantic meaning of each string of text…”);
determine, by the formative feedback engine, a subset of textual feedback from the plurality of textual feedback based on the client device (see ¶ [0061 and 0065] citations as in limitation above: “[0065] …In at least one embodiment, clustering performance evaluator 108 may output performance metrics to administrator client 114. In at least one embodiment, administrator 116 may update parameters of clustering component 106 through administrator client 114 based on performance metrics generated by clustering performance evaluator 108 and provided to administration client 114.”);
determine, by the formative feedback engine, the textual feedback as associated with the first semantically equivalent content (see ¶ [0067]: “[0067] In at least one embodiment, NLP clustering component 206 receives preprocessed text data from data preprocessing component 204. In at least one embodiment, NLP clustering component 206 generates vector representations of each string of text data 202, where vectorized representations may be used to identify semantic similarity between strings of text data. In at least one embodiment, a BERT NLP model is used to generate vectorized representations of text data 202 for identifying semantic similarity. In at least one embodiment, based on identified semantic similarity using vectorized representations of text data, said text data is grouped into clusters of semantically similar strings of text data. In at least one embodiment, where text data is of feedback related to an application, feedback containing “I cannot hear any sound” and “please fix headphone output issue” would be labeled by NLP clustering component to be semantically similar and identified as being part of a cluster, whereas “my game is lagging when I press fire” would be identified as not being semantically similar and not identified as part of a cluster related to sound issues.” and ¶ [0080]: “In at least one embodiment, NLP clustering component 706 receives preprocessed text data from data preprocessing component 704. In at least one embodiment, NLP clustering component 706 generates vector representations of each string of text data 702, where vectorized representations may be used to identify semantic similarity between strings of text data. In at least one embodiment, vectorized representation of text data by NLP clustering component 706 is proved to a dimensionality reduction component 708 to generate visualization plots. In at least one embodiment, dimensionality reduction component 708 uses t-Distributed Stochastic Neighbor Embedding (“t-SNE”) as an algorithm for dimensionality reduction and data visualization of text data 702 so to generate a cluster data visualization plot 710. In at least one embodiment, dimensionality reduction component 708 has a parameter toggling option between three levels to define data density of plot.”).
Walsh et al. and Chorakhalikar et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. to incorporate the teachings of Chorakhalikar et al. of determine, by the formative feedback engine, a plurality of semantically equivalent content to the first semantic representation within the semantic store; determine, by the formative feedback engine, a plurality of textual feedback based on the plurality of semantically equivalent content, wherein each of the textual feedback within the plurality of textual feedback corresponds to a respective semantically equivalent content of the plurality of semantically equivalent content; determine, by the formative feedback engine, a subset of textual feedback from the plurality of textual feedback based on the client device; determine, by the formative feedback engine, the textual feedback as associated with the first semantically equivalent content which provides the benefit of aiding in monitoring accuracy and performance of clustering of user feedback ([0062] of Chorakhalikar et al.).
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but Fink et al. does teach:
rank, by the formative feedback engine, the subset of textual feedback by degree of similarity between a subset of semantically equivalent content to the first semantic representation, wherein the subset of semantically equivalent content corresponds to the subset of textual feedback (see ¶ [0014, 0018, and 0027] citations as in claim 10, above.);
determine, by the formative feedback engine, a first semantically equivalent content based on the ranking (see ¶ [0014, 0018, and 0027] citations as in claim 10, above.);
Walsh et al., Chorakhalikar et al., and Fink et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of Fink et al. of rank, by the formative feedback engine, the subset of textual feedback by degree of similarity between a subset of semantically equivalent content to the first semantic representation, wherein the subset of semantically equivalent content corresponds to the subset of textual feedback; determine, by the formative feedback engine, a first semantically equivalent content based on the ranking which provides the benefit of providing assistance to readers of electronic documents ([0068] of Fink et al.).
Claim 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over [ primary reference (include author and PG pub / patent / NPL reference title) ] as applied to claim 7 above, and further in view of Tran et al. (US 20220164532 A1).
Regarding claim 11, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claim 7, above.
Walsh et al. further teaches:
11. The method of claim 7, wherein generating, by the formative feedback engine, a commentary insight for the first content based on the textual feedback and the first content (see ¶ [0004, 0007, and 0048] citations as in claim 1, above.) further comprises:
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but Tran et al. does teach:
determining, by the formative feedback engine, a writing style associated with the client device (see ¶ [0071]: “According to one embodiment, the suggestion module 210 may include a personalization component 250 to support the modification ranking strategies. Since text editors 204 (e.g., e-mail application) are typically used by users 202 on their personal computing devices, the personalization component 250 may learn the writing styles of the users 202 so that potential suggestions or modifications to the text data 230 may be ranked to follow individual writing styles of the users 202.”); and
generating, by the formative feedback engine, the commentary insight in the writing style of the client device (see ¶ [0071] citation as in limitation above.).
Walsh et al., Chorakhalikar et al., and Tran et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of Tran et al. of determining, by the formative feedback engine, a writing style associated with the client device; and generating, by the formative feedback engine, the commentary insight in the writing style of the client device which provides the benefit of improving the technical field of data protection ([0020] of Tran et al.).
Claims 12 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Walsh et al. (US 20250036880 A1) in view of Chorakhalikar et al. (US 20250053818 A1) as applied to claims 7 and 15 above, and further in view of PADMANABHAN et al. (US 20250005299 A1).
Regarding claim 12, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claim 7, above.
Walsh et al. further teaches:
12. The method of claim 7, wherein querying, by the formative feedback engine, the semantic store based on the semantic representation of the first content and determining, by the formative feedback engine, the textual feedback from the semantic store based on the semantic representation of the first content (see ¶ [0064 and 0067] citation as in claim 7, above.) comprises:
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but PADMANABHAN et al. does teach:
performing, by the formative feedback engine, a Retrieval-Augmented Generation (RAG) operation for the semantic representation of the first content using the semantic store (see ¶ [0026]: “…The generative language model system 200 also includes a feedback store 218, which in turn includes stored information such as an audit trail 220, LLM output 222, toxicity scores 224, data sources 226, and alerts and filters 228. The generative language model system 200 also includes a secure data retrieval interface 230, which in turn includes interfaces for retrieving CRM data via database queries, semantic search, or hybrid search involving a combination of lexical and semantic search through retrieval augmented generation at 232 and for retrieving feedback data at 234. The generative language model system 200 also includes CRM apps 236 and communicates with generative language models 240 through 242.”); and
determining, by the formative feedback engine, the textual feedback from the RAG operation (see ¶ [0026]: “…The generative language model system 200 also includes a secure data retrieval interface 230, which in turn includes interfaces for retrieving CRM data via database queries, semantic search, or hybrid search involving a combination of lexical and semantic search through retrieval augmented generation at 232 and for retrieving feedback data at 234…”).
Walsh et al., Chorakhalikar et al., and PADMANABHAN et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of PADMANABHAN et al. of performing, by the formative feedback engine, a Retrieval-Augmented Generation (RAG) operation for the semantic representation of the first content using the semantic store; and determining, by the formative feedback engine, the textual feedback from the RAG operation which provides the benefit of provide for improved quality or reduced cost ([0077] of PADMANABHAN et al.).
Regarding claim 20, Walsh et al. in combination with Chorakhalikar et al. teaches the limitations as in claim 15, above.
Walsh et al. further teaches:
20. The computer readable storage media of claim 15,
wherein the processor-executable instructions to identify, by the formative feedback engine, the first textual feedback (see ¶ [0004-0007] citation as in claim 15 above.) cause the one or more processors to further execute processor-executable instructions stored in the computer readable storage media (see ¶ [0057 and 0103] citations as in claim 1, above) to:
Chorakhalikar et al. further teaches:
identify, by the formative feedback engine, the first textual feedback based on the comparison of the first semantic representation to the semantic store (see ¶ [0067 and 0080] citation as in claim 15 above.)
Walsh et al. and Chorakhalikar et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. to incorporate the teachings of Chorakhalikar et al. of identify, by the formative feedback engine, the first textual feedback based on the comparison of the first semantic representation to the semantic store which provides the benefit of aiding in monitoring accuracy and performance of clustering of user feedback ([0062] of Chorakhalikar et al.).
However, Walsh et al. in combination with Chorakhalikar et al. do not explicitly teach, but PADMANABHAN et al. does teach:
perform, by the formative feedback engine, a Retrieval-Augmented Generation (RAG) operation for the first semantic representation using the semantic store (see ¶ [0026] citation as in claim 12, above.); and
determine, by the formative feedback engine, the first textual feedback from the RAG operation (see ¶ [0026] citation as in claim 12, above.).
Walsh et al., Chorakhalikar et al., and PADMANABHAN et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in natural language processing. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Walsh et al. in combination with Chorakhalikar et al. to incorporate the teachings of PADMANABHAN et al. of performing, by the formative feedback engine, a Retrieval-Augmented Generation (RAG) operation for the semantic representation of the first content using the semantic store; and determining, by the formative feedback engine, the textual feedback from the RAG operation which provides the benefit of provide for improved quality or reduced cost ([0077] of PADMANABHAN et al.).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Keisha Y Castillo-Torres whose telephone number is (571)272-3975. The examiner can normally be reached Monday - Friday, 9:00 am - 4:00 pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at (571)272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Keisha Y. Castillo-Torres
Examiner
Art Unit 2659
/Keisha Y. Castillo-Torres/Examiner, Art Unit 2659
Prosecution Insights