DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim(s) 1-24 is/are pending and has/have been examined.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/30/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claim 6 is objected to because of the following informalities:
Claim 6 recites “said a course text content input window”. The Examiner suggests amending the claims to remove the additional “a”, which appears to be a typo.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 10-18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 10-18 recite a number of terms that seem to have conflicting antecedent basis or were not updated to reflect the terms recited in independent claim 1 when it was amended. These terms include “a presentation”, “task specific performance evaluation prompts”, and “a competency score”/”competency scores”. The Examiner suggests reviewing claims 10-18 in light of the amendments to claim 1, and amending the claim terms as necessary to ensure consistency and clarity.
Claim 10 recites “presentation text” in line 3. The Examiner suggests amending the claim(s) to recite –said presentation text-- in order to maintain clear antecedent basis.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-24 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim(s) 1, 19, and 24, the limitation(s) of ((claims 1 and 19) analyzing, generating, depicting, selecting), (claims 1 and 24) evaluating, (claims 1 and 24) generating, ((claim 24) generating, and depicting), as drafted, are processes that, under broadest reasonable interpretation, covers performance of the limitation in the mind and/or with pen and paper but for the recitation of generic computer components, as well mathematical calculations in prose. More specifically, the mental process of a human looking at a course description and writing down objectives and evaluation metrics using given instructions, writing the objectives and metrics down on a piece of paper, having another person circle a specific item in the list, reviewing an assignment to determine how well the assignment matches the objectives and metrics using additional instructions, calculating a grade for each element of the assignment pertaining to the objectives and metrics, deciding on an explanation for why each element was given the grade it was, and writing the explanation out on another piece of paper. The machine learning algorithm and large language model read to learned rules for how to perform the different tasks, and the different prompts read to additional instructions provided for each task. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind and/or with pen and paper but for the recitation of generic computer components, and/or recites mathematical calculations in prose, then it falls within the --Mental Processes—and –Mathematical Concepts-- groupings of abstract ideas. Accordingly, the claim(s) recite(s) an abstract idea.
This judicial exception is not integrated into a practical application because the recitation of a display surface of a computing device in claim 1, and a computer readable medium, system, and display surface of a computing device of claims 19 and 24, and reads to generalized computer components, based upon the claim interpretation wherein the structure is interpreted using pgs 6-8 in the specification. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim(s) is/are directed to an abstract idea.
The claim(s) do(es) not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of using generalized computer components to analyze, generate, depict, select, evaluate, generate, generate, and depict, amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim(s) is/are not patent eligible.
With respect to claim(s) 2, the claim(s) recite(s) specific models, which reads on a human using specific sets of rules. No additional limitations are present.
With respect to claim(s) 3, 4, 7, 9, 11-14, 17, 20, and 21, the claim(s) recite(s) different characteristics of the prompts, which reads on a human receiving specific instructions to perform the processes in a specific way or to provide specific output. No additional limitations are present.
With respect to claim(s) 5 and 23, the claim(s) recite(s) selecting learning objectives and criteria alters the objective and/or criteria, which reads on a human editing the objectives and metrics that have been circled on the page. No additional limitations are present.
With respect to claim(s) 6 and 8, the claim(s) recite(s) an input window and inputting text into the window, which reads on a human writing down specific information in a specific box drawn on a piece of paper, such as a box on a form. No additional limitations are present.
With respect to claim(s) 10, the claim(s) recite(s) evaluating, generating, generating, and depicting, which reads on a human reviewing an assignment to determine how well the assignment matches the objectives and metrics using additional instructions, calculating a grade for each element of the assignment pertaining to the objectives and metrics, deciding on an explanation for why each element was given the grade it was, and writing the explanation out on another piece of paper. No additional limitations are present.
With respect to claim(s) 15 and 16, the claim(s) recite(s) the reasoning statements have specific characteristics, which reads on a human deciding on an explanation for why each element was given the grade it was, where the explanation includes specific information. No additional limitations are present.
With respect to claim(s) 18, the claim(s) recite(s) adjusting competency scores, which reads on a human recognizing a difference between the grades based on who did the grading, and adjusting the grades using a specific calculation to reduce variation between graders. No additional limitations are present.
With respect to claim(s) 22, the claim(s) recite(s) that the course text content has specific characteristics, which reads on a human looking at a course description that includes specific materials. No additional limitations are present.
These claims further do not remedy the judicial exception being integrated into a practical application and further fail to include additional elements that are sufficient to amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-15 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zertuche (U.S. Patent No. 11,068,649), hereinafter Zertuche, in view of Sridhar et al. (“Harnessing LLMs in Curricular Design: Using GPT-4 to Support Authoring of Learning Objectives”, arXiv:2306.17459v1, 30 Jun 2023), hereinafter Sridhar, in view of Zheng et al. (“Personalized Feedback Generation Using LLMs: Enhancing Student Learning in STEM Education”, JACS, Vol. 3(10), pp. 8-22, 10 October 2023), hereinafter Sridhar, and further in view of Coyne et al. (“Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction”, arXiv:2303.14342v2, 30 May 2023), hereinafter Coyne.
Regarding claim 1, Zertuche teaches
A method (a method (2:4-6)), comprising:
analyzing course text content of a course by a … algorithm including a … language model… (a course syllabus and other course description information, i.e. course text content of a course, is analyzed using a natural language analysis to identify expected target outcomes of the course, i.e. analyzing…by a…algorithm including a…language model (15:20-29),(18:2-29));
automatically generating course learning objectives and course learning criteria for said course based on said analyzing of said course text content by said … language model … (a course syllabus and other course description information, i.e. course text content, is analyzed using a natural language analysis, i.e. based on said analyzing of said course text content by said…language model, to identify expected target outcomes of the course, i.e. automatically generating course learning objectives…for said course, as well as a weighting for each outcome, i.e. course learning criteria for said course (15:20-29),(18:2-40));
depicting said course learning objectives and said course learning criteria in a course learning objectives list on a display surface of a computing device, wherein each course learning objective and each course learning criteria within said course learning objective list associated with a course learning objective selection element (interactive dashboards may be displayed as a user interface on a user device, i.e. depicting…on a display surface of a computing device, to present analysis results, such as the learning outcomes and weight for each outcome, i.e. course learning objectives and said course learning criteria, may be reflected as lists in the dashboards, i.e. depicting…in a course learning objectives list, where the dashboard includes selection controls that let the user specify subsets of data to be further processed, i.e. each course learning objective and each course learning criteria within said course learning objective list associated with a course learning objective selection element (7:44-52),(9:25-51),(18:5-40),(19:57-67),(20:8-23));
selecting said course learning objectives and said course learning criteria for said course by user interaction with said course learning objective selection element (the dashboard includes selection controls, i.e. said course learning objective selection element, that let the user specify subsets of data to be further processed, i.e. selecting said course learning objectives and said course learning criteria for said course by user interaction (7:44-52),(9:25-51),(18:5-40),(19:57-67),(20:8-23)); and
evaluating submitted course presentations by said …algorithm including said … language model for each selected course learning objective and each course learning criteria using course evaluation prompts, … (an assessment may be written, orally administered, or practical, such as public speaking, i.e. course presentations, where the assessment service provides assessment data including score data, category, and outcome data including a weight for each outcome, to the analysis engine, i.e. evaluating submitted course presentations by said… algorithm including said…language model…using course evaluation prompts, where the analysis engine determines whether all the outcomes that the course purports to cover are actually being tested by the categories in the assessment for the course, and whether the weights of the outcomes corresponds to the weight of the assessment, i.e. evaluating…for each selected course learning objective and each course learning criteria (4:15-41),(5:61-6:11),(18:2-51),(19:20-56)); and
automatically generating competency scores for each selected course learning objective and each course learning criteria based on evaluation of said submitted course presentations by said … algorithm including said … language model using said course evaluation prompts (the analysis engine includes one or more correlation modules that perform an analysis to correlate outcome data with assessment data, i.e. based on evaluation of said submitted course presentations by said…algorithm including said…language model using said course evaluation prompts, where the correlation can produce output data that indicates how well students are performing with respect to the learning outcomes of the course, based on the scores earned for questions on categories associated with the learning outcomes, where the categories and objectives also include weights, i.e. automatically generating competency scores for each selected course learning objective and each course learning criteria (4:15-53),(6:12-34),(7:26-43),(15:20-42),(20:24-27)).
While Zertuche provides determining learning outcomes and whether student assessments meet those outcomes, Zertuche does not specifically teach the use of a machine learning algorithm including a large language model, and thus does not teach
analyzing course text content of a course by a machine learning algorithm including a large language model using course learning objective prompts;
automatically generating course learning objectives and course learning criteria….
Sridhar, however, teaches analyzing course text content of a course by a machine learning algorithm including a large language model using course learning objective prompts (a combination of prompts including a user prompt including course name and goals, i.e. course text content of a course…using course learning objective prompts, instructs a GPT model to generate learning objectives for the provided course information, i.e. analyzing…by a machine learning algorithm including a large language model Figs. 2 and 3,(Sec. 3.1-3.2));
automatically generating course learning objectives and course learning criteria…(a combination of prompts instructs a GPT model to generate learning objectives for the provided course information, i.e. automatically generating course learning objectives, where the learning objectives include measurable behavior, i.e. course learning criteria Figs. 2 and 3,(Sec. 1, 3.1-3.2)).
Zertuche and Sridhar are analogous art because they are from a similar field of endeavor in using systems to evaluate course materials for learning objectives. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the determining learning outcomes and whether student assessments meet those outcomes teachings of Zertuche with the use of a GPT model and a combination of prompts to generate lists of learning objectives for a course as taught by Sridhar. It would have been obvious to combine the references to enable LLMs to efficiently produce high-quality learning objectives to save instructor time (Sridhar Intro).
While Zertuche in view of Sridhar provides determining learning outcomes and whether student assessments meet those outcomes, Zertuche in view of Sridhar does not specifically teach the use of a machine learning algorithm including a large language model to evaluate course presentations, and thus does not teach
evaluating submitted course presentations by said machine learning algorithm including said large language model for each selected course learning objective and each course learning criteria using course evaluation prompts, …; and
automatically generating competency scores for each selected course learning objective and each course learning criteria based on evaluation of said submitted course presentations by said machine learning algorithm including said large language model using said course evaluation prompts.
Zheng, however, teaches evaluating submitted course presentations by said machine learning algorithm including said large language model for each selected course learning objective and each course learning criteria using course evaluation prompts, … (the LLM-based feedback generation framework uses prompts, i.e. by said machine learning algorithm including said large language model…using course evaluation prompts, to identify key concepts and related errors made by the student, i.e. evaluating submitted course presentations…for each selected course learning objective and each course learning criteria Tables 1,2,3,(Sec. 3.1-3.2)); and
automatically generating competency scores for each selected course learning objective and each course learning criteria based on evaluation of said submitted course presentations by said machine learning algorithm including said large language model using said course evaluation prompts (the LLM-based feedback generation framework uses prompts, i.e. by said machine learning algorithm including said large language model using said course evaluation prompts, to identify key concepts and related errors made by the student, i.e. for each selected course learning objective and each course learning criteria based on evaluation of said submitted course presentations, to determine feedback that produces a grade and identifies competency in different areas and the change in competency over time, i.e. automatically generating competency scores Tables 1,2,3,5,9,(Sec. 3.1-3.2, 4.2)).
Zertuche, Sridhar, and Zheng are analogous art because they are from a similar field of endeavor in using systems to evaluate educational materials. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the determining learning outcomes and whether student assessments meet those outcomes teachings of Zertuche, as modified by Sridhar, with the use of LLM-based feedback generation to identify key concepts and errors made by the student, as well as provide feedback as taught by Zheng. It would have been obvious to combine the references to enable personalized feedback to provide targeted guidance for different student populations that improves student engagement (Zheng (Sec. 4.2 and 5.2)).
While Zertuche in view of Sridhar and Zheng provides different prompting strategies for the LLM, Zertuche in view of Sridhar and Zheng does not specifically teach zero-shot prompts, and thus does not teach
said course evaluation prompts comprise one or more zero shot prompts.
Coyne, however, teaches said course evaluation prompts comprise one or more zero shot prompts (LLM models used for natural language processing tasks can be prompted in different ways, including zero-shot settings (Abstract, Sec. 2.2,Sec. 3)).
Where Zheng teaches the prompts are for generating feedback on student work pertaining to a domain and key concepts Tables 1,2,3,(Sec. 3.1-3.2).
Zertuche, Sridhar, Zheng, and Coyne, are analogous art because they are from a similar field of endeavor in using language processing systems to evaluate text information. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the different prompting strategies for the LLM teachings of Zertuche, as modified by Sridhar and Zheng, with the specific use of zero-shot prompts as taught by Coyne. It would have been obvious to combine the references to identify the best prompt approach depending on the model being used (Coyne Sec. 5.1).
Regarding claim 2, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 1, and Coyne further teaches
said large language model is selected from the group of large language models consisting of:
ChatGPT*, GPT-3*, GPT-3.5*, and GPT-4* (the models used are from OpenAI, which can include ChatGPT, GPT-3, GPT-3.5, and GPT4 (Abstract, Sec. 2.1)).
Where the motivation to combine is the same as previously presented.
Regarding claims 3 and 4, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 1, and Coyne further teaches
said task specific course learning objectives prompts comprise one or more of:
a zero-shot prompt, a few-shot prompt, and a few-shot chain of thought prompt (prompt settings include zero-shot, few-shot, and chain of thought (Sec. 5.1, Sec. 8)).
Where Sridhar teaches that the prompts are specifically for generating learning objectives (Sec. 3.2).
And where the motivation to combine is the same as previously presented.
Regarding claim 5, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 1, and Zertuche further teaches
selecting said course learning objectives and said course learning criteria for said course by user interaction with said course learning objective selection element alters said course learning objectives or said course learning criteria automatically generated by said large language model using said task specific course learning objective prompts (the dashboard can perform selections of data points to generate a CQI ticket, i.e. selecting … by user interaction with said course learning objective selection element, where the analysis engine may collect and analyze data from the course syllabus or other materials to determine the target outcomes to be covered by each course, i.e. said course learning objectives and said course learning criteria for said course, where course information may be modified, such as through input of course information by the professor, i.e. alters said course learning objectives or said course learning criteria automatically generated (6:59-65),(7:44-52),(9:25-51), (13:49-14:4),(20:55-21:53),(24:33-50)).
Where Sridhar teaches a prompted GPT model to determine learning objectives (Figs. 2 and 3,(Sec. 1, 3.1-3.2)).
And where the motivation to combine is the same as previously presented.
Regarding claim 6, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 1, and Zertuche further teaches
depicting on said display surface of said computing device a course text content input window (the course information, such as a course description or syllabus, can be input to the platform by a professor, such as through interaction with a UI that allows a user to perform editing on words or enter data, i.e. depicting … a course text content input window, where the user may access the dashboards using a user device of any suitable type, i.e. on said display surface of said computing device (6:59-65),(9:16-24),(18:21-29),(19:62-3),(30:37-48)); and
inputting said course text content of a course into said a course text content input window (the course information, such as a course description or syllabus, i.e. course text content of a course, can be input to the platform by a professor, such as through interaction with a UI that allows a user to perform editing on words or enter data, i.e. inputting…into said a course text content input window (6:59-65),(9:16-24),(18:21-29)).
Regarding claim 7, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 1, and Sridhar further teaches
said course learning objectives prompts guide said machine learning algorithm to extract one or more main topics from said course text content of a course input into said course text content input window (a system and user prompt are used by the GPT model to generate learning objectives, i.e. said course learning objectives prompts guide said machine learning algorithm, where the user prompt includes the course name and description, i.e. said course text content of a course input, and the learning objectives relevant to a given topic to be determined and output, i.e. extract one or more main topics from said course text content of a course input Figs. 2 and 3,(Sec. 1, 3.1-3.2)).
Where Zertuche further teaches that course information can be input to the platform through interaction with a UI (6:59-65),(9:16-24),(18:21-29),(19:62-3),(30:37-48).
And where the motivation to combine is the same as previously presented.
Regarding claim 8, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 7, and Zertuche further teaches
depicting on said display surface of said computing device a prompt input window (the course information, such as a course description or syllabus, can be input to the platform by a professor, such as through interaction with a UI that allows a user to perform editing on words or enter data, i.e. depicting … a prompt input window, where the user may access the dashboards using a user device of any suitable type, i.e. on said display surface of said computing device (6:59-65),(9:16-24),(18:21-29),(19:62-3),(30:37-48)).
And Sridhar further teaches inputting, by user interaction, one or more course learning objectives prompts into said prompt input window (a system and user prompt are used by the GPT model to generate learning objectives, where the user prompt is a message that includes the course name and description, i.e. inputting…by user interaction one or more course learning objectives prompts Figs. 2 and 3,(Sec. 1, 3.1-3.2)).
And where the motivation to combine is the same as previously presented.
Regarding claim 9, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 8, and Sridhar further teaches
said task specific course learning objectives prompts instruct said machine learning algorithm to depict said one or more main topics on said display surface of said computing device in a topic list format, wherein said topic list format includes each main topic extracted from said course text content followed by one or more course learning criteria extracted from said course text content (a system and user prompt are used by the GPT model to generate learning objectives, i.e. said task specific course learning objectives prompts instruct said machine learning algorithm to depict said one or more main topics, where the user prompt includes the course name and description, and the learning objectives relevant to a given topic to be determined and output in a numbered list format, where the learning objectives include measurable information, i.e. depict said one or more main topics…in a topic list format said topic list format includes each main topic extracted from said course text content followed by one or more course learning criteria extracted from said course text content Figs. 2 and 3,(Sec. 1, 3.1-3.2)).
Where Zertuche teaches that the learning outcomes and weight for each outcome are reflected as lists in the dashboards (7:44-52),(9:25-51),(18:5-40),(19:57-67),(20:8-23).
And where the motivation to combine is the same as previously presented.
Regarding claim 10, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 1, and Zertuche teaches
evaluating presentation text of a presentation by said …algorithm using task specific performance evaluation prompts, wherein evaluating presentation text comprises identifying relationships between said presentation text and said prior generated course learning objectives and prior generated course learning criteria (an assessment may be written, orally administered, or practical, such as public speaking, i.e. presentation text of a presentation, where the assessment service provides assessment data including score data, category, and outcome data including a weight for each outcome, to the analysis engine, i.e. evaluating presentation text of a presentation by said… algorithm using task specific performance evaluation prompts, where the analysis engine determines whether all the outcomes that the course purports to cover are actually being tested by the categories in the assessment for the course, and whether the weights of the outcomes corresponds to the weight of the assessment, i.e. identifying relationships between said presentation text and said prior generated course learning objectives and prior generated course learning criteria (4:15-41),(5:61-6:11),(18:2-51),(19:20-56));
automatically generating a competency score by said … algorithm using said task specific performance evaluation prompts based on the level of the identified relationships between said presentation text and each of said course learning objectives and each of said course learning criteria prior generated by said … algorithm using said task specific course learning objective prompts (the analysis engine includes one or more correlation modules that perform an analysis to correlate outcome data with assessment data, i.e. based on the level of the identified relationships between said presentation text and each of said course learning objectives and each of said course learning criteria prior generated by said…algorithm using said task specific course learning objective prompts, where the correlation can produce output data that indicates how well students are performing with respect to the learning outcomes of the course, based on the scores earned for questions on categories associated with the learning outcomes, where the categories and objectives also include weights, i.e. automatically generating a competency score (4:15-53),(6:12-34),(7:26-43),(15:20-42),(20:24-27));
automatically generating —results-- by said … algorithm using said task specific performance evaluation prompt for each course learning objective and each course learning criteria (the analysis correlates outcome data with assessment data, where the output data indicates how well students are performing with respect to the learning outcomes of the course, based on the scores earned for questions on categories associated with the learning outcomes, where the categories and objectives also include weights, i.e. by said…algorithm using said task specific performance evaluation prompt for each course learning objective and each course learning criteria, and the results of the analysis are provided on interactive dashboards or in reports, i.e. automatically generating results Figs. 4, 6, 9,(4:15-53),(6:12-34),(7:26-43),(11:38-48),(15:20-42),(20:8-27));
depicting a presentation evaluation on a display surface of said first computing device (the analysis correlates outcome data with assessment data, where the output data indicates how well students are performing with respect to the learning outcomes of the course, and the results of the analysis are provided on interactive dashboards or in reports on a user device, i.e. depicting a presentation evaluation on a display surface of said first computing device Figs. 4, 6, 9,(4:15-53),(6:12-34),(7:26-52),(11:38-48),(15:20-42),(19:57-67),(20:8-27)),
wherein said presentation evaluation depicts said course learning objectives and said course learning criteria each associated with said competency score (the analysis correlates outcome data with assessment data, where the output data indicates how well students are performing with respect to the learning outcomes of the course, based on the scores earned for questions on categories associated with the learning outcomes, where the categories and objectives also include weights, i.e. depicts said course learning objectives and said course learning criteria each associated with said competency score, and the results of the analysis are provided on interactive dashboards or in reports, i.e. said presentation evaluation depicts Figs. 4, 6, 9,(4:15-53),(6:12-34),(7:26-43),(11:38-48),(15:20-42),(20:8-27)),
wherein said presentation evaluation depicts said —results-- associated with each course learning criteria (the analysis correlates outcome data with assessment data, where the output data indicates how well students are performing with respect to the learning outcomes of the course, based on the scores earned for questions on categories associated with the learning outcomes, where the categories and objectives also include weights, i.e. depicts said results associated with each course learning criteria, and the results of the analysis are provided on interactive dashboards or in reports, i.e. said presentation evaluation depicts Figs. 4, 6, 9,(4:15-53),(6:12-34),(7:26-43),(11:38-48),(15:20-42),(20:8-27)).
Where Zheng further teaches evaluating presentation text of a presentation by said machine learning algorithm using task specific performance evaluation prompts, wherein evaluating presentation text comprises identifying relationships between said presentation text and said prior generated course learning objectives and prior generated course learning criteria (the LLM-based feedback generation framework uses prompts, i.e. by said machine learning algorithm using task specific performance evaluation prompts, to process raw text from student responses, i.e. evaluating presentation text of a presentation, to identify key concepts and related errors made by the student, i.e. identifying relationships between said presentation text and said prior generated course learning objectives and prior generated course learning criteria Tables 1,2,3,(Sec. 3.1-3.2));
automatically generating a competency score by said machine learning algorithm using said task specific performance evaluation prompts based on the level of the identified relationships between said presentation text and each of said course learning objectives and each of said course learning criteria … (the LLM-based feedback generation framework uses prompts, i.e. by said machine learning algorithm including said large language model using said course evaluation prompts, to identify key concepts and related errors made by the student, i.e. for each selected course learning objective and each course learning criteria based on evaluation of said submitted course presentations, to determine feedback that produces a grade and identifies competency in different areas and the change in competency over time, i.e. automatically generating competency scores Tables 1,2,3,5,9,(Sec. 3.1-3.2, 4.2));
automatically generating supportive reasoning statements by said machine learning algorithm using said task specific performance evaluation prompt for each course learning objective and each course learning criteria (the LLM-based feedback generation framework uses prompts, i.e. by said machine learning algorithm using said task specific performance evaluation prompt, to identify key concepts and related errors made by the student, i.e. for each course learning objective and each course learning criteria, to determine feedback that can include clear explanations, multimodal presentations, and different levels of comprehensiveness, i.e. automatically generating supportive reasoning statements Tables 1,2,3,5,9,11,(Sec. 3.1-3.2, 4.2));
wherein said presentation evaluation depicts said supportive reasoning statements associated with each course learning criteria (feedback includes multimodal presentations, such as explanations and visual aids, i.e. presentation evaluation depicts said supportive reasoning statements, related to the concepts and errors, i.e. associated with each course learning criteria Tables 1,2,3,5,9,11,(Sec. 3.1-3.2, 4.2)).
Where Sridhar further teaches each of said course learning objectives and each of said course learning criteria prior generated by said machine learning algorithm using said task specific course learning objective prompts (a combination of prompts instructs a GPT model to generate learning objectives for the provided course information, i.e. each of said course learning objectives…prior generated by said machine learning algorithm using said task specific course learning objective prompts, where the learning objectives include measurable behavior, i.e. each of said course learning criteria Figs. 2 and 3,(Sec. 1, 3.1-3.2)).
And where the motivation to combine is the same as previously presented.
Regarding claim 11, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 10, and Zheng further teaches
said task specific performance evaluation prompts includes a transcription component prompt including said presentation text of said presentation (different prompts are utilized by the LLM as input, i.e. task specific performance evaluation prompts includes a transcription component prompt, where the first input to the first component of the LLM includes the raw text of the student response , i.e. including said presentation text of said presentation Table 1, 2, (Sec. 3.1)).
Where the motivation to combine is the same as previously presented.
Regarding claim 12, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 10, and Zertuche further teaches
said task specific performance evaluation prompts include a course learning objectives prompt and a course learning criteria prompt including prior generated course learning objectives and course learning criteria (the analysis engine determines whether all the outcomes that the course purports to cover are actually being tested by the categories in the assessment for the course, and whether the weights of the outcomes corresponds to the weight of the assessment, i.e. prior generated course learning objectives and course learning criteria, where the analysis engine receives learning outcome data including target learning outcomes, i.e. a course learning objectives prompt, and the weight for each outcome, i.e. a course learning criteria prompt (4:15-41),(5:61-6:11),(18:2-51),(19:20-56)).
Regarding claim 13, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 10, and Zertuche further teaches
said task specific performance evaluation prompts include an instruction prompt instructs said machine learning algorithm to score identified relationships between said presentation text and each of said course learning objectives and each of said course learning criteria with competency score value within a numerical range (the analysis engine includes one or more correlation modules that perform an analysis to correlate input outcome data with input assessment data including score data, i.e. include an instruction prompt instructs said …algorithm to score identified relationships between said presentation text and each of said course learning objectives and each of said course learning criteria, where the correlation can produce output data that indicates how well students are performing with respect to the learning outcomes of the course, based on the scores earned for questions on categories associated with the learning outcomes, where the categories and objectives also include weights, and score data may be normalized to be on a particular scale, i.e. competency score value within a numerical range (4:15-53),(6:12-34),(7:26-43),(15:20-42),(20:24-27),(23:19-39),(24:51-67)).
Where Zheng teaches the evaluation is performed by a prompted LLM-based feedback generation framework (Tables 1,2,3,5,9,(Sec. 3.1-3.2, 4.2)).
And where the motivation to combine is the same as previously presented.
Regarding claim 14, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 10, and Zheng further teaches
said task specific performance evaluation prompts instructs said machine learning algorithm to generate supportive reasoning statements for each course learning criteria (the LLM-based feedback generation framework uses prompts, i.e. task specific performance evaluation prompts instructs said machine learning algorithm, to identify key concepts and related errors made by the student, i.e. for each course learning criteria, to determine feedback that can include clear explanations, multimodal presentations, and different levels of comprehensiveness, i.e. generate supportive reasoning statements Tables 1,2,3,5,9-11,(Sec. 3.1-3.2, 4.2)).
Where the motivation to combine is the same as previously presented.
Regarding claim 15, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 14, and Zheng further teaches
said supportive reasoning statements include machine learning reasoning by said machine learning algorithm explaining identified relationships between said presentation text and each of said course learning objectives and each of said course learning criteria (the feedback from the LLM includes clear explanations, multimodal presentations, and different levels of comprehensiveness, i.e. supportive reasoning statements include machine learning reasoning by said machine learning algorithm, including explicitly identifying errors based on the key concepts, pinpointing exactly where reasoning when wrong, and acknowledging what was done correctly before addressing errors, i.e. explaining identified relationships between said presentation text and each of said course learning objectives and each of said course learning criteria Tables 1,2,3,5,9-11,(Sec. 3.1-3.2, 4.2)).
Where the motivation to combine is the same as previously presented.
Regarding claim 17, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 10, and Zheng further teaches
said task specific performance evaluation prompts includes a formatting prompt to provide a formal specification of the output from the machine learning algorithm (different prompt styles are used to enable the LLM, i.e. said task specific performance evaluation prompts…from the machine learning algorithm, to provide outputs with specific characteristics, such as providing exemplar feedback pairs, i.e. includes a formatting prompt to provide a formal specification of the output Tables 1,2,3,5,9,(Sec. 3.1-3.2, 4.2)).
Where Sridhar further teaches that a prompt can provide a specific format for the output (Fig. 3).
And where the motivation to combine is the same as previously presented.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zertuche, in view of Sridhar, in view of Zheng, in view of Coyne, and further in view of Jacobsen and Weber (“The Promises and Pitfalls of LLMs as Feedback Providers: A Study of Prompt Engineering and the Quality of AI-Driven Feedback” OSF Preprints, 28 September 2023), hereinafter Jacobsen.
Regarding claim 16, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 14, and Zheng further teaches
said supportive reasoning statements include… examples of said identified relationships between said presentation text and each of said course learning objectives and each of said course learning criteria (the feedback from the LLM includes clear explanations, multimodal presentations, and different levels of comprehensiveness, i.e. supportive reasoning statements, including explicitly identifying errors based on the key concepts, pinpointing exactly where reasoning when wrong, and acknowledging what was done correctly before addressing errors, i.e. examples of said identified relationships between said presentation text and each of said course learning objectives and each of said course learning criteria Tables 1,2,3,5,9-11,(Sec. 3.1-3.2, 4.2)).
While Zertuche in view of Sridhar, Zheng, and Coyne provides clear explanations as feedback, Zertuche in view of Sridhar, Zheng, and Coyne does not specifically teach the use of verbatim text extracts in the feedback, and thus does not teach
supportive reasoning statements include verbatim text extracts as examples.
Jacobsen, however, teaches supportive reasoning statements include verbatim text extracts as examples (good feedback examples include statements such as “The verb 'recognize' is on the lower end of Bloom's taxonomy; it's more about recall than application or analysis.", or “Next, the verbs you've chosen, 'recognize' and 'understand,' are a bit vague in the context of Bloom's taxonomy”, i.e. verbatim text extracts as examples Table 2,(Sec. 4.2)).
Zertuche, Sridhar, Zheng, Coyne, and Jacobsen are analogous art because they are from a similar field of endeavor in providing feedback after analysis of input. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the clear explanations as feedback teachings of Zertuche, as modified by Sridhar, Zheng, and Coyne, with the feedback including extracted text from the input as taught by Jacobsen. It would have been obvious to combine the references to enable prompting an LLM to produce high quality feedback (Jacobsen Abstract)).
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zertuche, in view of Sridhar, in view of Zheng, in view of Coyne, and further in view of Chen and He (“Automated Essay Scoring by Maximizing Human-machine Agreement”, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1741–1752, Seattle, Washington, USA, 18-21 October 2013), hereinafter Chen.
Regarding claim 18, Zertuche in view of Sridhar, Zheng, and Coyne teaches claim 10.
While Zertuche in view of Sridhar, Zheng, and Coyne provides using an LLM to provide feedback for student responses, Zertuche in view of Sridhar, Zheng, and Coyne does not specifically teach adjusting scores to reduce variance, and thus does not teach
adjusting competency scores generated by said machine learning algorithm to reduce variance between manual scoring and machine scoring using a consensus algorithm.
Chen, however, teaches adjusting competency scores generated by said machine learning algorithm to reduce variance between manual scoring and machine scoring using a consensus algorithm (automated essay scoring can rank algorithms for learning a rating, i.e. adjusting competency scores generated by said machine learning algorithm…using a consensus algorithm, where the agreement between the human and machine raters is directly incorporated into the loss function, i.e. reduce variance between manual scoring and machine scoring using a consensus algorithm (Abstract, Intro)).
Zertuche, Sridhar, Zheng, Coyne, and Chen are analogous art because they are from a similar field of endeavor in providing feedback after analysis of input. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the using an LLM to provide feedback for student responses teachings of Zertuche, as modified by Sridhar, Zheng, and Coyne, with the incorporation of agreement between human and machine raters into the loss function for an AES system as taught by Chen. It would have been obvious to combine the references to enable AES systems to achieve performance comparable to professional human raters (Chen Abstract).
Claim(s) 19, 20, and 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zertuche, in view of Sridhar.
Regarding claim 19, Zertuche teaches
A non-transitory computer readable medium encoded with a machine learning algorithm that, when executed, cause a system to perform actions to depict course learning objectives and course learning criteria (a storage medium having instructions stored thereon which, when executed by a processor cause the processor to perform operations (2:58-66)), the actions comprising:
analyzing course text content of a course by said …algorithm including a … language model … (a course syllabus and other course description information, i.e. course text content of a course, is analyzed using a natural language analysis to identify expected target outcomes of the course, i.e. analyzing…by a…algorithm including a…language model (15:20-29),(18:2-29));
automatically generating course learning objectives and course learning criteria for said course based on said analyzing of said course text content by said … language model … (a course syllabus and other course description information, i.e. course text content, is analyzed using a natural language analysis, i.e. based on said analyzing of said course text content by said…language model, to identify expected target outcomes of the course, i.e. automatically generating course learning objectives…for said course, as well as a weighting for each outcome, i.e. course learning criteria for said course (15:20-29),(18:2-40));
depicting said course learning objectives and said course learning criteria in a course learning objectives list on a display surface of a computing device, wherein each course learning objective and each course learning criteria within said course learning objective list associated with a course learning objective selection element (interactive dashboards may be displayed as a user interface on a user device, i.e. depicting…on a display surface of a computing device, to present analysis results, such as the learning outcomes and weight for each outcome, i.e. course learning objectives and said course learning criteria, may be reflected as lists in the dashboards, i.e. depicting…in a course learning objectives list, where the dashboard includes selection controls that let the user specify subsets of data to be further processed, i.e. each course learning objective and each course learning criteria within said course learning objective list associated with a course learning objective selection element (7:44-52),(9:25-51),(18:5-40),(19:57-67),(20:8-23)); and
receiving, by user interaction with said course learning objective selection element, selection of said course learning objectives and said course learning criteria for said course (the dashboard includes selection controls, i.e. said course learning objective selection element, that let the user specify subsets of data to be further processed, i.e. selecting said course learning objectives and said course learning criteria for said course by user interaction (7:44-52),(9:25-51),(18:5-40),(19:57-67),(20:8-23)).
While Zertuche provides determining learning outcomes and whether student assessments meet those outcomes, Zertuche does not specifically teach the use of a machine learning algorithm including a large language model, and thus does not teach
analyzing course text content of a course by said machine learning algorithm including a large language model using task specific course learning objective prompts;
automatically generating course learning objectives and course learning criteria….
Sridhar, however, teaches analyzing course text content of a course by said machine learning algorithm including a large language model using task specific course learning objective prompts (a combination of prompts including a user prompt including course name and goals, i.e. course text content of a course…using task specific course learning objective prompts, instructs a GPT model to generate learning objectives for the provided course information, i.e. analyzing…by a machine learning algorithm including a large language model Figs. 2 and 3,(Sec. 3.1-3.2));
automatically generating course learning objectives and course learning criteria…(a combination of prompts instructs a GPT model to generate learning objectives for the provided course information, i.e. automatically generating course learning objectives, where the learning objectives include measurable behavior, i.e. course learning criteria Figs. 2 and 3,(Sec. 1, 3.1-3.2)).
Zertuche and Sridhar are analogous art because they are from a similar field of endeavor in using systems to evaluate course materials for learning objectives. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the determining learning outcomes and whether student assessments meet those outcomes teachings of Zertuche with the use of a GPT model and a combination of prompts to generate lists of learning objectives for a course as taught by Sridhar. It would have been obvious to combine the references to enable LLMs to efficiently produce high-quality learning objectives to save instructor time (Sridhar Intro).
Regarding claim 20, Zertuche in view of Sridhar teaches claim 19, and Sridhar further teaches
said large language model using task specific course learning objective prompts does not require training or re-training to generate course learning objectives and course learning criteria for course text content associated with different courses (the system and user prompts does not identify training as part of the prompt for developing the LOs, and the system prompt for determining LOs says that the user will provide a course name and description, but is itself independent of the course description, i.e. does not require training or re-training to generate course learning objectives and course learning criteria for course text content associated with different courses Figs. 2 and 3,(Sec. 3.1-3.2)).
Where the motivation to combine is the same as previously presented.
Regarding claim 23, Zertuche in view of Sridhar teaches claim 19, and Zertuche further teaches
manual selection of said course learning objectives and course learning criteria, wherein manual selection which alters said course learning objectives and said course learning criteria of a course (the dashboard can perform selections of data points to generate a CQI ticket, i.e. manual selection, where the analysis engine may collect and analyze data from the course syllabus or other materials to determine the target outcomes to be covered by each course, i.e. said course learning objectives and said course learning criteria, where course information may be modified, such as through input of course information by the professor, i.e. alters said course learning objectives and said course learning criteria of a course (6:59-65),(7:44-52),(9:25-51), (13:49-14:4),(20:55-21:53),(24:33-50)).
Claim(s) 21 and 22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zertuche, in view of Sridhar, and further in view of Coyne.
Regarding claim 21, Zertuche in view of Sridhar teaches claim 20.
While Zertuche in view of Sridhar provides different kinds of prompting strategies, Zertuche in view of Sridhar does not specifically teach zero-shot prompts, and thus does not teach
said task specific course learning objective prompts comprise zero shot prompts.
Coyne, however, teaches said task specific course learning objective prompts comprise zero shot prompts (LLM models used for natural language processing tasks can be prompted in different ways, including zero-shot settings (Abstract, Sec. 2.2,Sec. 3)).
Where Sridhar teaches the prompts are for generating learning objectives including measurable behavior Figs. 2 and 3,(Sec. 1, 3.1-3.2).
Zertuche, Sridhar, and Coyne, are analogous art because they are from a similar field of endeavor in using language processing systems to evaluate text information. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the different prompting strategies for the LLM teachings of Zertuche, as modified by Sridhar, with the specific use of zero-shot prompts as taught by Coyne. It would have been obvious to combine the references to identify the best prompt approach depending on the model being used (Coyne Sec. 5.1).
Regarding claim 22, Zertuche in view of Sridhar and Coyne teaches claim 21, and Sridhar further teaches
said course text content comprises a transcription of oral or written words, an essay, an article, a dissertation, a manuscript, a paper, a thesis, a treatise, an exposition, a composition, or combinations thereof (the user prompt includes a course name and text describing the course and its goals, i.e. said course text content comprises a transcription of …written words Figs. 2 and 3,(Sec. 3.1-3.2)).
Where the motivation to combine is the same as previously presented.
Claim(s) 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zertuche, in view of Zheng, and further in view of Sridhar.
Regarding claim 24, Zertuche teaches
A non-transitory computer readable medium encoded with a machine learning algorithm that, when executed, cause a system to perform actions to depict course learning objectives and course learning criteria (a storage medium having instructions stored thereon which, when executed by a processor cause the processor to perform operations (2:58-66)), the actions comprising:
evaluating presentation text of a presentation by said … algorithm using task specific performance evaluation prompts, wherein evaluating presentation text comprises identifying relationships between said presentation text and said prior generated course learning objectives and prior generated course learning criteria (an assessment may be written, orally administered, or practical, such as public speaking, i.e. presentation text of a presentation, where the assessment service provides assessment data including score data, category, and outcome data including a weight for each outcome, to the analysis engine, i.e. evaluating presentation text of a presentation by said… algorithm using task specific performance evaluation prompts, where the analysis engine determines whether all the outcomes that the course purports to cover are actually being tested by the categories in the assessment for the course, and whether the weights of the outcomes corresponds to the weight of the assessment, i.e. identifying relationships between said presentation text and said prior generated course learning objectives and prior generated course learning criteria (4:15-41),(5:61-6:11),(18:2-51),(19:20-56));
automatically generating a competency score by said … algorithm using said task specific performance evaluation prompts based on the level of the identified relationships between said presentation text and each of said course learning objectives and each of said course learning criteria prior generated by said … algorithm using said task specific course learning objective prompts (the analysis engine includes one or more correlation modules that perform an analysis to correlate outcome data with assessment data, i.e. based on the level of the identified relationships between said presentation text and each of said course learning objectives and each of said course learning criteria prior generated by said…algorithm using said task specific course learning objective prompts, where the correlation can produce output data that indicates how well students are performing with respect to the learning outcomes of the course, based on the scores earned for questions on categories associated with the learning outcomes, where the categories and objectives also include weights, i.e. automatically generating a competency score (4:15-53),(6:12-34),(7:26-43),(15:20-42),(20:24-27));
automatically generating —results-- by said …algorithm using said task specific performance evaluation prompt for each course learning objective and each course learning criteria (the analysis correlates outcome data with assessment data, where the output data indicates how well students are performing with respect to the learning outcomes of the course, based on the scores earned for questions on categories associated with the learning outcomes, where the categories and objectives also include weights, i.e. by said…algorithm using said task specific performance evaluation prompt for each course learning objective and each course learning criteria, and the results of the analysis are provided on interactive dashboards or in reports, i.e. automatically generating results Figs. 4, 6, 9,(4:15-53),(6:12-34),(7:26-43),(11:38-48),(15:20-42),(20:8-27));
depicting a presentation evaluation on a display surface of said first computing device (the analysis correlates outcome data with assessment data, where the output data indicates how well students are performing with respect to the learning outcomes of the course, and the results of the analysis are provided on interactive dashboards or in reports on a user device, i.e. depicting a presentation evaluation on a display surface of said first computing device Figs. 4, 6, 9,(4:15-53),(6:12-34),(7:26-52),(11:38-48),(15:20-42),(19:57-67),(20:8-27)),
wherein said presentation evaluation depicts said course learning objectives and said course learning criteria each associated with said competency score (the analysis correlates outcome data with assessment data, where the output data indicates how well students are performing with respect to the learning outcomes of the course, based on the scores earned for questions on categories associated with the learning outcomes, where the categories and objectives also include weights, i.e. depicts said course learning objectives and said course learning criteria each associated with said competency score, and the results of the analysis are provided on interactive dashboards or in reports, i.e. said presentation evaluation depicts Figs. 4, 6, 9,(4:15-53),(6:12-34),(7:26-43),(11:38-48),(15:20-42),(20:8-27)),
wherein said presentation evaluation depicts said —results-- associated with each course learning criteria (the analysis correlates outcome data with assessment data, where the output data indicates how well students are performing with respect to the learning outcomes of the course, based on the scores earned for questions on categories associated with the learning outcomes, where the categories and objectives also include weights, i.e. depicts said results associated with each course learning criteria, and the results of the analysis are provided on interactive dashboards or in reports, i.e. said presentation evaluation depicts Figs. 4, 6, 9,(4:15-53),(6:12-34),(7:26-43),(11:38-48),(15:20-42),(20:8-27)).
While Zertuche provides determining learning outcomes and whether student assessments meet those outcomes, Zertuche does not specifically teach the use of a machine learning algorithm including a large language model to evaluate course presentations, and thus does not teach
evaluating presentation text of a presentation by said machine learning algorithm using task specific performance evaluation prompts, wherein evaluating presentation text comprises identifying relationships between said presentation text and said prior generated course learning objectives and prior generated course learning criteria;
automatically generating a competency score by said machine learning algorithm using said task specific performance evaluation prompts based on the level of the identified relationships between said presentation text and each of said course learning objectives and each of said course learning criteria …;
automatically generating supportive reasoning statements by said machine learning algorithm using said task specific performance evaluation prompt for each course learning objective and each course learning criteria;
wherein said presentation evaluation depicts said supportive reasoning statements associated with each course learning criteria.
Zheng, however, teaches evaluating presentation text of a presentation by said machine learning algorithm using task specific performance evaluation prompts, wherein evaluating presentation text comprises identifying relationships between said presentation text and said prior generated course learning objectives and prior generated course learning criteria (the LLM-based feedback generation framework uses prompts, i.e. by said machine learning algorithm using task specific performance evaluation prompts, to process raw text from student responses, i.e. evaluating presentation text of a presentation, to identify key concepts and related errors made by the student, i.e. identifying relationships between said presentation text and said prior generated course learning objectives and prior generated course learning criteria Tables 1,2,3,(Sec. 3.1-3.2));
automatically generating a competency score by said machine learning algorithm using said task specific performance evaluation prompts based on the level of the identified relationships between said presentation text and each of said course learning objectives and each of said course learning criteria …(the LLM-based feedback generation framework uses prompts, i.e. by said machine learning algorithm including said large language model using said course evaluation prompts, to identify key concepts and related errors made by the student, i.e. for each selected course learning objective and each course learning criteria based on evaluation of said submitted course presentations, to determine feedback that produces a grade and identifies competency in different areas and the change in competency over time, i.e. automatically generating competency scores Tables 1,2,3,5,9,(Sec. 3.1-3.2, 4.2));
automatically generating supportive reasoning statements by said machine learning algorithm using said task specific performance evaluation prompt for each course learning objective and each course learning criteria (the LLM-based feedback generation framework uses prompts, i.e. by said machine learning algorithm using said task specific performance evaluation prompt, to identify key concepts and related errors made by the student, i.e. for each course learning objective and each course learning criteria, to determine feedback that can include clear explanations, multimodal presentations, and different levels of comprehensiveness, i.e. automatically generating supportive reasoning statements Tables 1,2,3,5,9,11,(Sec. 3.1-3.2, 4.2));
wherein said presentation evaluation depicts said supportive reasoning statements associated with each course learning criteria (feedback includes multimodal presentations, such as explanations and visual aids, i.e. presentation evaluation depicts said supportive reasoning statements, related to the concepts and errors, i.e. associated with each course learning criteria Tables 1,2,3,5,9,11,(Sec. 3.1-3.2, 4.2)).
Zertuche and Zheng are analogous art because they are from a similar field of endeavor in using systems to evaluate educational materials. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the determining learning outcomes and whether student assessments meet those outcomes teachings of Zertuche, with the use of LLM-based feedback generation to identify key concepts and errors made by the student, as well as provide feedback as taught by Zheng. It would have been obvious to combine the references to enable personalized feedback to provide targeted guidance for different student populations that improves student engagement (Zheng (Sec. 4.2 and 5.2)).
While Zertuche in view of Zheng provides determining learning outcomes and whether student assessments meet those outcomes, Zertuche in view of Zheng does not specifically teach the use of a machine learning algorithm to generate course learning objectives and criteria, and thus does not teach
each of said course learning objectives and each of said course learning criteria prior generated by said machine learning algorithm using said task specific course learning objective prompts.
Sridhar, however, teaches each of said course learning objectives and each of said course learning criteria prior generated by said machine learning algorithm using said task specific course learning objective prompts (a combination of prompts instructs a GPT model to generate learning objectives for the provided course information, i.e. each of said course learning objectives…prior generated by said machine learning algorithm using said task specific course learning objective prompts, where the learning objectives include measurable behavior, i.e. each of said course learning criteria Figs. 2 and 3,(Sec. 1, 3.1-3.2)).
Zertuche, Zheng, and Sridhar are analogous art because they are from a similar field of endeavor in using systems to evaluate educational materials. Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the determining learning outcomes and whether student assessments meet those outcomes teachings of Zertuche, as modified by Zheng, with the use of a GPT model and a combination of prompts to generate lists of learning objectives for a course as taught by Sridhar. It would have been obvious to combine the references to enable LLMs to efficiently produce high-quality learning objectives to save instructor time (Sridhar Intro).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICOLE A K SCHMIEDER whose telephone number is (571)270-1474. The examiner can normally be reached 8:00 - 5:00 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at (571) 272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICOLE A K SCHMIEDER/Primary Examiner, Art Unit 2659
Prosecution Insights