DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 – 18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Step 1: Is the claimed invention a statutory category of invention?
Claims 1, 7 and 13 are directed to a method / computer program / system for generating assessment questions automatically (Step 1, Yes).
Step 2A, Prong 1: Does the claim recite an abstract idea?
The limitation of steps: … receiving input data related to a user-specified topic, wherein the input data comprises a user-specified learning objective; generating a set of multiple-choice questions (MCQs) based on the received input data using an AI model; analyzing the generated MCQs for errors using the AI model by: detecting and discarding generated MCQs having problematic distractors, wherein problematic distractors include distractors that do not satisfy length or difficulty criteria, and ensuring compliance with formatting standards; evaluating the generated MCQs for alignment with the user-specified learning objective and adherence to predefined quality criteria using an evaluation AI model, wherein the generated MCQs are evaluated as accepted or rejected; discarding any generated MCQs evaluated as rejected by the evaluation AI model; determining whether there are a target number of generated MCQs evaluated as accepted by the evaluation AI model; in response to having less than the target number of generated MCQs evaluated as accepted, repeating the generation, analysis, and evaluation processes until there are a target number of generated MCQs evaluated as accepted; and outputting the target number of generated MCQs evaluated as accepted by the evaluation AI model as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components. The claimed method akin to mental process of observations, evaluations, and judgements. The mere nominal recitation of computer components for performing these steps does not take the claim limitation outside of the mental processes grouping. Thus, the claim recites a mental process (Step 2A, Prong 1: yes).
Step 2A, Prong 2: Does the claim recite additional elements that integrate the judicial exception into a practical application?
Per the 2019 Revised Patent Subject Matter Eligibility Guidance, if a claim as a whole integrates the recited judicial exception into a practical application of that exception, a claim is not "directed to" a judicial exception. Alternatively, a claim that does not integrate a recited judicial exception into a practical application is directed to the exception. Evaluating whether a claim integrates an abstract idea into a practical application is performed by a) identifying whether there are any additional elements recited in the claim beyond the abstract idea, and b) evaluating those additional elements individual and in combination to determine whether they integrate the abstract idea into a practical application, using one or more of the considerations laid out by the Supreme Court and the Federal Circuit. Exemplary considerations indicative that an additional element (or combination of elements) may have or has not been integrated into a practical application are set forth in the 2019 PEG.
With respect to the instant claims, claims 1, 7 and 13 recite the additional elements of: a computer system; a non-transitory computer-readable medium comprising a computer program, that, when executed by a computer system and the system comprising: one or more processors; one or more memory units coupled to the one or more processors. It is particularly noted that the use of processor "as a tool" to perform an abstract method are indicated in the 2019 PEG as examples that an additional element has not been integrated into a practical application. Even in combination, the recited additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits, such as an improvement to a computing system, on practicing the abstract idea (STEP 2A, Prong 2: NO).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
Claims 1, 7 and 13 recite the additional elements of: a computer system; a non-transitory computer-readable medium comprising a computer program, that, when executed by a computer system and the system comprising: one or more processors; one or more memory units coupled to the one or more processors set forth above for Step 2A, Prong 2. Regarding these limitations: Applicant's specification describes these features in generic manner "… The methods described with respect to FIGS. 1-2 are embodied in software and performed by a computer system (comprising one or more computing devices) executing the software. A person skilled in the art would understand that a computer system has one or more memory units, disks, or other physical, computer-readable storage media for storing software instructions, as well as one or more processors for executing the software instructions” in the Applicant’s published application, para. [0041]). There is no indication in the Specification that Applicants have achieved an advancement or improvement in computer for generating MCQs. Dependent claims 2 – 6, 8 – 12, 14 – 18 inherit the deficiencies of their respective parent claims through their dependencies and do not recite additional limitations sufficient to direct the claims to more than the claimed abstract idea, and are thus rejected for the same reasons.
Claim Objections
Claims 1 – 3, 7 – 9, 13 – 15 are objected to because of the following informalities:
The abbreviated term “AI model” was not clearly explained in the claims.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 – 2, 4, 7 – 8, 10, 13 – 14 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Katirai et la. (US 12,412,484 B1) in view of Xue et al. (US 2025/0363908 A1).
Re claims 1, 7, 13:
Katirai teaches 1. A method, performed by a computer system, for generating assessment questions automatically (Katirai, Abstract), comprising:
receiving input data related to a user-specified topic, wherein the input data comprises a user-specified learning objective (Katirai, col. 13, line 10 – col. 14, line 19, “Topic T”; col. 20, lines 56 - 59);
generating a set of multiple-choice questions (MCQs) based on the received input data using an AI model (Katirai, figs. 3A – 3C; Abstract, “LLM”);
analyzing the generated MCQs for errors using the AI model (Katirai, col. 22, line 62 – col. 23, line 32; figs. 2A – 2E) by:
detecting and discarding generated MCQs having problematic distractors, wherein problematic distractors include distractors that do not satisfy length or difficulty criteria (Katirai, col. 24, “TABLE 4”, “too easy”; col. 27, line 1 – col. 28, line 9, “Choices for the cognitive and difficulty level tags are also set out in the Al prompt”), and
ensuring compliance with formatting standards (Katirai, col. 10, lines 1 – 9, “standardized exams allow only certain question types (QT) (e.g., format types)”; col. 11, lines 10 – 24, “QT formats”);
evaluating the generated MCQs for alignment with the user-specified learning objective and adherence to predefined quality criteria using an evaluation AI model (Katirai, col. 20, lines 56 – 58, “(3) LEARNING_OBJECTIVE-Natural language text explanation of a learning objective associated with the question”; col. 26, lines 28 – 51; col. 30 – 31, “TABLE 6”),
wherein the generated MCQs are evaluated as accepted or rejected (Katirai, col. 23, line 34 – col. 24, line 32);
discarding any generated MCQs evaluated as rejected … (Katirai, col. 23, line 34 – col. 24, line 32, “"accept" or "reject" the presented Question Unit Q”); and outputting the target number of generated MCQs evaluated as accepted (Katirai, col. 6, lines 61 – 67, “These outputs can include natural language examination questions”; col. 23, lines 34 - 38).
Katirai teaches 7. A non-transitory computer-readable medium comprising a computer program, that, when executed by a computer system, enables the computer system to perform the following steps for generating assessment questions automatically (Katirai, Abstract), the steps comprising:
receiving input data related to a user-specified topic, wherein the input data comprises a user-specified learning objective (Katirai, col. 13, line 10 – col. 14, line 19, “Topic T”; col. 20, lines 56 - 59);
generating a set of multiple-choice questions (MCQs) based on the received input data using an AI model (Katirai, figs. 3A – 3C; Abstract, “LLM”);
analyzing the generated MCQs for errors using the AI model by (Katirai, col. 22, line 62 – col. 23, line 32; figs. 2A – 2E):
detecting and discarding generated MCQs having problematic distractors, wherein problematic distractors include distractors that do not satisfy length or difficulty criteria (Katirai, col. 24, “TABLE 4”, “too easy”; col. 27, line 1 – col. 28, line 9, “Choices for the cognitive and difficulty level tags are also set out in the Al prompt”), and
ensuring compliance with formatting standards (Katirai, col. 10, lines 1 – 9, “standardized exams allow only certain question types (QT) (e.g., format types)”; col. 11, lines 10 – 24, “QT formats”);
evaluating the generated MCQs for alignment with the user-specified learning objective and adherence to predefined quality criteria using an evaluation AI model (Katirai, col. 20, lines 56 – 58, “(3) LEARNING_OBJECTIVE-Natural language text explanation of a learning objective associated with the question”; col. 26, lines 28 – 51; col. 30 – 31, “TABLE 6”),
wherein the generated MCQs are evaluated as accepted or rejected (Katirai, col. 23, line 34 – col. 24, line 32);
discarding any generated MCQs evaluated as rejected … (Katirai, col. 23, line 34 – col. 24, line 32, “"accept" or "reject" the presented Question Unit Q”); … outputting the target number of generated MCQs evaluated as accepted (Katirai, col. 6, lines 61 – 67, “These outputs can include natural language examination questions”; col. 23, lines 34 - 38).
Katirai teaches 13. A computer system for generating assessment questions automatically (Katirai, Abstract), the system comprising:
one or more processors; one or more memory units coupled to the one or more processors, wherein the one or more memory units store instructions that, when executed by the one or more processors (Katirai, col. 52, line 66 – col. 53, line 45), cause the system to perform the operations of:
receiving input data related to a user-specified topic, wherein the input data comprises a user-specified learning objective (Katirai, col. 13, line 10 – col. 14, line 19, “Topic T”; col. 20, lines 56 - 59);
generating a set of multiple-choice questions (MCQs) based on the received input data using an AI model (Katirai, figs. 3A – 3C; Abstract, “LLM”);
analyzing the generated MCQs for errors using the AI model by (Katirai, col. 22, line 62 – col. 23, line 32; figs. 2A – 2E):
detecting and discarding generated MCQs having problematic distractors, wherein problematic distractors include distractors that do not satisfy length or difficulty criteria (Katirai, col. 24, “TABLE 4”, “too easy”; col. 27, line 1 – col. 28, line 9, “Choices for the cognitive and difficulty level tags are also set out in the Al prompt”), and
ensuring compliance with formatting standards (Katirai, col. 10, lines 1 – 9, “standardized exams allow only certain question types (QT) (e.g., format types)”; col. 11, lines 10 – 24, “QT formats”);
evaluating the generated MCQs for alignment with the user-specified learning objective and adherence to predefined quality criteria using an evaluation AI model (Katirai, col. 20, lines 56 – 58, “(3) LEARNING_OBJECTIVE-Natural language text explanation of a learning objective associated with the question”; col. 26, lines 28 – 51; col. 30 – 31, “TABLE 6”),
wherein the generated MCQs are evaluated as accepted or rejected (Katirai, col. 23, line 34 – col. 24, line 32);
discarding any generated MCQs evaluated as rejected … (Katirai, col. 23, line 34 – col. 24, line 32, “"accept" or "reject" the presented Question Unit Q”); … and outputting the target number of generated MCQs evaluated as accepted (Katirai, col. 6, lines 61 – 67, “These outputs can include natural language examination questions”; col. 23, lines 34 - 38).
Katirai et la. (US 12,412,484 B1) teaches the generated MCQs are evaluated as accepted or rejected by user. Katirai does not explicitly disclose discarding any generated MCQs evaluated as rejected by the evaluation AI model; nor disclose a target number of generated MCQs.
Xue teaches a question generation apparatus includes: an input unit that acquires context data including text information; a question generation unit that generates a first set of multiple-choice questions for the context data by processing the context data using a first large language model; an evaluation unit that determines difficulty level evaluation values indicating cognitive difficulty levels for the first set of multiple-choice questions by evaluating the first set of multiple-choice questions based on a predetermined cognitive difficulty level evaluation criterion (Xue, Abstract).
Xue teaches
wherein the generated MCQs are evaluated as accepted or rejected (Xue, fig. 7, S706 - “Yes” or “No”; [0075]);
discarding any generated MCQs evaluated as rejected by the evaluation AI model (Xue, [0075], “excluding multiple-choice questions that do not satisfy the cognitive difficulty level threshold from among the set 308 of multiple-choice questions”; [0117], “The filtering unit 229 generates a subset of multiple-choice questions by excluding the multiple-choice questions that do not satisfy the first cognitive difficulty level threshold from the set 302 of multiple-choice questions”; col. 51, lines 20 – 29, “automated examination question generation system is described that leverages large language models (LLMs) like GPT-4o to enhance the efficiency of creating exam questions. The system addresses various challenges inherent in LLM-based question generation, such as overfitting, accuracy, ambiguity, and question diversity. The proposed method includes generating patient case scenarios for examinations (e.g., nursing certification exams) and generating multiple-choice, case study, and select-all-that-apply questions”);
determining whether there are a target number of generated MCQs evaluated as accepted by the evaluation AI model (Xue, fig. 7, S708, nmax; [0122], “the number of times the question generation processing may be repeated”);
in response to having less than the target number of generated MCQs evaluated as accepted, repeating the generation, analysis, and evaluation processes until there are a target number of generated MCQs evaluated as accepted (Xue, fig. 7, S708, nmax; [0122], “the number of times the question generation processing may be repeated”); and
outputting the target number of generated MCQs evaluated as accepted by the evaluation AI model (Xue, [0075], “output the subset 312 of multiple-choice questions to the user terminal 260”).
Therefore, in view of Xue, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method/system/computer program described in Katirai, by excluding and selecting questions using AI model as taught by Xue, since automating the creation of multiple-choice questions will significantly reduce the amount of manpower, time, cost, and effort required (Xue, [0004]) and leveraged by educators wishing to take advantage of the state-of-the-art generative models to support MCQ authoring efforts (Xue, [0007]).
Therefore, in view of Xue, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method/system/computer program described in Katirai, by generating a number of MCQs as taught by Xue, since it was known in the art to generate a predefined number of questions for a test.
Re claims 2, 8, 14:
2. The method of claim 1, wherein the AI model is an LLM and the evaluation AI model is a fine-tuned LLM that is fine-tuned based on an example set of acceptable and unacceptable MCQs and corresponding reasons. 8. The non-transitory computer-readable medium of claim 7, wherein the AI model is an LLM and the evaluation AI model is a fine-tuned LLM that is fine-tuned based on an example set of acceptable and unacceptable MCQs and corresponding reasons. 14. The computer system of claim 13, wherein the AI model is an LLM and the evaluation AI model is a fine-tuned LLM that is fine-tuned based on an example set of acceptable and unacceptable MCQs and corresponding reasons (Katirai, col. 54, lines 5 – 24; col. 24, lines 21 – 32, “API Server 106 determines that user's review results indicate that refined Question Unit Q has been rejected ( e.g., Accepted=no ), then the API Server 106 will return to Operation 266 and generate a further RI 25 prompt for LLM 114 that incorporates user feedback received in Operation 267C … RI prompt to rephrase that specific section of the question to address the declared deficiency and/or suggested improvements”).
Re claims 4, 10, 16:
4. The method of claim 1, wherein the input data comprises a certification/subject, domain, subdomain, and the target number of requested MCQs. 10. The non-transitory computer-readable medium of claim 7, wherein the input data comprises a certification/subject, domain, subdomain, and the target number of requested MCQs. 16. The computer system of claim 13, wherein the input data comprises a certification/subject, domain, subdomain, and the target number of requested MCQs (Katirai, col. 8, lines 24 – 59, “Scenario Type”, “Scenario Category”, “Scenario Sub-Category”; col. 9, lines 11 – 39, “Each <Scenario Category, Scenario Sub-Category> pair (e.g., <System, CN> or <Concept, CN>) can be associated with respective hierarchical pre-defined groups of further sub-categories and sub-sub categories”, “Level 2<L2>”, “Level 3<L3>”, “Level 4<L4>”; Xue, fig. 7, S708, nmax; [0122], “the number of times the question generation processing may be repeated”).
Claims 3, 9 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Katirai et la. (US 12,412,484 B1) in view of Xue et al. (US 2025/0363908 A1) as applied to claims 1, 7 and 13 above, and further in view of Ahmad et al. (US 2022/0284528 A1).
Re claims 3, 9, 15:
Katirai does not explicitly disclose generating the set of MCQs comprises training the AI model by inputting the example MCQs and outputting an AI-determined learning objective.
Ahmad teaches a system and a method for assigning a student to a cohort in real time on a platform (Ahmad, Abstract). Ahmad teaches 3. The method of claim 1, wherein the input data comprises example MCQs, and wherein generating the set of MCQs comprises training the AI model by inputting the example MCQs and outputting an AI-determined learning objective. 9. The non-transitory computer-readable medium of claim 7, wherein the input data comprises example MCQs, and wherein generating the set of MCQs comprises training the AI model by inputting the example MCQs and outputting an AI-determined learning objective. 15. The computer system of claim 13, wherein the input data comprises example MCQs, and wherein generating the set of MCQs comprises training the AI model by inputting the example MCQs and outputting an AI-determined learning objective (Ahmad, [0005], “Further, a question may be received from the student. The question may be received in at least an image form, a textual form, and an audio form. Furthermore, a plurality of parameters may be extracted from the question based on a machine learning model. The plurality of parameters may comprise a subject, a topic, a sub-topic, a category, and a language of the question. It may be understood that the question is numerical, conceptual and analytical”; [0018], “the system extracts a plurality of parameters from the question based on a machine learning model”). Therefore, in view of Ahmad, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method / computer program / system described in Katirai, by analyzing input questions from the students as taught by Ahmad, in order to promote effective and focused learning for the students, enable the students to associate with similar students in the cohort and simplify the learning experience for the student, help the student to evaluate their academic performance in real time and enable the student to learn as per the language of preference, time of preference, and level of understanding (Ahmad, [0082] – [0088]).
Claims 5, 11 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Katirai et la. (US 12,412,484 B1) in view of Xue et al. (US 2025/0363908 A1) as applied to claims 1, 7 and 13 above, and further in view of Attali et al. (US 2023/0080674 A1).
Re claims 5, 11, 17:
Katirai does not explicitly disclose wherein analyzing the generated MCQs for errors also comprises detecting and discarding generated MCQs having a similarity score greater than a threshold to the relevant external documents.
Attali teaches systems, apparatuses, and methods for automatically generating text items that may be used on an exam or test (Attali, Abstract). Attali teaches 5. The method of claim 4, wherein the input data also comprises relevant external documents, and wherein analyzing the generated MCQs for errors also comprises detecting and discarding generated MCQs having a similarity score greater than a threshold to the relevant external documents. 11. The non-transitory computer-readable medium of claim 10, wherein the input data also comprises relevant external documents, and wherein analyzing the generated MCQs for errors also comprises detecting and discarding generated MCQs having a similarity score greater than a threshold to the relevant external documents. 17. The computer system of claim 16, wherein the input data also comprises relevant external documents, and wherein analyzing the generated MCQs for errors also comprises detecting and discarding generated MCQs having a similarity score greater than a threshold to the relevant external documents (Attali, [0039] – [0056], “Filter the generated test items (the possible correct and/or incorrect responses) to remove categories of subject matter that are not appropriate or do not satisfy one or more desired metrics … [0049] similarity to the source text; [0050] similarity to the chosen correct answer; [0051] similarity to unchosen potential correct answers; [0052] similarity to other incorrect answers”). Therefore, in view of Attali, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method/computer program/system described in Attali, by filtering generated test items based on similarity to source text as taught by Attali, since Attali explains that one can
compute the textual similarity between a generated Alternative Passage and the corresponding Source Passage. This may be helpful in filtering out those Alternative Passages that are "too similar" to the Source Passage (i.e., those that might inadvertently produce a correct answer to the Source passage), as well as those that are too different (i.e., those that would produce answers that are significantly and obviously different from the correct one for a Source Passage (Attali, [0164]).
Claims 6, 12 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Katirai et la. (US 12,412,484 B1) in view of Xue et al. (US 2025/0363908 A1) as applied to claims 1, 7 and 13 above, and further in view of Galitsky (US 2022/0075954 A1).
Re claims 6, 12, 18:
Katirai teaches wherein the predefined quality criteria comprise detecting whether multiple answer choices are correct, whether an answer choice has insufficient or unhelpful feedback (Katirai, col. 33, line 34 – col. 34, line 49). Katirai does not explicitly disclose the predefined quality criteria comprise detecting whether multiple answer choices are correct, whether an answer choice has a similarity score greater than a threshold to an MCQ stem. Galitsky teaches an invention related to generating questions from text (Galitsky, Abstract). Galitsky teaches 6, 12, 18 wherein the predefined quality criteria comprise detecting whether multiple answer choices are correct, whether an answer choice has a similarity score greater than a threshold to an MCQ stem (Galitsky, [0207] – [0208]). Therefore, in view of Galitsky, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method / computer program / system described in Katirai, by providing threshold to a MCQ stem as taught by Galitsky, since fully automatic method for generating distractor suggestions for multiple-choice questions used in high-stakes medical exams has been proposed. The system uses a question stem and the correct answer as an input and produces a list of suggested distractors ranked based on their similarity to the stem and the correct answer, combining concept embeddings with information retrieval methods (Galitsky, [0207]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JACK YIP whose telephone number is (571)270-5048. The examiner can normally be reached Monday thru Friday; 9:00 AM - 5:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XUAN THAI can be reached at (571) 272-7147. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACK YIP/ Primary Examiner, Art Unit 3715