Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Remarks
This Office Action is responsive to Applicants' Amendment filed on February 2, 2025, in which claims 1 and 8 are currently amended. Claims 1-5, 7, and 8 are currently pending.
Response to Arguments
The rejections to claims 1-5, 7, and 8 under 35 U.S.C. § 112(b) are hereby withdrawn, as necessitated by applicant's amendments and remarks made to the rejections.
Applicant’s arguments with respect to rejection of claims 1-5, 7, and 8 under 35 U.S.C. 103 based on amendment have been considered, however, are not persuasive.
With respect to Applicant's arguments on p. 14 of the Remarks submitted 2/25/2026 that Su "does not identify or classify questions into different types based on frequencies of incorrect answers, correct answers, or irregularity of correct answers", Examiner respectfully disagrees. Examiner notes that the argument imposes an artificial requirement that the claim itself does not demand. The instant claim as amended recites "identifying a plurality of different question types of the plurality of questions by using the solving result information of the user, based on patterns of user responses to the plurality of questions, wherein the plurality of different question types include a first question type, a second question type, and a third question type, and the patterns of the user responses include (i) a frequency of incorrect answers, (ii) a frequency of correct answers, and (iii) irregularity of correct answers". In Su questions are classified into types via their associated knowledge concepts, and the model is explicitly trained on irregular sequences of correct/incorrect responses (see FIG. 2) for these concepts. A recurrent model trained on such sequences necessarily captures how often answers are correct or incorrect and how consistent those outcomes are over time (the "frequency" and "irregularity"). Su's concept level mastery states and updates therefore encode response pattern-based classification behavior, satisfying the limitation under a functional reading.
With respect to Applicant's arguments on p. 14 of the Remarks submitted 2/25/2026 that Su "does not [...] assigns predetermined weights to learning data according to such identified or classified question types", Examiner does not rely on Su to disclose this limitation but rather on Liu who explicitly modifies the system in Liu for concept specific weights.
With respect to Applicant's arguments on p. 15 of the Remarks submitted 2/25/2026 that "Liu's weights are associated with knowledge concepts, not question types identified by user response patterns, nor involve assigning predetermined weights based on frequencies of incorrect answers, correct answers, or irregularity of correct answers", Examiner respectfully disagrees. Examiner asserts that distinction collapses under a reasonable reading of Liu where Liu's "knowledge concepts" are exactly the system's operative question types where each exercise is mapped to a concept and the model tracks a per-concept mastery state. Those concept states are not static labels, they are learned from the student's response patterns (right/wrong sequences) over time, i.e. how the user performs on each type of question. The impact weights Bt then scale updates differently per concept at each step, meaning the system assigns different weights to learning data based on the concept/question type as informed by those response patterns. So Liu's weights are not merely "associated with knowledge concepts", they are functionally derived from and applied to question types learned through user response behavior, which squarely meets the claim. Examiner asserts that for these reasons and those further detailed below the interpretation of the combination of Su and Liu to cover the claim limitations is reasonable and should be maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 7, and 8 are rejected under U.S.C. §103 as being unpatentable over the combination of Su (“Exercise-Enhanced Sequential Modeling for Student Performance Prediction”, 2018) and Liu (“EKT: Exercise-Aware Knowledge Tracing for Student Performance Prediction”, 2019).
Regarding claim 1, Su teaches A method for operating a learning content recommendation system, the method comprising: ([Abstract] "In online education systems, for offering proactive services to students (e.g., personalized exercise recommendation), a crucial demand is to predict student performance (e.g., scores) on future exercising activities […] we propose a new LSTM architecture to trace student states (i.e., knowledge states) in their sequential exercising process with the combination of exercise representations. For making final predictions, we design two strategies under EERNN, i.e., EERNNM with Markov property and EERNNA with Attention mechanism. Extensive experiments on large-scale real-world data clearly demonstrate the effectiveness of EERNN framework. Moreover, by incorporating the exercise correlations, EERNN can well deal with the cold start problems from both student and exercise perspectives")
transmitting question information including information on a plurality of questions to a user; ([p. 5] "an exercise eT+1 at T +1 step is posted to a student" Student interpreted as synonymous with user. Exercise eT+1 at T+1 step interpreted as question information.)
receiving solving result information that is the user's response for each of the plurality of questions;([p. 3 §3] "In an online education system, there are S students and E exercises, where students do exercises individually. We record the exercising process of each student as si = {(ei 1,ri 1),(ei 2,ri 2),...,(ei T,ri T)},where ei j represents the jth exercise solved by student i and ri j denotes the corresponding score" [p. 5] "the student applies current state hT to solve the exercise" Recording eij interpreted as receiving solving result information that is the user's (student si) response.)
training a user characteristic model by inputting a plurality of pieces of learning data to the user characteristic model, ([p. 3 §3] "Figure 2 shows the solution overview of our study. From the figure, in the training stage, we train EERNN framework by modeling all student exercising processes with the exercise texts")
each of the plurality of pieces of learning data including a piece of question information and a piece of solving result information matched with each other, ([p. 5 Eqn. 6] "at t-th step, let rt be the predicted score on exercise et through EERNN framework, rt is the actual score, thus the overall loss for a specific student is defined as: L=-sum from t=1 to T ((rt log r~t +(1−rt)log(1− r~t))." Su explicitly pairs question information rt with solving result information r~t for exercise et.)
predicting a correct answer probability for a new question input to the user characteristic model and not solved by the user by using the user characteristic model, ([p. 1 §1] "a crucial demand is to predict their performance (e.g., score), i.e., forecasting whether or not a student could answer the exercise (e.g., e5) correctly in the future" [p. 3] "in the testing stage, EERNN could predict each student performance on future exercises given her individual sequential exercising record" [p. 5] "(3) predicts the performance rT+1 on exercise eT+1 of her" See Eqn. 4-6 for how these predictions are made. In Eqn. 4 since sigmoid maps to (0,1) rT+1 which is interpreted as the probability of a correct answer)
wherein the user characteristic model is a bidirectional long short term memory (LSTM)- based artificial intelligence model,([Abstract] "we first design a bidirectional LSTM to learn each exercise representation from its text description without any expertise and in formation loss" [p. 2] "we first design a bidirectional LSTM to automatically characterize each exercise semantics by exploiting its text")
wherein the training of the user characteristic model includes: identifying a plurality of different question types of the plurality of questions by using the solving result information of the user, based on patterns of user responses to the plurality of questions, ([p. 3 §3] "Figure 2 shows the solution overview of our study. From the figure, in the training stage, we train EERNN framework by modeling all student exercising processes with the exercise texts" See also FIG. 2 student exercising record solving result information (patterns of checks or X) for each user/student)
wherein the plurality of different question types include a first question type, a second question type, and a third question type, and the patterns of the user responses include (i) a frequency of incorrect answers, (ii) a frequency of correct answers, and (iii) irregularity of correct answers, ([p. 3 §3] "Generally, if student I answers exercise j right, rij equals to 1, otherwise it equals to 0" [p. 4] "the exercise with right score (i.e., 1) and wrong score (i.e., 0) have different influences on student states during the exercising process, we need to find a appropriate way to distinguish these different effects for a specific student" [p. 4] "The student states are influenced by both the exercises and the corresponding scores she got. (2) Students usually learn and forget in their long term sequential exercising process" Su repeatedly emphasizes that student state is learned from sequences of correct/incorrect responses. A recurrent model trained over sequence is learning the frequency at which incorrect and correct occurs and the corresponding irregularity)
wherein the training of the user characteristic model includes: embedding the plurality of pieces of learning data in an embedding layer of the user characteristic model, ([p. 3 §3] "we first take Word2vec (Mikolov et al. 2013 to transform each word wi in exercise ei into a d0-dimensional pre-trained word embedding vector" [p. 5] "xj is the exercise embedding at j-th exercise step […] (2) leverage Exercise Embedding in M to extract exercise embedding xp T+1;" Word2Vec interpreted as an embedding layer of the user characteristic model generating word embedding wm. See also FIG. 3 which shows that the Bi-LSTM of the user characteristic model explicitly generates exercise embeddings and student embeddings)
inputting the embedded plurality of pieces of learning data to an LSTM layer of the user characteristic model, and ([p. 3 §3] "the forward layer with hidden word state vm-> is computed based on both the previous hidden state vm-1-> and the current word wm" See also Eqn. 1 which shows word vector wm is input into LSTM gates im, fm, om, and cm)
adjusting weights of the user characteristic model in a forward sequence of the plurality of pieces of learning data while the plurality of pieces of learning data pass through a plurality of forward LSTM cells of the LSTM layer and adjusting the weights of the user characteristic model in a backward sequence of the plurality of pieces of learning data while the plurality of pieces of learning data pass through a plurality of backward LSTM cells of the LSTM layer ([p. 3 §3] "we build a bidirectional LSTM taking the word sequence in both forward and backward directions, respectively. As illustrated in Figure 4, at each word step m, the forward layer with hidden word state vm-> is computed based on both the previous hidden state vm-1-> and the current word wm; while the backward layer updates hidden word state vm<- with the future hidden state vm+1<- and the current word wm" [p. 5] "Objective function. The whole parameters to be updated in both proposed models mainly come from three parts, i.e., parameters in Exercise Embedding {ZE w∗,ZE v∗,bE ∗}, param eters in Student Embedding {ZS x∗,ZS h∗,bS ∗} and parameters in Prediction Output {W∗,b∗}" Su explicitly states that the weight parameters are updated to minimize the loss function)
wherein the adjusting of the weights of the user characteristic model in the forward sequence of the plurality of pieces of learning data includes assigning different weights to the plurality of pieces of learning data input to the user characteristic model in the forward sequence of the plurality of pieces of learning data based on a degree of influence of each of the plurality of questions on the correct answer probability ([p. 4] "Particularly, the input weight matrix ZS x∗∈Rdh×4dv in Eq. (3)can be divided into two parts, i.e., the positive one ZS+ x∗ ∈Rdh×2dv and the negative one ZS− x∗ ∈Rdh×2dv, which can separately capture the influences of exercise ei with both right and wrong responses for a specific student during her exercising process" Splitting the weight matrix into positive and negative matrices interpreted as assigning different weights to the plurality of pieces of learning data based on a degree of influence of each of the plurality of questions on the correct answer probability)
wherein the first question type corresponds to (i) the frequency of incorrect answers in which the user provides an incorrect answer for the first question type more frequently than each of the plurality of different question types excluding the first question type, wherein the second question type corresponds to (ii) the frequency of correct answers in which the user provides a correct answer for the second question type more frequently than each of the plurality of different question types excluding the second question type, and wherein the third question type corresponds to (iii) the irregularity of correct answers in which the user irregularly provides a correct answer for questions having the third question type([p. 3 §3] "Generally, if student I answers exercise j right, rij equals to 1, otherwise it equals to 0" [p. 4] "the exercise with right score (i.e., 1) and wrong score (i.e., 0) have different influences on student states during the exercising process, we need to find a appropriate way to distinguish these different effects for a specific student" [p. 4] "The student states are influenced by both the exercises and the corresponding scores she got. (2) Students usually learn and forget in their long term sequential exercising process" Su repeatedly emphasizes that student state is learned from sequences of correct/incorrect responses. A recurrent model trained over sequence is learning the frequency at which incorrect and correct occurs and the corresponding irregularity).
However, Su does not explicitly teach and assigning different weights to the plurality of pieces of learning data based on the plurality of different question types, wherein the assigning of the different weights includes assigning a predetermined weight to each learning data of the plurality of pieces of learning data based on an identified question type corresponding to the learning data among the plurality of different question types,
wherein the adjusting of the weights of the user characteristic model in the forward sequence of the plurality of pieces of learning data includes assigning a weight to each of the first question type, the second question type and the third question type among question types of the plurality of questions in which each of the first question type, the second question type and the third question type has a higher degree of influence on the correct answer probability.
Liu, in the same field of endeavor, teaches and assigning different weights to the plurality of pieces of learning data based on the plurality of different question types, wherein the assigning of the different weights includes assigning a predetermined weight to each learning data of the plurality of pieces of learning data based on an identified question type corresponding to the learning data among the plurality of different question types,([p. 6 §4.3] "we extend the knowledge states of a certain student from the integrated vectorial representation in EERNN, i.e., ht 2 Rdh, to a matrix with multiple vectors, i.e., Ht 2 Rdh K, where each vector represents how much she has mastered an explicit knowledge concept (e.g., “Function”). Meanwhile, in EKT, we assume the student’s knowledge state matrix Ht changes over time influenced by both text content (i.e., et) and knowledge concept (i.e., kt)of each exercise" [pp. 6-7 §5] "Knowledge Embedding. Given the student’s exercising process s [...] the goal of Knowledge Embedding is to explore the impacts of each exercise on improving student states from this exercise’s knowledge concepts kt, and this impact weight is denoted by bt. Intuitively, at step t, if this exercise is related to the ith concept" [p. 8] "Fig. 6 shows the detailed process of this mastery level estimation on knowledge concepts" Liu explicitly modifies EERNN to include an impact weight Bt which is a learned gate that scales (adjusts) how much each concept (type) LSTM state is updated at step t. Liu also explicitly anticipates that each student is more or less likely to answer a question correctly based on their mastery of a given concept, EKT aggregates mastery in Eqn. 10 to predict the probability of a correct answer. See also FIG. 5 and 6.)
wherein the adjusting of the weights of the user characteristic model in the forward sequence of the plurality of pieces of learning data includes assigning a weight to each of the first question type, the second question type and the third question type among question types of the plurality of questions in which each of the first question type, the second question type and the third question type has a higher degree of influence on the correct answer probability, ([p. 6 §4.3] "we extend the knowledge states of a certain student from the integrated vectorial representation in EERNN, i.e., ht 2 Rdh, to a matrix with multiple vectors, i.e., Ht 2 Rdh K, where each vector represents how much she has mastered an explicit knowledge concept (e.g., “Function”). Meanwhile, in EKT, we assume the student’s knowledge state matrix Ht changes over time influenced by both text content (i.e., et) and knowledge concept (i.e., kt)of each exercise" [pp. 6-7 §5] "Knowledge Embedding. Given the student’s exercising process s [...] the goal of Knowledge Embedding is to explore the impacts of each exercise on improving student states from this exercise’s knowledge concepts kt, and this impact weight is denoted by bt. Intuitively, at step t, if this exercise is related to the ith concept" [p. 8] "Fig. 6 shows the detailed process of this mastery level estimation on knowledge concepts" Liu explicitly modifies EERNN to include an impact weight Bt which is a learned gate that scales (adjusts) how much each concept (type) LSTM state is updated at step t. Liu also explicitly anticipates that each student is more or less likely to answer a question correctly based on their mastery of a given concept, EKT aggregates mastery in Eqn. 10 to predict the probability of a correct answer. See also FIG. 5 and 6.).
Su as well as Liu are directed towards Bi-LSTM for Knowledge Tracing. Therefore, Su as well as Liu are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Su with the teachings of Liu by expanding EERNN (the model in Su) to include knowledge states reflecting concept mastery. Liu provides as additional motivation for combination ([p. 2 §1] “we extend EERNN and propose an explainable Exercise aware Knolwedge Tracing (EKT) framework to track student states on multiple explicit concepts simultaneously. Specifically, we extend the integrated state vector of each student to a knowledge state matrix that updates over time, where each vector represents her mastery level of a certain concept.”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 2, the combination of Su and Liu teaches The method of claim 1, wherein the adjusting of the weights of the user characteristic model in the backward sequence of the plurality of pieces of learning data includes assigning different weights to the plurality of pieces of learning data input to the user characteristic model based on the degree of influence of each of the plurality of questions on the correct answer probability (Su [p. 2437 §3] "Figure 2 shows the solution overview of our study. From the figure, in the training stage, we train EERNN Framework by modeling all student exercising processes with the exercise texts. After that, in the testing stage, EERNN could predict each student performance on future exercises given her individual sequential exercising record [...] Besides, the input weighted matrices [...] recurrent weighted matrices [...] and bias weighted vectors [...] are all the network parameters in Exercise Embedding." [p. 2442 §5] "we proposed another LSTM architecture to trace student states…For making prediction" See also Eqn. 1 which shows that the weighting is relative to the words fed into the Bidirectional LSTM.)
in the backward sequence of the sequence of the plurality of pieces of learning data input to the user characteristic model. (Su [p. 2437 §3] "To make full use of the contextual word information of each exercise, we build a bidirectional LSTM taking the word sequence in both forward and backward directions").
Regarding claim 3, the combination of Su and Liu teaches The method of claim 1, wherein the question information includes tag information on a subject matter of a question, a question type, a key word, and a text format. (Liu [p. 2 §1] "Fig. 1. Example: Left box shows the exercising process of a student, where she has already done four exercises and is going to answer exercise e5. Right table shows the corresponding materials of exercises that contain their contents and knowledge concepts." See FIG. 1 which shows each exercise has corresponding concept label (tag), knowledge concept (question type) contained in the label (tag), key word (e1, e2, etc.), and text format in exercise content).
Regarding claim 4, the combination of Su and Liu teaches The method of claim 1, further comprising: calculating a correct answer probability for a specific question based on the user characteristic model;(Su [p. 1 §1] "a crucial demand is to predict their performance (e.g., score), i.e., forecasting whether or not a student could answer the exercise (e.g., e5) correctly in the future" [p. 3] "in the testing stage, EERNN could predict each student performance on future exercises given her individual sequential exercising record" [p. 5] "(3) predicts the performance rT+1 on exercise eT+1 of her" See Eqn. 4-6 for how these predictions are made. In Eqn. 4 since sigmoid maps to (0,1) rT+1 which is interpreted as the probability of a correct answer)
and providing learning content that is expected to have higher learning efficiency than other learning content based on the calculated correct answer probability. (Liu [p. 6 §4.2] "xj is the exercise embedding at jth exercising step and hj is the corresponding student state in the history. Cosine Similarities aj are denoted as the attention scores for measuring the importance of each exercise ej in the history for new exercise eTþ1").
Regarding claim 7, the combination of Su and Liu teaches The method of claim 1, wherein the forward sequence of the plurality of pieces of learning data input to the user characteristic model is a sequence in which the user solves a question.(Su [p. 3 §3] “Figure 2 shows the solution overview of our study. From the figure, in the training stage, we train EERNN framework by modeling all student exercising processes with the exercise texts.” Examiner notes Figure 2 shows the model is trained with solving results as well as question information).
Regarding claim 8, the combination of Su and Liu teaches same as claim 1(Su same as claim 1)
a memory storing instructions; and a processor configured to execute the instructions to(Su [p. 6] "All models are implemented by PyTorch (Paszke and Chintala) using Python on a Linux server with four 2.0GHz Intel Xeon E5-2620 CPUsandaTeslaK20mGPU.Allmod els are tuned to have the best performance").
Claim 5 is rejected under U.S.C. §103 as being unpatentable over the combination of Su and Liu and Sekeroglu (“Student Performance Prediction and Classification Using Machine Learning Algorithms”, 2019).
Regarding claim 5, the combination of Su and Liu teaches The method of claim 4.
However, the combination of Su and Liu doesn't explicitly teach the providing of the learning content includes calculating a tag matching ratio with the specific question for each question based on a tag information included in each question. and providing the specific question and a question of which the calculated tag matching ratio is greater than a preset value to the user.
Sekeroglu, in the same field of endeavor, teaches The method of claim 4, wherein the providing of the learning content includes calculating a tag matching ratio with the specific question for each question based on a tag information included in each question; and providing the specific question and a question of which the calculated tag matching ratio is greater than a preset value to the user. ([p. 9 §3.1] "For 30% of testing ratio of Maths course which is the second experiment in this group, similar results are obtained as in 40% of testing ratio and highest prediction rates are achieved by SVR and lowest by BP" 30% and 40% testing ratio interpreted as preset value. See Table 1-3. R2 value is also a ratio of SSR/SST.).
The combination of Su and Liu as well as Sekeroglu are directed towards using deep learning for student performance prediction. Therefore, the combination of Su and Liu as well as Sekeroglu are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Su and Liu with the teachings of Sekeroglu by using tag matching ratios to provide questions. Sekeroglu provides as additional motivation for combination ([p. 10 §4] "Analysis of educational data especially the effect of social environment and family on the students' performance is highly important to improve the quality of education for future generations by enhancing the factors. For this reason, analysis of different and varied datasets in order to predict and classify the behaviour of students in related courses and provide early intervention to increase the performances has vital importance").
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Baraniuk (US9704102B2) discloses a machine learning method of estimating learners knowledge based on frequency of correct/incorrect responses
Saini (US20190333400A1) is directed towards a machine learning model for detecting learner propensity and for generating hint recommendations.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY VINCENT BOSTWICK whose telephone number is (571)272-4720. The examiner can normally be reached M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached on (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SIDNEY VINCENT BOSTWICK/Examiner, Art Unit 2124
/MIRANDA M HUANG/Supervisory Patent Examiner, Art Unit 2124
Prosecution Insights