First, a couple of notes:
\nThe model consists of 3 very simple (mathematically) building blocks:
\nLearning Factor Analysis tries to determine the odds of making an error based on the number of encounters the student has had with some relevant concept so far. Performance Factor Analysis is similar, but measures the odds of making an error as a function of number of correct answers and number of wrong answers so far. Both use the same general kind of power function:
\n$$ \\mathrm{odds}(\\mathrm{error}) = \\alpha \\prod_i \\mathrm{count_i}^{\\gamma_i}$$
\nwhere
\ni is some factor being taken into account (encounters in LFA, correct/wrong answers in PFA)alpha and gamma_i are some parameters which define the \"shape\" of the the learning curve - the parameters to fit with MLcount_i is the count for that particular factor, for some student at some moment in time(e.g. \"7 encounters with this concept\" or \"4 correct answers\"In practice, this amounts to doing logistic regression with the logarithms of your count_is as input. (Taking the log of both sides of the equation so far gives you a linear equation to find the log-odds, which is exactly what logistic regression does). The intercept and coefficients from your logistic regression are log(alpha) and gamma_is in the equation above.
Side note: The papers on LFA/PFA for some reason never explicitly mention that they take the log of the counts. Not sure if they neglected to take the log, or to mention it. But the evidence from studying real-world learning curves points to the specific power function above, so the log would be necessary. And empirically, it did work better with the log.
\nI took this power law idea even further than PFA, and used five different counters for each \"concept\" (tag or, after the hierarchical clustering, sub-tag):
\nThe diagnostic/feedback split was done based on whether that particular question instance had an explanation (given by prior_question_had_explanation and then painstakingly shifted back to the question it belongs to)
I did also try combinations with fewer parameters (e.g. ignoring the feedback/diagnostic split; ignoring lectures). But the Bayesian Information Criterion (discussed in more detail in the next section) nearly always turned out better for the full split (except on one or two tags), so I went with that.
\nThere's the slight wrinkle that lots of questions actually have several tags, and so several predictions from each tag. For those, I found the most effective thing was to average the log-odds predictions for each tag and then use that as the log-odds value (which is conceptually equivalent to doing a geometric average of the odds).
\nI trained each regression on all available training data for that tag/subtag, and only ever evaluated the output on the training data itself until I actually submitted (The AUC tended to be about 0.02 better on the training data than the test data).
\nSide note: I also did a test run of a single regression over all the tags - or at least all the ones that actually intersected with each other - but it wouldn't converge, even with just two dimensions (correct/wrong). Surprisingly, even the unconverged version tended to do a tiny bit better than the equivalent per-tag average (also only correct/wrong, no sub-tag clustering). I say this is surprising because a single regression has a single intercept, which means that all users answering their very first question were lumped in the same exact group, and generally, it should be worse at distinguishing between beginner users.
\nThe goodness-of-fit for the regression varied quite a bit from tag to tag. So I decided to split the questions and lectures further into \"sub-tags\".
\nI determined the \"similarity\" of questions within the same tag like this:
\nAfter that, I used scipy's hierarchical clustering library to derive a hierachical tree where each leaf is a question/lecture, and they are progressively joined into clusters of similar questions.
\nNow I could split each tag's content into sub-tags by splitting the hierarchical tree at the root. For each tree, I decided whether to split it by re-doing the logistic regression from step one with the split, and comparing it to the quality of the regression without the split. Specifically, I calculated the Baysian Information Criterion (BIC) for each version, and chose to split if the BIC was smaller. The BIC takes into account the log-likelihood of the model (log of the chance of seeing the exact data we see, if the model is right), the number of parameters in the model, and the (log of) number of total data points. In general, a more complex model is more prone to overfitting, since there are more \"knobs\" one can turn to fit the training data precisely. But if the BIC for a more complex model is smaller than the simpler version, then the increase in complexity is probably justified by the increase in how well the model explains the data.
\nEven though the BIC is considered to be pretty harsh and pessimistic about model complexity, it happily split the tags down into over 2500 subtags, sometimes with just one question in a \"sub-tag\". I considered dialing it back artificially a bit, by introducing a bigger threshold for splitting. But in the end, I trusted the math and it seems to have worked out.
\nSide note: I also tried an alternative method for choosing what splits to attempt: instead of doing the hierarchical clustering step, for bigger tags, I tried splitting out questions based on whether they had some other tag . However, this was much worse than the hierarchical clustering: I found no splits that actually reduced the BIC, whereas the hierarchical clustering went a bit crazy with the splitting, if anything. So it seems like tags might not actually be a great way of grouping questions together for the purpose of analyzing learning curves.
\nThis is quite a simple idea:
\nIt's a bit scary how well this works. I went from 0.728 with no residuals to 0.763 with both per-user and per-question residuals.
\nLots of things I wanted to add, but ran out of time:
\nFirst, a couple of notes:
\nThe model consists of 3 very simple (mathematically) building blocks:
\nLearning Factor Analysis tries to determine the odds of making an error based on the number of encounters the student has had with some relevant concept so far. Performance Factor Analysis is similar, but measures the odds of making an error as a function of number of correct answers and number of wrong answers so far. Both use the same general kind of power function:
\n$$ \\mathrm{odds}(\\mathrm{error}) = \\alpha \\prod_i \\mathrm{count_i}^{\\gamma_i}$$
\nwhere
\ni is some factor being taken into account (encounters in LFA, correct/wrong answers in PFA)alpha and gamma_i are some parameters which define the \"shape\" of the the learning curve - the parameters to fit with MLcount_i is the count for that particular factor, for some student at some moment in time(e.g. \"7 encounters with this concept\" or \"4 correct answers\"In practice, this amounts to doing logistic regression with the logarithms of your count_is as input. (Taking the log of both sides of the equation so far gives you a linear equation to find the log-odds, which is exactly what logistic regression does). The intercept and coefficients from your logistic regression are log(alpha) and gamma_is in the equation above.
Side note: The papers on LFA/PFA for some reason never explicitly mention that they take the log of the counts. Not sure if they neglected to take the log, or to mention it. But the evidence from studying real-world learning curves points to the specific power function above, so the log would be necessary. And empirically, it did work better with the log.
\nI took this power law idea even further than PFA, and used five different counters for each \"concept\" (tag or, after the hierarchical clustering, sub-tag):
\nThe diagnostic/feedback split was done based on whether that particular question instance had an explanation (given by prior_question_had_explanation and then painstakingly shifted back to the question it belongs to)
I did also try combinations with fewer parameters (e.g. ignoring the feedback/diagnostic split; ignoring lectures). But the Bayesian Information Criterion (discussed in more detail in the next section) nearly always turned out better for the full split (except on one or two tags), so I went with that.
\nThere's the slight wrinkle that lots of questions actually have several tags, and so several predictions from each tag. For those, I found the most effective thing was to average the log-odds predictions for each tag and then use that as the log-odds value (which is conceptually equivalent to doing a geometric average of the odds).
\nI trained each regression on all available training data for that tag/subtag, and only ever evaluated the output on the training data itself until I actually submitted (The AUC tended to be about 0.02 better on the training data than the test data).
\nSide note: I also did a test run of a single regression over all the tags - or at least all the ones that actually intersected with each other - but it wouldn't converge, even with just two dimensions (correct/wrong). Surprisingly, even the unconverged version tended to do a tiny bit better than the equivalent per-tag average (also only correct/wrong, no sub-tag clustering). I say this is surprising because a single regression has a single intercept, which means that all users answering their very first question were lumped in the same exact group, and generally, it should be worse at distinguishing between beginner users.
\nThe goodness-of-fit for the regression varied quite a bit from tag to tag. So I decided to split the questions and lectures further into \"sub-tags\".
\nI determined the \"similarity\" of questions within the same tag like this:
\nAfter that, I used scipy's hierarchical clustering library to derive a hierachical tree where each leaf is a question/lecture, and they are progressively joined into clusters of similar questions.
\nNow I could split each tag's content into sub-tags by splitting the hierarchical tree at the root. For each tree, I decided whether to split it by re-doing the logistic regression from step one with the split, and comparing it to the quality of the regression without the split. Specifically, I calculated the Baysian Information Criterion (BIC) for each version, and chose to split if the BIC was smaller. The BIC takes into account the log-likelihood of the model (log of the chance of seeing the exact data we see, if the model is right), the number of parameters in the model, and the (log of) number of total data points. In general, a more complex model is more prone to overfitting, since there are more \"knobs\" one can turn to fit the training data precisely. But if the BIC for a more complex model is smaller than the simpler version, then the increase in complexity is probably justified by the increase in how well the model explains the data.
\nEven though the BIC is considered to be pretty harsh and pessimistic about model complexity, it happily split the tags down into over 2500 subtags, sometimes with just one question in a \"sub-tag\". I considered dialing it back artificially a bit, by introducing a bigger threshold for splitting. But in the end, I trusted the math and it seems to have worked out.
\nSide note: I also tried an alternative method for choosing what splits to attempt: instead of doing the hierarchical clustering step, for bigger tags, I tried splitting out questions based on whether they had some other tag . However, this was much worse than the hierarchical clustering: I found no splits that actually reduced the BIC, whereas the hierarchical clustering went a bit crazy with the splitting, if anything. So it seems like tags might not actually be a great way of grouping questions together for the purpose of analyzing learning curves.
\nThis is quite a simple idea:
\nIt's a bit scary how well this works. I went from 0.728 with no residuals to 0.763 with both per-user and per-question residuals.
\nLots of things I wanted to add, but ran out of time:
\n