{"cells":[{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport random","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Spoiler: unfortunately, the answer seems to be \"not much\", or at least \"less than I'd have liked to\".\n"},{"metadata":{},"cell_type":"markdown","source":"For this competition, I went with a relatively interpretable model ([described here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/211451)) because, well, I was interested in interpreting it from two specific points of view:\n\n1. student learning - for example, what can we learn about how different users learn this type of subject?\n2. educational content analysis - for example, what does user behavior data on the current content tell us about the concepts/skills involved in this subject, and how can can this information be used to make educational decisions?\n\nHowever, my model actually suggests that this kind of information is difficult to get using this dataset, because:\n\n1. **The between-user differences in initial skill level dwarf any within-user learning effects that are achieved**, so that even a model explicitly designed to generate learning curves primarily fits between-user differences. In fact, the learning curves derived from this data predict that a particular student's odds of answering questions correctly will mostly stay constant.\n2. **The effect of the choices the Santa app makes based on its own evaluation of the user is inextricably mixed with the effect of the actual educational content**, and  in many cases can be shown to dominate the fit of the model, as well. This makes it difficult, if not impossible, to use this dataset to draw any kind of education-related conclusion, particularly since the Santa app's decision making process is largely a black box."},{"metadata":{},"cell_type":"markdown","source":"# Within-user learning vs. between-user differentiation"},{"metadata":{},"cell_type":"markdown","source":"My model is based on Learning Factor Analysis(LFA) and its extension, Performance Factor Analysis(PFA).\n\nIn LFA, logistic regression is used to fit the following learning curve formula:\n\n$$ \\mathrm{odds}(\\mathrm{error}) = \\alpha \\cdot \\mathrm{encounters}^{\\gamma}$$\n\nWhere:\n\n- $\\alpha$ captures the average student's initial state of knowledge with the concept;\n- encounters is the number of encounters the student has had with the concept (e.g. questions answered, lectures watched)\n- $\\gamma$ is a \"learning rate\" term which affects how quickly or slowly the student gets better at answering questions within this concept. Note, in particular, that the learning curve indicates that **students get better at the task if and only if $\\gamma$ is negative**.\n\nPFA extends this idea by splitting up the encounters into questions answered correctly and questions answered incorrectly:\n\n$$ \\mathrm{odds}(\\mathrm{error}) = \\alpha \\cdot \\mathrm{correct}^{\\gamma_c} \\cdot \\mathrm{wrong}^{\\gamma_w} $$\n\n\nIn this case, $\\gamma_c$ and $\\gamma_w$ are \"learning rates\" for correct and wrong answers, respectively. This model is more nuanced, but it becomes harder to assign casuality in the same way as in LFA; for example, a positive $\\gamma_w$ should probably not be interpreted as \"students get worse at this concept each time they answer a question wrong\", but rather as \"students who have already answered lots of questions incorrectly are more likely to answer one incorrectly again\". In this way, PFA tries to capture differentiation between different students while still retaining information about students' learning rates.\n\nThere are three important special cases to consider when trying to interpret the results of PFA regression:\n1. When $\\gamma_c = \\gamma_w = \\gamma$, the PFA formula is reduced back to the LFA formula:\n$$\\mathrm{odds}(\\mathrm{error}) = \\alpha \\cdot \\mathrm{correct}^{\\gamma} \\cdot \\mathrm{wrong}^{\\gamma} = \\alpha \\cdot \\mathrm{questions}^{\\gamma}$$\nIn this case, it's reasonable to say that **students tend to learn at rate $\\gamma$ from each question encountered**, regardless of whether they answered that question correctly.\n\n2. When $\\gamma_c = - \\gamma_w$, the PFA formula is reduced to a recursive formula about the odds of making an error:\n$$\\mathrm{odds}(\\mathrm{error}) = \\alpha \\cdot \\mathrm{correct}^{\\gamma_c} \\cdot \\mathrm{wrong}^{-\\gamma_c} =\n\\alpha \\cdot  (\\frac{\\mathrm{correct}}{\\mathrm{wrong}})^{\\gamma_c} = \\alpha \\cdot \\mathrm{odds}(\\mathrm{correct})^{\\gamma_c}\n$$\nThis means that the odds of making an error do not depend on how much practice the student has had with the concept, only on the previously observed odds of making an error; in other words, **the learning curve suggests that no learning is happening**.\n\n3. When  $\\gamma_c > 0$, having answered questions correctly means that a student is **less** likely to answer future questions correctly. This suggests either some kind of \"unlearning\", or (more likely) some complex interactions between concepts, which I'll come back to in the next section.\n\n\nThe graph below summarizes these three special cases visually:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.plot([-0.8, 0], [-0.8, 0], c='green', label='equal learning rate from correct and incorrect questions')\nplt.plot([-0.8, 0], [0.8, 0], c='red', label='constant odds of making an error (no learning)')\nplt.axvspan(0, 0.5, facecolor='orange', alpha=0.3, label='unlearning effect')\n\nplt.xlabel('gamma correct')\nplt.ylabel('gamma wrong')\n\nplt.xlim(-0.8,0.5)\n\nplt.legend(bbox_to_anchor=(1.04,1), loc=\"upper left\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And here are the same special cases superimposed on a graph of the $\\gamma_c$ and $\\gamma_w$ values from a PFA regression done independently for each tag in the dataset:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"per_tag_df = pd.read_csv('../input/performance-factor-analysis-per-tag/logit_results.csv')\nper_tag_df.to_csv('logit_results.csv', index=False)\n\ncorrect_wrong_df = per_tag_df[per_tag_df['dimensions']=='full_correct full_wrong'].copy()\ncorrect_wrong_df['correct_slope']=correct_wrong_df['gammas'].apply(lambda x: float(x.split(' ')[0]))\ncorrect_wrong_df['wrong_slope']=correct_wrong_df['gammas'].apply(lambda x: float(x.split(' ')[1]))\n\nplt.plot([-0.6, 0], [-0.6, 0], c='green', label='equal learning rate from correct and incorrect questions')\nplt.plot([-0.8, 0], [0.8, 0], c='red', label='constant odds of making an error (no learning)')\nplt.axvspan(0, 0.5, facecolor='orange', alpha=0.3, label='unlearning effect')\nplt.xlabel('gamma correct')\nplt.ylabel('gamma wrong')\nplt.legend(bbox_to_anchor=(1.04,1), loc=\"upper left\")\n\nplt.xlim(-1.7,0.15)\n\n\nplt.scatter(correct_wrong_df['correct_slope'], correct_wrong_df['wrong_slope'])\n\nplt.title('Gamma coefficients of Per-tag Performance Factor Analysis')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that the learning rates tend to cluster quite tightly around the \"constant odds of making an error\" line, which indicates that for most tags, **any learning that is happening within the app is dwarfed by the variance in individual users' abilities obtained outside of the app**\n\nIn my final model,I actually used a more elaborate version of PFA, which further split out the effect of diagnostic questions vs. those that came with feedback; the $\\gamma$ graphs look similar for those splits:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fd_cw_df = per_tag_df[per_tag_df['dimensions']=='diagnostic_correct diagnostic_wrong feedback_correct feedback_wrong'].copy()\nfd_cw_df['d_c_slope']=fd_cw_df['gammas'].apply(lambda x: float(x.split(' ')[0]))\nfd_cw_df['d_w_slope']=fd_cw_df['gammas'].apply(lambda x: float(x.split(' ')[1]))\nfd_cw_df['f_c_slope']=fd_cw_df['gammas'].apply(lambda x: float(x.split(' ')[2]))\nfd_cw_df['f_w_slope']=fd_cw_df['gammas'].apply(lambda x: float(x.split(' ')[3]))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.plot([-0.6, 0], [-0.6, 0], c='green', label='equal learning rate from correct and incorrect questions')\nplt.plot([-0.8, 0], [0.8, 0], c='red', label='constant odds of making an error (no learning)')\nplt.axvspan(0, 0.5, facecolor='orange', alpha=0.3, label='unlearning effect')\nplt.xlabel('gamma correct')\nplt.ylabel('gamma wrong')\nplt.legend(bbox_to_anchor=(1.04,1), loc=\"upper left\")\n\nplt.xlim(-1.7,0.15)\nplt.title('Gamma coefficients of Per-tag PFA (feedback-only encounters)')\n\nplt.scatter(fd_cw_df['f_c_slope'], fd_cw_df['f_w_slope'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.plot([-1.0, 0], [-1.0, 0], c='green', label='equal learning rate from correct and incorrect questions')\nplt.plot([-0.6, 0], [0.6, 0], c='red', label='constant odds of making an error (no learning)')\nplt.axvspan(0, 0.5, facecolor='orange', alpha=0.3, label='unlearning effect')\nplt.xlabel('gamma correct')\nplt.ylabel('gamma wrong')\nplt.legend(bbox_to_anchor=(1.04,1), loc=\"upper left\")\n\nplt.xlim(-1.7,0.15)\nplt.title('Gamma coefficients of Per-tag PFA (diagnostic-only encounters)')\n\nplt.scatter(fd_cw_df['d_c_slope'], fd_cw_df['d_w_slope'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interestingly,the *diagnostic* encounters - those that did not come with feedback to the user about whether they got it right, and why - actually seem to fit the green \"equal learning rate\" line better than the feedback questions, as if the students learn better when they are not given feedback. I'll come back to this in the next section, as well."},{"metadata":{},"cell_type":"markdown","source":"Finally, I also split out the tags into over 2500 subtags, and performed  the same PFA-style regression on each of these subtags. At first glance, the $\\gamma$ graphs for these look somewhat better, at least for the feedback questions (the diagnostic graph is more of a round undirected blob):"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tags_df = pd.read_csv('../input/tag-splitting-by-correlated-questions/new_tags.csv')\ntags_df.to_csv('new_tags.csv', index=False)\ntags_df['questions'] =  tags_df['questions'].apply(lambda x: set([int(t) for t in x.split()]))\n\ntags_df['orig_tag'] = tags_df['tag'].apply(lambda x: x.split('.')[0])\n\ntags_df['gammas'] =  tags_df['gammas'].apply(lambda x: [float(t) for t in x.split()])\ntags_df['f_c_slope']=tags_df['gammas'].apply(lambda x: x[0])\ntags_df['f_w_slope']=tags_df['gammas'].apply(lambda x: x[1])\ntags_df['d_c_slope']=tags_df['gammas'].apply(lambda x: x[2])\ntags_df['d_w_slope']=tags_df['gammas'].apply(lambda x: x[3])\n\ntags_df['num_qs'] = tags_df['questions'].apply(lambda x: len(x))\n\ntags_df.set_index('tag', drop=False, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.plot([-2.8, 0], [-2.8, 0], c='green', label='equal learning rate from correct and incorrect questions')\nplt.plot([-2, 0], [2, 0], c='red', label='constant odds of making an error (no learning)')\nplt.axvspan(0, 3, facecolor='orange', alpha=0.3, label='unlearning effect')\nplt.xlabel('gamma correct')\nplt.ylabel('gamma wrong')\nplt.legend(bbox_to_anchor=(1.04,1), loc=\"upper left\")\n\nplt.xlim(-3,2)\nplt.title('Gamma coefficients of subtag PFA (feedback-only encounters)')\n\nplt.scatter(tags_df['f_c_slope'], tags_df['f_w_slope'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.plot([-2.8, 0], [-2.8, 0], c='green', label='equal learning rate from correct and incorrect questions')\nplt.plot([-2, 0], [2, 0], c='red', label='constant odds of making an error (no learning)')\nplt.axvspan(0, 3, facecolor='orange', alpha=0.3, label='unlearning effect')\nplt.xlabel('gamma correct')\nplt.ylabel('gamma wrong')\nplt.legend(bbox_to_anchor=(1.04,1), loc=\"upper left\")\n\nplt.xlim(-3,2.5)\nplt.title('Gamma coefficients of subtag PFA (diagnostic-only encounters)')\n\nplt.scatter(tags_df['d_c_slope'], tags_df['d_w_slope'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The $\\gamma$ coefficients of feedback-only questions seem to be more aligned with the green \"learning\" line in the subtag split; but this is because we haven't factored in *how many questions are in each subtag*. In actality, the majority of the subtags in the \"learning\" quadrant (both $\\gamma$s negative) **only have one question in them**:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.plot([-2.8, 0], [-2.8, 0], c='green', label='equal learning rate from correct and incorrect questions')\nplt.plot([-2, 0], [2, 0], c='red', label='constant odds of making an error (no learning)')\nplt.axvspan(0, 3, facecolor='orange', alpha=0.3, label='unlearning effect')\nplt.xlabel('gamma correct')\nplt.ylabel('gamma wrong')\nplt.legend(bbox_to_anchor=(1.5,1), loc=\"upper left\")\n\nplt.xlim(-3,2)\nplt.title('Gamma coefficients of subtag PFA (feedback-only encounters)')\n\nplt.scatter(tags_df['f_c_slope'], tags_df['f_w_slope'], c=np.log(tags_df['num_qs']))\nplt.colorbar(label='log(number of questions) in subtag')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It would make a lot of sense for sub-tags with only one question to be \"learnable\" without giving strong evidence for the learnability of concepts, in general: if a person has already encountered the only question in this sub-tag, and received feedback indicating which answer is correct, they **should** be much more likely to answer the same question correctly next time! The same logic applies to sub-tags that have only a few questions, since any learning effect is likely to factor in repeated encounters with the same question.\n\nBelow, I've split the same graph into \"subtags with few questions\" and \"subtags with many questions\" by using a semi-arbitrary threshold for \"many\" questions. I think this shows the trends even more clearly:"},{"metadata":{"trusted":true},"cell_type":"code","source":"many_qs = 3","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.plot([-2, 0], [-2, 0], c='green', label='equal learning rate from correct and incorrect questions')\nplt.plot([-1, 0], [1, 0], c='red', label='constant odds of making an error (no learning)')\nplt.axvspan(0, 3, facecolor='orange', alpha=0.3, label='unlearning effect')\nplt.xlabel('gamma correct')\nplt.ylabel('gamma wrong')\nplt.legend(bbox_to_anchor=(1.5,1), loc=\"upper left\")\n\nplt.xlim(-1.7,0.3)\nplt.title('Gamma coefficients of subtag PFA (feedback-only encounters, many questions)')\n\nplt.scatter(tags_df.loc[tags_df['num_qs']>many_qs,'f_c_slope'], tags_df.loc[tags_df['num_qs']>many_qs, 'f_w_slope'], c=np.log(tags_df.loc[tags_df['num_qs']>many_qs, 'num_qs']))\nplt.colorbar(label='log(number of questions) in subtag')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.plot([-3, 0], [-3, 0], c='green', label='equal learning rate from correct and incorrect questions')\nplt.plot([-1, 0], [1, 0], c='red', label='constant odds of making an error (no learning)')\nplt.axvspan(0, 3, facecolor='orange', alpha=0.3, label='unlearning effect')\nplt.xlabel('gamma correct')\nplt.ylabel('gamma wrong')\n\n\nplt.xlim(-3,2)\nplt.title('Gamma coefficients of subtag PFA (feedback-only encounters, few questions)')\n\nplt.legend(bbox_to_anchor=(1.5,1), loc=\"upper left\")\nplt.scatter(tags_df.loc[tags_df['num_qs']<=many_qs,'f_c_slope'], tags_df.loc[tags_df['num_qs']<=many_qs, 'f_w_slope'], c=np.log(tags_df.loc[tags_df['num_qs']<=many_qs, 'num_qs']))\nplt.colorbar(label='log(number of questions) in subtag')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- In both graphs, subtags with more questions tend to cluster around the \"no learning\" line\n- The \"many questions\" graph in general is a fairly tight cluster with a seemingly linear trend in the \"no learning\" direction, though with a steeper slope than the red 1:1 line itself.\n- In the \"few questions\" graph, most of the tags in the \"learning\" quadrant (negative $\\gamma$s) are those with just one question.\n"},{"metadata":{},"cell_type":"markdown","source":"## Current user state vs. last user state\n\nAnother symptom that users don't tend to learn much compared to their starting knowledge state is the behavior of my model when using different per-user inputs. \n\nI trained the model using *current* user state, and then tested it on a subset of the training data using two versions of user inputs.\n\n1. use *current* user state - the same state to which the model was fit\n2. use the *last* user state available - the last user state for that user in the training data.\n\nThe *current* user state output had an auc of about 0.74, while the *last* user state output had an auc of about 0.78. This means that **having more information about the user overall is much more important than having information about the user's current knowledge state**, even in a model that both was trained on the current knowledge state, and was expicitly designed with the goal of capturing how that knowledge state changes!\n\nThis can only be true if the user's knowledge state doesn't change that much over the course of using the app, and any change that does happen can more or less be ignored by the model without noticeable detriment."},{"metadata":{},"cell_type":"markdown","source":"# Effect of the app's choices\n\n\nTo illustrate what I mean by \"the effect of the app's choices\", let's look at the $\\gamma$ graph for diagnostic questions again:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.plot([-1.0, 0], [-1.0, 0], c='green', label='equal learning rate from correct and incorrect questions')\nplt.plot([-0.6, 0], [0.6, 0], c='red', label='constant odds of making an error (no learning)')\nplt.axvspan(0, 0.5, facecolor='orange', alpha=0.3, label='unlearning effect')\nplt.xlabel('gamma correct')\nplt.ylabel('gamma wrong')\nplt.legend(bbox_to_anchor=(1.04,1), loc=\"upper left\")\n\nplt.xlim(-1.7,0.15)\nplt.title('Gamma coefficients of Per-tag PFA (diagnostic-only encounters)')\n\nplt.scatter(fd_cw_df['d_c_slope'], fd_cw_df['d_w_slope'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that several tags are actually qute close to the green line - which means that for those tags, **users who answered  more diagnostic questions about this tag are more likely to do better on this concept in the future**. In fact, this effect is much more pronounced for diangoistic questions than for questions with feedback for the same tags. But does this mean that *users learn more from encountering diagnostic questions than questions with feedback*? That doesn't seem to make sense. \n\nTo arrive at a more plausible interpretation, we need to remember that **this data comes from an \"intelligent tutor\" app, which makes its own decision based on its own heuristics about user knowledge state**. In particular, the app makes decisions on what kinds of questions to show to a particular user, both in the diagnostic and feedback section. So, one reasonable explanation for the data (though perhaps not the only one) might be that:\n\n- The app chooses to show these tags in the diagnostic section only to those users who have demonstrated they're good at other concepts - in a sense, they are \"advanced\" users\n- Further, the diagnostic section might be *more* likely to show additional questions from these tags once the user already answered one or a few correctly.\n- In the feedback section of the app, all users end up having to answer questions with these tags\n    - The users who do better on these questions are more likely to belong to the group of \"advanced\" users identified in the diagnostic section; so, there's a correlation between seeing the tag in the diagnostic section and doing well on it in the feedback section.\n    \nAlthough this particular effect might not be the strongest, it's a good demonstration of how the app's biases (which are a bit of a black box to us) can muddy the waters when we try to use the data to draw some kind of educational conclusion."},{"metadata":{},"cell_type":"markdown","source":"## Sub-tags: what gets grouped together?\n\n\nA stronger effect can be seen when examining how the data affects the hierarchical decomposition/clustering part of my model.\n\nIn the model, I split tag content into \"sub-tags\" based on similarity in user behavior with respect to the content:\n\n- Do users tend to answer particular pairs of questions with similar chances of being correct?\n- Does watching particular lectures tend to correlate with answering particular questions correctly?\n\nThe idea behind this clustering was to **group conceptually similar content together**. However, in practice, it seems that the algorithm ends up **grouping content served to similar users** - or the users the app had decided were similar. In other words, **the model fits more to the behavior of the app than the behavior of its users**."},{"metadata":{},"cell_type":"markdown","source":"First, let's take a look at how lectures were grouped. If the grouping actually clustered conceptually-similar content, we might expect that **lectures on different topics end up in different clusters, while lectures that end up in the same cluster are about the same topic**. \n\nBut let's examine the histogram of *how many sub-clusters the lectures in each tag were split across*:"},{"metadata":{"trusted":true},"cell_type":"code","source":"num_subtags_with_lectures = tags_df.groupby('orig_tag')['lectures'].count()\nnum_subtags_with_lectures.hist()\nplt.title('For each tag, how many sub-tags end up containing lectures?')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This histogram shows that:\n\n- Around 40 tags did not have any lectures\n- **For over 140 of the tags, all lectures were lumped together into a single sub-tag**, sometimes sticking together through as many as 5 consecutive splits!\n- Only 5 tags had instances where lectures were actually split across sub-tags.\n\n\nThis could mean that either:\n\n- The overwhelming majority of lectures within each tag are all about the same concept\n\nor\n- Lectures are overwhelmingly shown to the same subset of users, and not shown to other users. Thus, the correlation primarily captures some classification of users as decided by the Santas app.\n\nThe first option seems quite unlikely: why would there be 4-5 lectures for a tag, all covering the same topic? \n\nOne further option is just that my algorithm just does a poor job of relating lectures to concepts/questions, but I don't think this is the case: Before I decided how to deal with lectures in the sub-tag split, I implemented a question split and PFA regression such that all lectures for a tag factored into *each* sub-tag's PFA. I saw that at each split, the PFA for one of the halves considered lectures quite important, while the other one considered them quite unimportant. This can be explained with the user-split hypothesis: lecture data plays a very small role in the \"lecture-unimportant\" half simply because lecturess were barely ever shown to users in that half."},{"metadata":{},"cell_type":"markdown","source":"We can further validate the hypothesis that sub-tags primarily capture some kind of implicit user categorization by examining how user activity on questions is split across sub-tags. Specifically:\n\n**Consider the initial split of questions within a tag, t, into two subtags, call them t.0 and t.1; for each user, what fraction of questions came from t.0 vs. t.1?**\n\n- If the split primarily captures conceptual differences, we would expect that most users answer a comparable number of questions from each sub-tag - a unimodal, perhaps Gaussian-looking distribution. Users get served roughly equivalent sets of questions, but they answer them differently based on their understanding of the underlying concepts.\n- However, if the split primarily captures some kind of implicit classification of the users, we would expect a bimodal distribution: the app chose to serve some users only questions from the first sub-tag, and other users only got questions from the second sub-tag.\n\nLet's look at this distribution for a particular tag:"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"split_tags_df = pd.read_csv('../input/tag-splitting-by-correlated-questions/new_tags.csv')\nsplit_tags_df['first_split_tag'] = split_tags_df['tag'].apply(lambda x: '.'.join(x.split('.')[:2]))\nper_first_split_df = split_tags_df[split_tags_df['tag'].apply(lambda x: '.'in x)].copy()\nfirst_split_qs = per_first_split_df.groupby('first_split_tag')['questions'].apply(lambda x: ' '.join(x))\nfirst_split_qs = first_split_qs.apply(lambda x: set(int(q) for q in x.split()))\nsplit_tags = set([x.split('.')[0] for x in first_split_qs.index])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def get_first_split_info(t):\n    tag_df = pd.read_csv(f'../input/riiid-training-data-broken-out-per-tag/tag_{t}.csv')\n    tag_df = tag_df[tag_df['answered_correctly']!=-1]\n    tag_df['this_question_had_explanation'].ffill(inplace=True)\n    tag_df['this_question_had_explanation'].fillna(False, inplace=True) # fallthroug default: if can't ffill, assume false\n    tag_df = tag_df[tag_df['this_question_had_explanation']]\n    \n    # count encounters for each user\n    tag_df['encounter_count'] = tag_df.groupby('user_id').cumcount()\n    \n    tag_0 = tag_df[tag_df['content_id'].isin(first_split_qs[f'{t}.0'])]\n    tag_1 = tag_df[tag_df['content_id'].isin(first_split_qs[f'{t}.1'])]\n    \n    # For each user, what fraction of questions came from sub-tag 0 vs 1?\n    user_split_counts = pd.DataFrame(index=tag_df['user_id'].unique())\n    user_split_counts['0_count'] = tag_0.groupby('user_id')['answered_correctly'].count()\n    user_split_counts['1_count'] = tag_1.groupby('user_id')['answered_correctly'].count()\n    user_split_counts.fillna(0, inplace=True)\n    user_split_counts['total'] = user_split_counts['0_count'] + user_split_counts['1_count']\n    user_split_counts['frac_0'] = user_split_counts['0_count']/user_split_counts['total']\n    return user_split_counts['frac_0']\n    # user_split_counts['frac_0'].hist(alpha=0.5, stacked=True) # TODO: return df and do one hist for stacking to actually happen\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_first_split_info('152').hist()\nplt.title('For each user, what fraction of questions from tag 152 came from sub-tag 152.0?')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Indeed, **a majority of the users were either only served questions from sub-tag 152.0, or only served questions from sub-tag 152.1**\n\nThere's also a small, secondary distribution of users that were actually served question from both sub-tags; and it seems like it could be roughly normally distributed. So this is not always a deterministic, hard classification of users.\n\nBelow is the same histogram, but stacked for 10 randomly chosen tags (10 because I did not want to wait for all the data to get processed). The pattern seems to persist across tags."},{"metadata":{"trusted":true},"cell_type":"code","source":"split_10 = [get_first_split_info(t) for t in random.sample(split_tags, 10)]\nplt.hist(split_10, stacked=True)\nplt.title('Sub-tag split histogram for 10 random tags')\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I think it's worth highlighting the two separate implications here, both of which are significant:\n\n- The Santa app tends to categorize users and only serve each class of users a subset of questions and lectures\n- Trying to capture conceptual clusters of content primarily captures this classification of users"},{"metadata":{},"cell_type":"markdown","source":"## \"Unlearning\" effect in multiple-tag regression\n\nAnother symptom of this \"user classification\" pattern by the Santa app can be seen in an alternative PFA model I didn't end up using for my final inference.\n\nThe PFA model I described so far (which is the one I used in my final model) fits each tag separately. For questions with multiple tags, it averages the predictions from each tag. \n\nI also tried doing a single regression on all questions that had more than one tag, to capture any interactions that may exist between the different tags. This regression actually tended to produce slightly better AUC results than the equivalent per-tag regression, but was much more unwieldy and difficult to scale up to the more advanced forms of PFA.\n\nThe graph of per-tag $\\gamma$ coefficients from the resulting regression are shown below."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"multitag_df = pd.read_csv('../input/multitag-full-regression-separate-coeffs/full_regression_coefs.csv')\nmultitag_df.to_csv('multitag_regression_coefs.csv')\nmultitag_df['tag'] = multitag_df['param'].apply(lambda x: int(x.split('_')[2]))\n\nmultitag_gammas = pd.DataFrame(index=multitag_df['tag'].unique())\nmultitag_gammas['full_correct'] = multitag_df[multitag_df['param'].apply(lambda x: x.startswith('full_correct_'))].set_index('tag')['value']\nmultitag_gammas['full_wrong'] = multitag_df[multitag_df['param'].apply(lambda x: x.startswith('full_wrong_'))].set_index('tag')['value']","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.plot([-0.6, 0], [-0.6, 0], c='green', label='equal learning rate from correct and incorrect questions')\nplt.plot([-0.6, 0], [0.6, 0], c='red', label='constant odds of making an error (no learning)')\nplt.axvspan(0, 0.7, facecolor='orange', alpha=0.3, label='unlearning effect')\nplt.xlabel('gamma correct')\nplt.ylabel('gamma wrong')\nplt.legend(bbox_to_anchor=(1.04,1), loc=\"upper left\")\n\nplt.xlim(-0.7,0.6)\nplt.title('Gamma coefficients of single regression for all \"co-occurring\" tags')\n\nplt.scatter(multitag_gammas['full_correct'], multitag_gammas['full_wrong'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see from the graph that when tag are allowed to interact in the model, **significantly more tags end up in the \"unlearning effect\" region**, where having more experience with a given tag is correlated with doing worse on related questions. In fact, 23 out of the 114 tags involved in this regression ended up in this region - that's more than 1 in 5.\n\nGiven that we already know the Santa app tends to separate users into groups and only serve certain questions to certain users, it seems plausible that this effect has to do with *what kinds of users these particular tags get served to*. One plausible explanation is that **questions from these tags are primarily shown to \"remedial\" users, and thus a user who has seen lots of this type of question is more likely to do worse on any given question, in general**. On the other hand, the opposite explanation is equally plausible - perhaps **users who  have answered this type of question correctly are subsequently considered \"advanced\", and will be asked more difficult questions, thus bringing their overall chance of correctness down**. \n\nAnd that really highlights why this app behavior factor makes it so difficult and dangerous to try and draw educational conclusions from this data."},{"metadata":{},"cell_type":"markdown","source":"# Conclusion\n\nThe fact that users don't seem to learn much from the app compared to what they already know coming into the app is perhaps not very surprising. Santa is a test-prep app for an english-language test, not a language-learning app. It operates exclusively through asking questions that might appear on the test, and occasionally showing short explanatory videos. Users can't (and probably don't) actually expect to learn the Engish language from scratch using this app.\n\n\nThe second part - the effect of the app's (black box) behavior on the nature of the data that is produced, and on its usefulness for drawing educational conclusions - is much more concerning.\n\n- Any model trained on the data is in danger of fitting primarily to the app's behavior, making it much less sensitive to any interesting effect(s) of learner behavior. I've demonstrated that this is the case with my own model, even though I was explicitly trying to capture these interesting effects of learner behavior.\n- Moreover, even when there seem to be some learning effects captured by the model, the results can be difficult to interpret correctly, as I've illustrated above: does asking diagnostic questions for certain tags teach people more effectively than asking those same questions and then giving feedback? does gaining experience in certain tags make it harder to answer questions with related tags? The answer to both of those is \"probably not\", simply because that doesn't make much sense from a learning perspective; and plausible explanations exist from an app-behavior perspective. But it seems likely that there are lots of other, less obviously wrong educational conclusions one can draw, that are nevertheless better explained by the app's behavior.\n- Falsifiability: If models based on this data primarily fit to the app's underlying assumptions about how users behave, to what extent is this data useful in verifying or disproving those assumptions? Would any model trained on this data implicitly treat the Santa app's assusmptions as axiomatic? For example, if there's very little data on how a certain class of users answer a certain type of question, can we trust any model built on that data to predict how that user would answer that question?\n\n\n Actually, I'd be curious to see if other people's models also show symptoms of these issues. Although it's harder to interpret the inner workings of NNs or even LGBMs, I think there are some potentially interesting things to examine just using the output of any model, e.g.:\n - Does your model perform better on *current* user data, or the *last available* data for that user? Does having more overall information about the user outweigh having precise information about the user's *current* knowledge state?\n - Given the data about the way that Santa seems to classify its users (e.g. the tag splits where most users are only ever shown quetions from one of the sub-tags), does your model perform better on instances that \"obey\" this implicit classification, or the relatively small number of cases that \"go against\" it? To what extent does your model achieve its fit from capturing and \"agreeing\" with the app's own heuristics? \n \n \nI've saved most of the data I use above as output of this notebook (the only exception is the dataset of per-tag subsets of the training data, but that can be derived directly from the training data file). This includes all coefficients and intercepts from all versions of logistic regressions,as well as all data about the sub-tag splits of the tags. Please play around with it if you are interested, and I'm happy to answer any questions about it."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}