{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Two Feature \"Model\"\n\nThis notebook tests the very simple method of starting with the average score for the given question (`content_id`), and adding or subtracting how much the user is above or below that average score on average. \n\nNot even really a model, but should establish a solid baseline."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\nimport gc\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.linear_model import LinearRegression\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Processing\n\nBecause it's such a simple model, we only need three columns from one dataset, and we can go ahead and drop all the lecture rows. "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"dtypes = {\n    'user_id': 'int32', \n    'content_id': 'int16', \n    'answered_correctly': 'int8', \n}\ntrain = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv', usecols=[2, 3, 7], dtype=dtypes)\ntrain = train.loc[train['answered_correctly'] >= 0].copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`user_resid_rolling` how many more questions they've gotten right than expected compared to the average user."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get rolling number correct and number answered per user\n\ntrain['content_mean'] = train.groupby('content_id')['answered_correctly'].transform('mean')\ntrain['resid'] = train['answered_correctly'] - train['content_mean']\n\ntrain['user_q_count'] = train.groupby('user_id').cumcount()\ntrain['user_resid_rolling'] = train.groupby('user_id')['resid']\\\n    .transform(lambda x: x.cumsum().shift(1)).fillna(0)\n\n# Get number of questions per user for creating validation set later\nusers = train.groupby('user_id')['user_q_count'].last()\n\ntrain.head(20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Testing Model\n\nTo create a validation set that somewhat represents the test set, I select 5% of all users at random to be completely in the validation set--we know nothing about them ahead of time. I also then add the last five questions from every other user to the validation set as well.\n\nI used the `shift` method to avoid an expensive `groupby` in selecting the last five rows for each user."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Save content means\ncontent_df = train.groupby('content_id')['answered_correctly'].mean()\n\n# Select a few new users\nthresh = 30\nnew_user_num = int(0.05 * len(users))\nprint(f\"{100 * (users <= thresh).mean():.2f}% of users have {thresh} or fewer questions\")\nnew_users = np.random.choice(users[users <= thresh].index, new_user_num)\nnew_user_mask = train['user_id'].isin(new_users)\n\n# Create mask for last few questions of everyone else\nnum_q_val = 5\nlate_q_mask = train['user_id'].shift(-1 * num_q_val) != train['user_id']\n\n# Create validation and training\nuse_cols = ['user_resid_rolling', 'user_q_count', 'content_mean', 'answered_correctly', 'user_id']\nval = train.loc[new_user_mask | late_q_mask, use_cols].copy()\nval['user_resid_rolling'] = val.groupby('user_id')['user_resid_rolling'].transform('first')\nX = train.loc[~(new_user_mask | late_q_mask), use_cols].sample(1000000)\nprint(val.shape)\nprint(X.shape)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How much better or worse a student is than average probably isn't reliable until they have several observations. Here I test the model using [Additive Smoothing](https://en.wikipedia.org/wiki/Additive_smoothing), attempting to find which parameter value is the best. 5 and 10 do equally well on the validation set."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fit model\nfor m in [1, 5, 10, 20]:\n    print(m)\n    X['user_mean_resid'] = (X['user_resid_rolling']) / (X['user_q_count'] + m)\n    val['user_mean_resid'] = (val['user_resid_rolling']) / (val['user_q_count'] + m)\n    pred_cols = ['user_mean_resid', 'content_mean']\n    \n    # Fit model\n    print(\"fitting\")\n    lr = LinearRegression(fit_intercept=False)\n    lr.fit(X[pred_cols], X['answered_correctly'])\n    \n    # Make predictions\n    print(\"Predicting\")\n    y_pred_train = lr.predict(X[pred_cols])\n    y_pred_test = lr.predict(val[pred_cols])\n    print(\"Scoring\")\n    auc_train = roc_auc_score(X['answered_correctly'], y_pred_train)\n    auc_test = roc_auc_score(val['answered_correctly'], y_pred_test)\n    print(f\"Train AUC: {auc_train:.3f}\")\n    print(f\"Test AUC: {auc_test:.3f}\")\n    print(f\"-- Coeff --\\nuser_mean_resid: {lr.coef_[0]:.3f}\\ncontent_mean: {lr.coef_[1]:.3f}\\n\\n\")\n    X['model_resids'] = X['answered_correctly'] - y_pred_train\n    del y_pred_train, y_pred_test, lr\n    gc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del X, val\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generate Predictions\n\nWhen I train the model on all data, the coefficients both come out to almost 1. This is about what we would expect.\n\nInstead of multiplying the features by almost 1, I just left them as they are for prediction. The result is we start by guessing the average correct value for a question, then we add or subtract how much better or worse than average a student has been on average in the past."},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation\nenv = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"m = 10\ntrain['user_resid_mean'] = train['user_resid_rolling'] / (train['user_q_count'] + m)\nuser_df = train.groupby('user_id')[['user_resid_mean']].last()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train final model\nlr = LinearRegression()\ntrain = train.sample(10000000)\nlr.fit(train[['user_resid_mean', 'content_mean']], train['answered_correctly'])\nlr.coef_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n    test_df = test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'user_id', 'content_id']]\n    user_ids = test_df['user_id'].values\n    content_ids = test_df['content_id'].values\n    user_id_mask = np.array([user_id in user_df.index for user_id in user_ids])\n    test_df['answered_correctly'] = content_df.loc[content_ids].values\n    if sum(user_id_mask) > 0:\n        test_df.loc[user_id_mask, 'answered_correctly'] += user_df.loc[user_ids[user_id_mask], 'user_resid_mean'].values\n    env.predict(test_df[['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}