{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import roc_auc_score, roc_curve\nfrom sklearn.impute import SimpleImputer as SI\n\n\nimport tensorflow as tf\nfrom tensorflow.keras import layers as L\nfrom tensorflow.keras.models import Model\nimport kerastuner as kt\nfrom kerastuner.tuners import RandomSearch\n\nss = StandardScaler()\nsi = SI(strategy=\"median\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading Data\n\nWe gotta take care of data loading as its a very huge dataset, hence we have to specify the dtypes of each column so that our RAM won't crash."},{"metadata":{"trusted":true},"cell_type":"code","source":"dtypes_dict = {'row_id': 'int64',\n               'timestamp': 'int64',\n               'user_id': 'int32', \n               'content_id': 'int16',\n               'content_type_id': 'int8',\n               'task_container_id': 'int16', \n               'user_answer': 'int8', \n               'answered_correctly': 'int8',\n               'prior_question_elapsed_time': 'float32', \n               'prior_question_had_explanation': 'boolean'\n              }\n\ntrain_df = pd.read_csv('../input/riiid-test-answer-prediction/train.csv',\n                      nrows=10**7,\n                      dtype=dtypes_dict,\n                      index_col=0)  # as row_id is same as index, I am making it default index col\ntrain_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will just look at the target cols and will make a simple NN starter model using keras.\n\nMore on EDA and interesting findings can be found here(under working): https://www.kaggle.com/mrutyunjaybiswal/understanding-the-answer-correctness-eda"},{"metadata":{},"cell_type":"markdown","source":"# Target and Features\n\n> Target\n- Did the `student(user_id)` answered the question `(answered_correctly)` from \n\n> Features\n- When? `(timestamp)`. Well, its actually the time diffrenece between the time of attempt to user's first interaction\n- a particular `content(content_id)` or \n- `type of content(content_type_id)` or \n- `task container(task_container_id)` or \n- `how much time` did S/he take to answer the previous question (or question bundle) `prior_question_elapsed_time` or \n- had he referred to any explanation or say it any tutorial for ansdwering the previous question bundle`(prior_question_had_explanation)`."},{"metadata":{"trusted":true},"cell_type":"code","source":"target_col = 'answered_correctly'\n\n# let's see with how many classes we are dealing with\ntrain_df[target_col].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ah, well! According to the host, `-1` depicts the data represents a nan value. So, we will discard those samples with `answered_correctly = -1`."},{"metadata":{"trusted":true},"cell_type":"code","source":"working_data = train_df[train_df[target_col]!=-1]\nworking_data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check for nan values if there is more."},{"metadata":{"trusted":true},"cell_type":"code","source":"working_data.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we have seen so far, there are ~39.4K nan values still in our dataset. How can we deal with that? I am thinking of handling it by not handling it. Why did I say so? We might have some way to fill up th time elapsed column, but how will we do the same for `prev_question_had_explanation` col. Though I have an idea to deal with that. Fill the nan value with the `max(no_of_false, no_of_true)` for each individual user. Well, I am skipping it for now.\n\nFor now, I am simply dropping the nan values."},{"metadata":{"trusted":true},"cell_type":"code","source":"working_data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, let's talk about below feature columns:\n\n- user_id\n- content_id\n- task_container_id\n\nLook at their value counts."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of unique users: \", working_data.user_id.nunique())\nprint(\"Number of unique content(or unique user interaction): \", working_data.content_id.nunique())\nprint(\"Number of unique tasks(or batch of lectures): \", working_data.task_container_id.nunique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Still you want to treat them as a categorical feature? Well, I don't. I am thinking of creating features per individual users/content/tasks though. "},{"metadata":{"trusted":true},"cell_type":"code","source":"userGroup = working_data.groupby(\"user_id\")[target_col].mean().reset_index()\nuserGroup","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"contentGroup = working_data.groupby(\"content_id\")[target_col].mean().reset_index()\ncontentGroup","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"taskGroup = working_data.groupby(\"task_container_id\")[target_col].mean().reset_index()\ntaskGroup","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So what I did, I kinda tried to measure performance in terms of how many times did they answer correctly per each user and content and task. Let's rename the cols and make them ready to merge with our working data. "},{"metadata":{"trusted":true},"cell_type":"code","source":"userGroup.columns = ['user_id', 'user_performance']\ncontentGroup.columns = ['content_id', 'content_performance']\ntaskGroup.columns = ['task_container_id', 'task_performance']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"working_data = working_data.reset_index()\nworking_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"working_data.loc[:, \"prior_question_elapsed_time\"] = working_data['prior_question_elapsed_time'].fillna(0)\nworking_data.loc[:, \"prior_question_had_explanation\"] = working_data['prior_question_had_explanation'].fillna(0)\nworking_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features = ['timestamp', 'prior_question_elapsed_time', 'prior_question_had_explanation']\ncat_cols = ['user_id', 'content_id', 'task_container_id']\nselected_data = working_data[features + cat_cols + [target_col]].copy()\nselected_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def preprocess(df):\n    \"\"\"\n    Merge user, task and content performance and return df with seleted features.\n    \"\"\"\n    df.loc[:, 'timestamp'] = df['timestamp'].rolling(window=5, min_periods=1, center=True).sum()\n    df.loc[:, 'prior_question_elapsed_time'] = df['prior_question_elapsed_time'].rolling(window=5, min_periods=1, center=True).sum()\n    df = df.merge(userGroup, how='left', on='user_id')\n    # deal with possible nan values\n    df.loc[:, 'user_performance'] = df['user_performance'].fillna(0.5)\n    df = df.merge(contentGroup, how='left', on='content_id')    \n    df.loc[:, 'content_performance'] = df['content_performance'].fillna(0.5)\n    df = df.merge(taskGroup, how='left', on='task_container_id') \n    df.loc[:, 'task_performance'] = df['task_performance'].fillna(0.5)\n    \n    # rescale the time values\n    df['timestamp'] = ss.fit_transform(df['timestamp'].values.reshape(-1, 1))\n    df['prior_question_elapsed_time'] = ss.fit_transform(df['prior_question_elapsed_time'].values.reshape(-1, 1))\n\n    df['prior_question_had_explanation'] = df['prior_question_had_explanation'].map({True:1, False: 0})\n    df['prior_question_had_explanation'] = si.fit_transform(df['prior_question_had_explanation'].values.reshape(-1, 1))\n\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preprocess(selected_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_features = ['timestamp',\n                  'prior_question_elapsed_time',\n                  'prior_question_had_explanation',\n                  'user_performance',\n                  'content_performance',\n                  'task_performance']\n\nfinal_train = preprocess(selected_data)[final_features + [target_col]]\nfinal_train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Modelling and Hyperparameter tunning with Keras Tuner"},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(hp):\n    inputs = tf.keras.Input(shape=(6, ))\n    x = inputs\n    x = tf.keras.layers.Dense(hp.Int('hidden_size', 30, 100, step=10, default=50), activation='relu')(x)\n    x = tf.keras.layers.Dropout(hp.Float('dropout', 0, 0.5, step=0.1, default=0.5))(x)\n    x = tf.keras.layers.Dense(hp.Int('hidden_size', 30, 100, step=10, default=50), activation='relu')(x)\n    x = tf.keras.layers.Dropout(hp.Float('dropout', 0, 0.5, step=0.1, default=0.5))(x)\n    outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n\n    model = tf.keras.Model(inputs, outputs)\n    model.compile(optimizer=tf.keras.optimizers.Adam(hp.Float('learning_rate', 1e-4, 1e-2, sampling='log')),\n                  loss='binary_crossentropy', \n                  metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tuner = RandomSearch(\n    build_model,\n    objective=\"val_accuracy\",\n    max_trials=5,\n    executions_per_trial=3\n)\ntuner.search_space_summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Cross-validation \n\n> I choose to use Stratified KFold cv with 5 splits. As there is 2:1 imbalancement between classes, this my choice of cv strategy as of now."},{"metadata":{"trusted":true},"cell_type":"code","source":"X= final_train.drop([target_col], axis=1).values\ny = final_train[target_col].values\nprint(X.shape)\nprint(y.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"cv = StratifiedKFold(n_splits=5)\nparams = {}\nmodels = {}\nfor i, (tr, val) in enumerate(cv.split(X, y)):\n    print(\"===================\")\n    print(f\"Fold: {i}\")\n    tuner.search(X[tr], y[tr],\n                 validation_data=(X[val], y[val]),\n                 epochs=20,\n                 callbacks=[tf.keras.callbacks.ModelCheckpoint(f\"model_cv{i}.h5\", save_best_only=True)])\n    \n    params[i] = tuner.get_best_models(1)[0]\n    models[i] = tuner.get_best_hyperparameters(1)[0]\n    \n    \n    pass","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation\n\nenv = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for test_df, sample_prediction_df in iter_test:\n    y_preds = []\n    test_df = preprocess(test_df)\n    x_test = test_df[final_features].values\n    \n    for model in models:\n        y_pred = model_v1.predict(x_test, verbose=1)\n        y_preds.append(y_pred)\n    \n    y_preds = sum(y_preds) / len(y_preds)\n    test_df[target_col] = y_preds\n    env.predict(test_df.loc[test_df['content_type_id'] == 0,\n               ['row_id', target_col]])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}