{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Riiid classification using NN & Random Forest"},{"metadata":{},"cell_type":"markdown","source":"The initial purpose of this notebook was to make a very simple baseline for my own usage to better understand the data and the submission process. As I got a fairly decent score (given the simplicity of the model) I decided to share the notebook in case it can be helpful to someone else. This model could be improved by adding more features."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Dense\nfrom numpy import loadtxt\nimport tensorflow as tf\nfrom sklearn.model_selection import train_test_split\nimport keras\nimport pandas as pd\nimport numpy as np\nimport riiideducation\nimport matplotlib.pyplot as plt\nfrom keras.callbacks import EarlyStopping","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Read the data\n<p>I read the data in feather format from this notebook https://www.kaggle.com/aralai/riiid-feather-dataset. It's much faster! \nIf you don't want to use feather, you can just replace the following lines with the csv format reading."},{"metadata":{"trusted":true},"cell_type":"code","source":"def reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### %%time\ntrain = pd.DataFrame();\nchunks = []\ncol_list = [\"user_id\", \"content_id\",\"answered_correctly\"]\nfor chunk in pd.read_csv('../input/riiid-test-answer-prediction/train.csv', usecols=col_list, iterator=True, chunksize= 5000, header=0):\n    chunks.append(reduce_mem_usage(chunk))\n\ntrain = pd.concat(chunks, ignore_index=True)\n\n\nquestions = reduce_mem_usage(pd.read_feather('../input/riiid-feather-dataset/questions.feather'))\nlectures = reduce_mem_usage(pd.read_feather('../input/riiid-feather-dataset/lectures.feather'))\nexample_test = reduce_mem_usage(pd.read_feather('../input/riiid-feather-dataset/example_test.feather'))\nexample_sample_submission = reduce_mem_usage(pd.read_feather('../input/riiid-feather-dataset/example_sample_submission.feather'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_list = ['content_id','prior_question_elapsed_time','prior_question_had_explanation']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_train_df(df):\n    questions_dataframe = prepare_train_features(df,'content_id')\n    questions_dataframe['content_id'] = list(questions_dataframe.index)\n    questions_dataframe = questions_dataframe.rename_axis(\"question_index\")\n    users_dataframe = prepare_train_features(df,'user_id')\n    users_dataframe['user_id'] = list(users_dataframe.index)\n    users_dataframe = users_dataframe.rename_axis(\"user_index\")\n    df = pd.merge(df,users_dataframe,how = 'inner',on = 'user_id')\n    df = pd.merge(df,questions_dataframe,how = 'inner',on = 'content_id')\n    df = df[df['answered_correctly'] != -1]\n    df['prior_question_elapsed_time'].fillna(-1,inplace = True)\n    df['prior_question_had_explanation'].fillna(-1,inplace = True)\n    df['prior_question_had_explanation'] *= 1 # convert from boolean to numbers\n    return df\n    \n    \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_train_features(df,col_name):\n    df = df[df.content_type_id==0][[col_name,'answered_correctly']].groupby(col_name).agg(['count','sum'])\n    if col_name == 'content_id':\n        col_name = 'question'\n        #TODO: Add question_entropy\n        # try decomposition methods?\n        \n    elif col_name == 'user_id':\n        col_name = 'student'\n        # TODO: user choice entropy\n        \n    df.columns=[col_name + '_total', col_name + '_correct']\n    df = df.astype('ufloat64')\n    df[col_name +'_incorrect'] = df[col_name + '_total'] - df[ col_name + '_correct']\n    df[col_name +'_correct_ratio'] = df[ col_name + '_correct']/df[col_name + '_total']\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ERFAN: This structure needs revision - I dont think it's super good for this task.\nmodel = Sequential()\nmodel.add(Dense(12, input_dim=len(feature_list), activation='relu'))\nmodel.add(Dense(8, activation='relu'))\nmodel.add(Dense(1, activation='sigmoid'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test = chunks[-1][feature_list] #.tonumpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test = chunks[-1]['answered_correctly'] #.tonumpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer = keras.optimizers.Adam(lr=0.001)\nes = EarlyStopping(monitor='val_loss', mode='min', verbose=1,patience=9)# ERFAN: stops when it reaches a plateau in validation loss\nmodel.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=[tf.keras.metrics.AUC(),'accuracy']) #ERFAN: added auc since that's what we are optimizing for - loss seems static.\n\nfor chunk in chunks : \n    X_train = prepare_train_df(chunk)\n    y_train = chunk['answered_correctly'].tonumpy()\n    X_train = X_train[feature_list].tonumpy() \n    model.fit(X_train, y_train, epochs=20, batch_size=50,validation_data=(X_test, y_test),verbose=1, callbacks=[es]) #Erfan: AUC seems static, loss decreases when batch_size < 100. Why do we want a bigger batchsize? ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save(\"neuralnetwork.h5\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"safe to run it. "},{"metadata":{"trusted":true},"cell_type":"code","source":"\n#train = train.sample(frac=0.010)\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.tags.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.part.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#df = train[\"user_id\",\"content_id\",\"answered_correctly\"]\n\n# ERFAN: Try to use/make different features user_id and content_id are probably not super informative to models.\n#df = train[['user_id','content_id','answered_correctly']]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_features(col_name):\n        #train.dropna()\n        df = reduce_mem_usage(train[train.content_type_id==0][[col_name,'answered_correctly']].groupby(col_name).agg(['count','sum']))\n        print(df)\n        if col_name == 'content_id':\n            col_name = 'question'\n            #TODO: Add question_entropy\n            # try decomposition methods?\n\n        elif col_name == 'user_id':\n            col_name = 'student'\n            # TODO: user choice entropy\n\n        df.columns=[col_name + '_total', col_name + '_correct']\n        df = df.astype('uint64')\n        df[col_name +'_incorrect'] = df[col_name + '_total'] - df[ col_name + '_correct']\n        df[col_name +'_correct_ratio'] = df[ col_name + '_correct']/df[col_name + '_total']\n        return df\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_dataframe = reduce_mem_usage(prepare_features('content_id'))\nquestions_dataframe['content_id'] = list(questions_dataframe.index)\nquestions_dataframe = reduce_mem_usage(questions_dataframe.rename_axis(\"question_index\"))\nquestions_dataframe","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#import matplotlib.pyplot as plt\n#fig = plt.figure()\n#ax = fig.add_axes([0,0,1,1])\n#langs = ['Answered Correctly', 'Answered Incorrectly']\n#results = [sum(questions_dataframe['question_correct']),sum(questions_dataframe['question_incorrect'])]\n#ax.bar(langs,results)\n#plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#import matplotlib.pyplot as plt\n\n# Pie chart, where the slices will be ordered and plotted counter-clockwise:\n#labels = 'Answered Correctly', 'Answered Incorrectly'\n#sizes = [sum(questions_dataframe['question_correct']), sum(questions_dataframe['question_incorrect'])]\n#explode = (0, 0.1)  # only \"explode\" the 2nd slice (i.e. 'Hogs')\n\n#fig1, ax1 = plt.subplots()\n#ax1.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%',\n       # shadow=True, startangle=90)\n#ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n\n#plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"users_dataframe = reduce_mem_usage(prepare_features('user_id'))\nusers_dataframe['user_id'] = list(users_dataframe.index)\nusers_dataframe = reduce_mem_usage(users_dataframe.rename_axis(\"user_index\")\nusers_dataframe","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.merge(reduce_mem_usage(train),reduce_mem_usage(users_dataframe),how = 'inner',on = 'user_id')\ndf = pd.merge(reduce_mem_usage(df),reduce_mem_usage(questions_dataframe),how = 'inner',on = 'content_id')\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df[df['answered_correctly'] != -1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.shape\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Replace Null values"},{"metadata":{"trusted":true},"cell_type":"code","source":"df['prior_question_elapsed_time'].fillna(-1,inplace = True)\ndf['prior_question_had_explanation'].fillna(-1,inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['prior_question_had_explanation'] *= 1 # convert from boolean to numbers","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#use this in presentation\n\ncols = list(df.columns.values) #Make a list of all of the columns in the df\ncols.pop(cols.index('answered_correctly')) #Remove b from list\ndf = df[cols+['answered_correctly']] #Create new dataframe with columns in the order you want\nf = plt.figure(figsize=(19, 15))\nplt.matshow(df.corr(), fignum=f.number)\nplt.xticks(range(df.shape[1]), df.columns, fontsize=14, rotation=45)\nplt.yticks(range(df.shape[1]), df.columns, fontsize=14)\ncb = plt.colorbar()\ncb.ax.tick_params(labelsize=14)\n# TODO FIX GRAPH ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.corr()['answered_correctly']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['prior_question_elapsed_time'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['prior_question_had_explanation'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['answered_correctly'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.decomposition import PCA\npca = PCA(n_components=5, svd_solver='full')\npca.fit(df[[i for i in df.columns if i!= 'answered_correctly']])\nprint(pca.explained_variance_ratio_)\npca_featurelist = pca.transform(df[[i for i in df.columns if i!= 'answered_correctly']])\npca_featurelist","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pca_featurelist.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import scipy.stats\nscipy.stats.spearmanr(pca_featurelist[:,0], df['answered_correctly'])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_list = ['content_id','prior_question_elapsed_time','prior_question_had_explanation', 'question_correct_ratio','student_correct_ratio']\nX = df[feature_list].to_numpy()\n\ny = df['answered_correctly'].to_numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = X.astype(float)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y = y.astype(float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.1, random_state = 0)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ERFAN: This structure needs revision - I dont think it's super good for this task.\nmodel = Sequential()\nmodel.add(Dense(12, input_dim=len(feature_list), activation='relu'))\nmodel.add(Dense(8, activation='relu'))\nmodel.add(Dense(1, activation='sigmoid'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer = keras.optimizers.Adam(lr=0.001)\nes = EarlyStopping(monitor='val_loss', mode='min', verbose=1,patience=9)# ERFAN: stops when it reaches a plateau in validation loss\nmodel.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=[tf.keras.metrics.AUC(),'accuracy']) #ERFAN: added auc since that's what we are optimizing for - loss seems static.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(X_train, y_train, epochs=30, batch_size=50,validation_data=(X_test, y_test),verbose=1, callbacks=[es]) #Erfan: AUC seems static, loss decreases when batch_size < 100. Why do we want a bigger batchsize?","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = model.predict_classes(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Random Forest"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\nfrom sklearn import metrics\nclf = RandomForestClassifier(max_depth=35, random_state=0)\n\nclf.fit(X_train,y_train)\nrf_predictions = clf.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fpr, tpr, thresholds = metrics.roc_curve(y_test, rf_predictions, pos_label=1)\nmetrics.auc(fpr,tpr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_train_df(df):\n    questions_dataframe = prepare_train_features(df,'content_id')\n    questions_dataframe['content_id'] = list(questions_dataframe.index)\n    questions_dataframe = questions_dataframe.rename_axis(\"question_index\")\n    users_dataframe = prepare_train_features(df,'user_id')\n    users_dataframe['user_id'] = list(users_dataframe.index)\n    users_dataframe = users_dataframe.rename_axis(\"user_index\")\n    df = pd.merge(df,users_dataframe,how = 'inner',on = 'user_id')\n    df = pd.merge(df,questions_dataframe,how = 'inner',on = 'content_id')\n    df = df[df['answered_correctly'] != -1]\n    df['prior_question_elapsed_time'].fillna(-1,inplace = True)\n    df['prior_question_had_explanation'].fillna(-1,inplace = True)\n    df['prior_question_had_explanation'] *= 1 # convert from boolean to numbers\n    return df\n    \n    \n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_train_features(df,col_name):\n    df = df[df.content_type_id==0][[col_name,'answered_correctly']].groupby(col_name).agg(['count','sum'])\n    if col_name == 'content_id':\n        col_name = 'question'\n        #TODO: Add question_entropy\n        # try decomposition methods?\n        \n    elif col_name == 'user_id':\n        col_name = 'student'\n        # TODO: user choice entropy\n        \n    df.columns=[col_name + '_total', col_name + '_correct']\n    df = df.astype('ufloat64')\n    df[col_name +'_incorrect'] = df[col_name + '_total'] - df[ col_name + '_correct']\n    df[col_name +'_correct_ratio'] = df[ col_name + '_correct']/df[col_name + '_total']\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_test_features(df,test_df,feature_list):\n    #['content_id','prior_question_elapsed_time','prior_question_had_explanation', 'question_correct_ratio','student_correct_ratio']\n    \n        \n    test_df = pd.merge(test_df,questions_dataframe[['content_id', 'question_correct_ratio']],how='left',on='content_id')\n    test_df = pd.merge(test_df,users_dataframe[['user_id','student_correct_ratio']],how='left',on='user_id')\n    test_df['prior_question_elapsed_time'].fillna(-1,inplace = True)\n    test_df['prior_question_had_explanation'] *= 1 # convert from boolean to numbers\n    test_df['prior_question_had_explanation'].fillna(-1,inplace = True)\n    test_df.fillna(-1,inplace = True)\n    return test_df[feature_list].to_numpy().astype(float)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission<p>\n<code>example_test</code> contains just a few dummy rows that can be used for development. The real submission must be made by using <code>riiideducation</code> package. To do that, we must create an environment and then loop through all the batches provided by <code>iter_test</code>."},{"metadata":{"trusted":true},"cell_type":"code","source":"env = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For each batch in <code>test_df</code>, we will predict the probability of answering correctly (<code>nb.predict</code>) and then we will send the resulting data back to the environment. This last part is done in <code>env.predict</code>. Notice that we must not create any <code>submission.csv</code> file, this is done automatically by <code>env.predict</code>."},{"metadata":{"trusted":true},"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n#     test_questions = test_df['content_id'].to_numpy()\n#     test_users = test_df['user_id'].to_numpy()\n    test_set = prepare_test_features(df,test_df,feature_list)\n    answered_correctly = model.predict(test_set)\n    answered_correctly = clf.predict(test_set)\n    test_df['answered_correctly'] = answered_correctly\n    env.predict(test_df.loc[test_df['content_type_id']==0,['row_id','answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}