{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nIn this competition, your challenge is to create algorithms for \"Knowledge Tracing,\" the modeling of student knowledge over time. The goal is to accurately predict how students will perform on future interactions. You will pair your machine learning skills using Riiid’s EdNet data.\n\nIn this notebook we will first take a general look at the data. After the exploratory analysis of the data, I will create some models to compare their accuracy."},{"metadata":{},"cell_type":"markdown","source":"<div class=\"alert alert-block alert-info\">\nShould you like this notebook or was it useful, please do UPVOTE! 👍.\n</div>"},{"metadata":{},"cell_type":"markdown","source":"### Importing libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\n# Plot\nimport matplotlib.pyplot as plt\nfrom matplotlib import pyplot\nimport seaborn as sns\n\n# Training and test data\nfrom sklearn.model_selection import train_test_split\n\n# AUC score\nfrom sklearn.metrics import roc_curve\nfrom sklearn.metrics import roc_auc_score\n\n# Model\nfrom sklearn.linear_model import LogisticRegression\nfrom xgboost import XGBClassifier\nfrom lightgbm import LGBMClassifier\n\n# Submission\nimport riiideducation\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading Data"},{"metadata":{},"cell_type":"markdown","source":"### Training data\n\nrow_id: ID code for the row.\n\ntimestamp: the time between this user interaction and the first event from that user.\n\nuser_id: ID code for the user.\n\ncontent_id: ID code for the user interaction\n\ncontent_type_id: 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\ntask_container_id: Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id. Monotonically increasing for each user.\n\nuser_answer: the user's answer to the question, if any. Read -1 as null, for lectures.\n\nanswered_correctly: if the user responded correctly. Read -1 as null, for lectures.\n\nprior_question_elapsed_time: How long it took a user to answer their previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Note that the time is the total time a user took to solve all the questions in the previous bundle.\n\nprior_question_had_explanation: Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback."},{"metadata":{},"cell_type":"markdown","source":"Since the training data is very large and the kaggle memory does not support it, so I will generate a sample of 1M observations."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"df_train = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv', \n                       nrows=10**6,\n                       dtype={'row_id': 'int64', \n                              'timestamp': 'int64', \n                              'user_id': 'int32',\n                              'content_id': 'int16',\n                              'content_type_id': 'int8',\n                              'task_container_id': 'int16',\n                              'user_answer': 'int8',\n                              'answered_correctly': 'int8',\n                              'prior_question_elapsed_time': 'float32',\n                              'prior_question_had_explanation': 'boolean'})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Exploratory Data Analysis"},{"metadata":{},"cell_type":"markdown","source":"Summary table of training data. Showing data type, missing, unique values and their first three values."},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"def resumetable(df):\n    print(f\"Dataset Shape: {df.shape}\")\n    summary = pd.DataFrame(df.dtypes,columns=['dtypes'])\n    summary = summary.reset_index()\n    summary['Name'] = summary['index']\n    summary = summary[['Name','dtypes']]\n    summary['Missing'] = df.isnull().sum().values    \n    summary['Uniques'] = df.nunique().values\n    summary['First Value'] = df.loc[0].values\n    summary['Second Value'] = df.loc[1].values\n    summary['Third Value'] = df.loc[2].values\n    return summary","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"resumetable(df_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Features overview"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 5))\n\nplt.suptitle('Time between this interaction and first event', fontsize = 18)\nplt.hist(df_train['timestamp'], bins = 50, color = \"skyblue\")\nplt.ylabel('Count', fontsize = 15)\nplt.xlabel('timestamp', fontsize = 15)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 5))\n\np = sns.distplot(df_train['user_id'])\np.set_title(\"Code for the user\", fontsize=18)\np.set_xlabel(\"user_id\", fontsize = 15)\np.set_ylabel(\"Probability\", fontsize = 15)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 5))\n\np = sns.distplot(df_train['content_id'])\np.set_title(\"The user interaction\", fontsize = 18)\np.set_xlabel(\"content_id\", fontsize = 15)\np.set_ylabel(\"Probability\", fontsize = 15)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 5))\n\np3 = sns.distplot(df_train['task_container_id'])\np3.set_title(\"Code for the batch of questions or lectures\", fontsize = 18)\np3.set_xlabel(\"task_container_id\", fontsize = 15)\np3.set_ylabel(\"Probability\", fontsize = 15)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 5))\n\np3 = sns.distplot(df_train['prior_question_elapsed_time'].dropna())\np3.set_title(\"How long it took a user to answer their previous question bundle\", fontsize = 18)\np3.set_xlabel(\"prior_question_elapsed_time\", fontsize = 15)\np3.set_ylabel(\"Probability\", fontsize = 15)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 5))\n\nfreq = len(df_train)\n\ng = sns.countplot(df_train['content_type_id'])\ng.set_title(\"\", fontsize = 18)\ng.set_xlabel(\"content_type_id\", fontsize = 15)\ng.set_ylabel(\"Count\", fontsize = 15)\n\nfor p in g.patches:\n    height = p.get_height()\n    g.text(p.get_x() + p.get_width() / 2., height + 3,\n          '{:1.2f}%'.format(height / freq * 100),\n          ha = \"center\", fontsize = 18)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 5))\n\nfreq = len(df_train)\n\ng = sns.countplot(df_train['user_answer'])\ng.set_title(\"The user's answer to the question\", fontsize = 18)\ng.set_xlabel(\"user_answer\", fontsize = 15)\ng.set_ylabel(\"Count\", fontsize = 15)\n\nfor p in g.patches:\n    height = p.get_height()\n    g.text(p.get_x() + p.get_width() / 2., height + 3,\n          '{:1.2f}%'.format(height / freq * 100),\n          ha = \"center\", fontsize = 18)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 5))\n\nfreq = len(df_train)\n\ng = sns.countplot(df_train['answered_correctly'])\ng.set_title(\"If the user responded correctly\", fontsize = 18)\ng.set_xlabel(\"answered_correctly\", fontsize = 15)\ng.set_ylabel(\"Count\", fontsize = 15)\n\nfor p in g.patches:\n    height = p.get_height()\n    g.text(p.get_x() + p.get_width() / 2., height + 3,\n          '{:1.2f}%'.format(height / freq * 100),\n          ha = \"center\", fontsize = 18)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 5))\n\nfreq = len(df_train)\n\ng = sns.countplot(df_train['prior_question_had_explanation'])\ng.set_title(\"Whether or not the user saw an explanation and the correct response (s) \\n after answering the previous question bundle\",\n            fontsize = 18)\ng.set_xlabel(\"prior_question_had_explanation\", fontsize = 15)\ng.set_ylabel(\"Count\", fontsize = 15)\n\nfor p in g.patches:\n    height = p.get_height()\n    g.text(p.get_x() + p.get_width() / 2., height + 3,\n          '{:1.2f}%'.format(height / freq * 100),\n          ha = \"center\", fontsize = 18)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 5))\ng = sns.scatterplot(data = df_train, x = \"timestamp\", y = \"prior_question_elapsed_time\", hue = \"prior_question_had_explanation\", \n                style = \"prior_question_had_explanation\")\ng.set_xlabel(\"timestamp\", fontsize = 15)\ng.set_ylabel(\"prior_question_elapsed_time\", fontsize = 15)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Removing the null value from the answered_correctly variable"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = df_train[df_train['answered_correctly']!=-1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 5))\n\nsns.relplot(\n    data= train, x = \"timestamp\", y = \"prior_question_elapsed_time\",\n    col = \"prior_question_had_explanation\", hue = \"answered_correctly\", style = \"answered_correctly\",\n    kind=\"scatter\"\n);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preparing features"},{"metadata":{"trusted":true},"cell_type":"code","source":"used_data_types_dict = {\n    'row_id': 'int64',\n    'timestamp': 'int64',\n    'user_id': 'int32',\n    'content_id': 'int16',\n    'answered_correctly': 'int8',\n    'prior_question_elapsed_time': 'float16'\n}\n\ntrain_df = pd.read_csv(\n    '/kaggle/input/riiid-test-answer-prediction/train.csv',\n    usecols = used_data_types_dict.keys(),\n    dtype=used_data_types_dict, \n    index_col = 0\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features_df = train_df.iloc[:int(9 /10 * len(train_df))]\ntrain_df = train_df.iloc[int(9 /10 * len(train_df)):]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_questions_only_df = features_df[features_df['answered_correctly']!=-1]\ngrouped_by_user_df = train_questions_only_df.groupby('user_id')\nuser_answers_df = grouped_by_user_df.agg({'answered_correctly': ['mean', 'count', 'std']}).copy()\nuser_answers_df.columns = ['mean_user_accuracy', 'questions_answered', 'std_user_accuracy']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"grouped_by_content_df = train_questions_only_df.groupby('content_id')\ncontent_answers_df = grouped_by_content_df.agg({'answered_correctly': ['mean', 'count', 'std'] }).copy()\ncontent_answers_df.columns = ['mean_accuracy', 'question_asked', 'std_accuracy']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\n\ndel features_df\ndel grouped_by_user_df\ndel grouped_by_content_df\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features = [\n    'timestamp',\n    'mean_user_accuracy', \n    'questions_answered',\n    'std_user_accuracy',\n    'mean_accuracy', \n    'question_asked',\n    'std_accuracy',\n    'prior_question_elapsed_time'\n]\ntarget = 'answered_correctly'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = train_df[train_df[target] != -1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = train_df.merge(user_answers_df, how='left', on='user_id')\ntrain_df = train_df.merge(content_answers_df, how='left', on='content_id')\ntrain_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = train_df[features + [target]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = train_df.replace([np.inf, -np.inf], np.nan)\ntrain_df = train_df.fillna(0)\ntrain_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Reference:\nhttps://www.kaggle.com/isaienkov/riiid-answer-correctness-prediction-eda-modeling"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Function to reduce the df size\ndef reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage().sum() / 1024**2    \n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)  \n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)    \n    end_mem = df.memory_usage().sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reducing memory\ntrain_df = reduce_mem_usage(train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model"},{"metadata":{},"cell_type":"markdown","source":"I will use some models and compare them."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Training and test data\ntrain_df, test_df = train_test_split(train_df, random_state = 123, test_size = 0.2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Logistic Regression"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating the model\nmodel_LR = LogisticRegression()\n\n# Training the model\nmodel_LR.fit(train_df[features], train_df[target])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ns_probs = [0 for _ in range(len(train_df[target]))]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# predict probabilities\nLR_probs = model_LR.predict_proba(train_df[features])\n\n# keep probabilities for the positive outcome only\nLR_probs = LR_probs[:, 1]\n\n# calculate scores\nns_auc = roc_auc_score(train_df[target], ns_probs)\nLR_auc = roc_auc_score(train_df[target], LR_probs)\n\n# result print\nprint('Logistic: ROC AUC = %.3f' % (LR_auc * 100))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# calculate roc curves\nns_fpr, ns_tpr, _ = roc_curve(train_df[target], ns_probs)\nLR_fpr, LR_tpr, _ = roc_curve(train_df[target], LR_probs)\n\n# figure size\nplt.rcParams[\"figure.figsize\"] = (9, 5)\n\n# plot the roc curve for the model\npyplot.plot(ns_fpr, ns_tpr, linestyle = '--', label = 'No Skill')\npyplot.plot(LR_fpr, LR_tpr, linestyle = '-', label = 'Logistic')\n\n# axis labels\npyplot.xlabel('False Positive Rate', fontsize = 15)\npyplot.ylabel('True Positive Rate', fontsize = 15)\n\n# show the legend\npyplot.legend(fontsize = 15)\n\n# show the plot\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Extreme Gradient Boosting - XGBoost"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating the model\nmodel_XGB = XGBClassifier()\n\n# Training the model\nmodel_XGB.fit(train_df[features], train_df[target])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# predict probabilities\nXGB_probs = model_XGB.predict_proba(train_df[features])\n\n# keep probabilities for the positive outcome only\nXGB_probs = XGB_probs[:, 1]\n\n# calculate scores\nns_auc = roc_auc_score(train_df[target], ns_probs)\nXGB_auc = roc_auc_score(train_df[target], XGB_probs)\n\n# result print\nprint('XGBoost: ROC AUC = %.3f' % (XGB_auc * 100))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# calculate roc curves\nns_fpr, ns_tpr, _ = roc_curve(train_df[target], ns_probs)\nXGB_fpr, XGB_tpr, _ = roc_curve(train_df[target], XGB_probs)\n\n# figure size\nplt.rcParams[\"figure.figsize\"] = (9, 5)\n\n# plot the roc curve for the model\npyplot.plot(ns_fpr, ns_tpr, linestyle = '--', label = 'No Skill')\npyplot.plot(XGB_fpr, XGB_tpr, linestyle = '-', label = 'XGBoost', color = \"red\")\n\n# axis labels\npyplot.xlabel('False Positive Rate', fontsize = 15)\npyplot.ylabel('True Positive Rate', fontsize = 15)\n\n# show the legend\npyplot.legend(fontsize = 15)\n\n# show the plot\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### LightGBM"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating the model\nmodel_LGBM = LGBMClassifier()\n\n# Training the model\nmodel_LGBM.fit(train_df[features], train_df[target])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# predict probabilities\nLGBM_probs = model_LGBM.predict_proba(train_df[features])\n\n# keep probabilities for the positive outcome only\nLGBM_probs = LGBM_probs[:, 1]\n\n# calculate scores\nns_auc = roc_auc_score(train_df[target], ns_probs)\nLGBM_auc = roc_auc_score(train_df[target], LGBM_probs)\n\n# result print\nprint('Logistic: ROC AUC = %.3f' % (LGBM_auc * 100))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# calculate roc curves\nns_fpr, ns_tpr, _ = roc_curve(train_df[target], ns_probs)\nLGBM_fpr, LGBM_tpr, _ = roc_curve(train_df[target], LGBM_probs)\n\n# figure size\nplt.rcParams[\"figure.figsize\"] = (9, 5)\n\n# plot the roc curve for the model\npyplot.plot(ns_fpr, ns_tpr, linestyle = '--', label = 'No Skill')\npyplot.plot(LGBM_fpr, LGBM_tpr, linestyle = '-', label = 'LGBM', color = \"green\")\n\n# axis labels\npyplot.xlabel('False Positive Rate', fontsize = 15)\npyplot.ylabel('True Positive Rate', fontsize = 15)\n\n# show the legend\npyplot.legend(fontsize = 12)\n\n# show the plot\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The XGboost and LightGBM models showed very close accuracy, with a slight advantage for the XGboost. However, XGboost's processing time is much longer than LightGBM, which has a disadvantage.\n* With the time gain in LightGBM processing, I will adjust some parameters to see if we have an increase in accuracy.\n* But before creating a new model with new parameters, let's see which features were most important for the previous LightGBM model."},{"metadata":{"trusted":true},"cell_type":"code","source":"feat_importance = pd.DataFrame()\nfeat_importance[\"feature\"] = train_df[features].columns\nfeat_importance[\"value\"] = model_LGBM.feature_importances_\nfeat_importance.sort_values(by='value', ascending=False, inplace=True)\n\nplt.figure(figsize=(8,10))\nax = sns.barplot(y=\"feature\", x=\"value\", data=feat_importance)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"new_feat = [\n    'timestamp',\n    'mean_accuracy', \n    'question_asked',\n    'prior_question_elapsed_time'\n]\n\ntrain_df_new = train_df[new_feat]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating the model\nmodel_LGBM_par = LGBMClassifier(\n    objective='binary',\n    boosting='gbdt',\n    learning_rate = 0.05,\n    max_depth = 8,\n    num_leaves = 80,\n    n_estimators = 400,\n    bagging_fraction = 0.8,\n    feature_fraction = 0.9)\n\n# Training the model\nmodel_LGBM_par.fit(train_df_new, train_df[target])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# predict probabilities\nLGBM_par_probs = model_LGBM_par.predict_proba(train_df_new)\n\n# keep probabilities for the positive outcome only\nLGBM_par_probs = LGBM_par_probs[:, 1]\n\n# calculate scores\nns_auc = roc_auc_score(train_df[target], ns_probs)\nLGBM_par_auc = roc_auc_score(train_df[target], LGBM_par_probs)\n\n# result print\nprint('Logistic: ROC AUC = %.3f' % (LGBM_par_auc * 100))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"env = riiideducation.make_env()\n\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n    test_df = test_df.merge(user_answers_df, how = 'left', on = 'user_id')\n    test_df = test_df.merge(content_answers_df, how = 'left', on = 'content_id')\n    test_df['prior_question_had_explanation'] = test_df['prior_question_had_explanation'].fillna(value = False).astype(bool)\n    test_df.fillna(value = -1, inplace = True)\n    \n    test_df['answered_correctly'] = model_LGBM_par.predict_proba(test_df[new_feat])[:,1]\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# To be continued..."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}