{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Deep Learning Adventures Present\n### Attention and Transformers Bonus Edition | Applications of transformers for a non-NLP task\n### https://www.meetup.com/Deep-Learning-Adventures/events/281617686\n\nHello dear friends 🌎 🌍,\n\nWe hope you are enjoying our latest sessions 🏖️ deploying models and cool applications to our phones 📱 and edge devices 🕹️\n\nOur friend Dmitri has offered to lead some very interesting papers, code and content all on the attention mechanism in deep learning models and transformers! 😲\n\nAttention and Transformers Bonus Edition | Applications of transformers for a non-NLP task\n\n- Kaggle competition on knowledge tracing:\nRiiid Answer Correctness Prediction\nTrack knowledge states of 1M+ students in the wild\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/overview\n\n- Dmitri's approach, ranked # 54 out of 3,395 participants (top 2%).\nhttps://github.com/dlevonian/riiid-prediction ✅\n\n- Presentation: https://docs.google.com/presentation/d/1QWWgewJ12unyPUDRpnOLsygl1_0lVCwpWSJZGIRB1YY/edit?usp=sharing\n\nThe recording of this cool event 😎 is available at:\nhttps://bit.ly/dla-transformers","metadata":{}},{"cell_type":"markdown","source":"### Riiid Answer Correctness Prediction \nhttps://www.kaggle.com/c/riiid-test-answer-prediction\n\nIn this competition, your challenge is to create algorithms for \"Knowledge Tracing,\" the modeling of student knowledge over time. The goal is to accurately predict how students will perform on future interactions. You will pair your machine learning skills using Riiid’s EdNet data.","metadata":{}},{"cell_type":"markdown","source":"Inspired by: https://www.kaggle.com/erikbruin/riiid-comprehensive-eda-baseline/notebook","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport matplotlib.style as style\nstyle.use('fivethirtyeight')\nimport seaborn as sns\nfrom matplotlib.ticker import FuncFormatter\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-12-02T14:14:11.273425Z","iopub.execute_input":"2021-12-02T14:14:11.274039Z","iopub.status.idle":"2021-12-02T14:14:12.324227Z","shell.execute_reply.started":"2021-12-02T14:14:11.273894Z","shell.execute_reply":"2021-12-02T14:14:12.323588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\ntrain = pd.read_pickle(\"../input/riiid-train-data-multiple-formats/riiid_train.pkl.gzip\")\ntrain['prior_question_had_explanation'] = train['prior_question_had_explanation'].astype('boolean')\nprint(\"Train size:\", train.shape)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:14:12.325898Z","iopub.execute_input":"2021-12-02T14:14:12.326319Z","iopub.status.idle":"2021-12-02T14:15:21.764409Z","shell.execute_reply.started":"2021-12-02T14:14:12.326266Z","shell.execute_reply":"2021-12-02T14:15:21.763279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.memory_usage(deep=True)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:21.765588Z","iopub.execute_input":"2021-12-02T14:15:21.765814Z","iopub.status.idle":"2021-12-02T14:15:21.777459Z","shell.execute_reply.started":"2021-12-02T14:15:21.765788Z","shell.execute_reply":"2021-12-02T14:15:21.776721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:21.778594Z","iopub.execute_input":"2021-12-02T14:15:21.778825Z","iopub.status.idle":"2021-12-02T14:15:21.805778Z","shell.execute_reply.started":"2021-12-02T14:15:21.778797Z","shell.execute_reply":"2021-12-02T14:15:21.804799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nquestions = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv')\nlectures = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv')\nexample_test = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_test.csv')\nexample_sample_submission = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:21.809063Z","iopub.execute_input":"2021-12-02T14:15:21.809385Z","iopub.status.idle":"2021-12-02T14:15:21.872544Z","shell.execute_reply.started":"2021-12-02T14:15:21.809351Z","shell.execute_reply":"2021-12-02T14:15:21.871915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cids = train.content_id.value_counts()[:30]\n\nfig = plt.figure(figsize=(12,6))\nax = cids.plot.bar()\nplt.title(\"Thirty most used content id's\")\nplt.xticks(rotation=90)\nax.get_yaxis().set_major_formatter(FuncFormatter(lambda x, p: format(int(x), ','))) #add thousands separator\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:21.873634Z","iopub.execute_input":"2021-12-02T14:15:21.873906Z","iopub.status.idle":"2021-12-02T14:15:22.802619Z","shell.execute_reply.started":"2021-12-02T14:15:21.873873Z","shell.execute_reply":"2021-12-02T14:15:22.801537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#1 year = 31536000000 ms\nts = train['timestamp']/(31536000000/12)\nfig = plt.figure(figsize=(12,6))\nts.plot.hist(bins=100)\nplt.title(\"Histogram of timestamp\")\nplt.xticks(rotation=0)\nplt.xlabel(\"Months between this user interaction and the first event completion from that user\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:22.80506Z","iopub.execute_input":"2021-12-02T14:15:22.805871Z","iopub.status.idle":"2021-12-02T14:15:38.353585Z","shell.execute_reply.started":"2021-12-02T14:15:22.805819Z","shell.execute_reply":"2021-12-02T14:15:38.352716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"correct = train[train.answered_correctly != -1].answered_correctly.value_counts(ascending=True)\n\nfig = plt.figure(figsize=(12,4))\ncorrect.plot.barh()\nfor i, v in zip(correct.index, correct.values):\n    plt.text(v, i, '{:,}'.format(v), color='white', fontweight='bold', fontsize=14, ha='right', va='center')\nplt.title(\"Questions answered correctly\")\nplt.xticks(rotation=0)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:38.355082Z","iopub.execute_input":"2021-12-02T14:15:38.355371Z","iopub.status.idle":"2021-12-02T14:15:44.905775Z","shell.execute_reply.started":"2021-12-02T14:15:38.35534Z","shell.execute_reply":"2021-12-02T14:15:44.904773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"user_percent = train[train.answered_correctly != -1].groupby('user_id')['answered_correctly'].agg(Mean='mean', Answers='count')\nprint(f'the highest number of questions answered by a user is {user_percent.Answers.max()}')","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:44.907195Z","iopub.execute_input":"2021-12-02T14:15:44.907464Z","iopub.status.idle":"2021-12-02T14:15:56.304339Z","shell.execute_reply.started":"2021-12-02T14:15:44.907432Z","shell.execute_reply":"2021-12-02T14:15:56.303332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"user_percent = user_percent.query('Answers <= 1000').sample(n=200, random_state=1)\n\nfig = plt.figure(figsize=(12,6))\nx = user_percent.Answers\ny = user_percent.Mean\nplt.scatter(x, y, marker='o')\nplt.title(\"Percent answered correctly versus number of questions answered User\")\nplt.xticks(rotation=0)\nplt.xlabel(\"Number of questions answered\")\nplt.ylabel(\"Percent answered correctly\")\nz = np.polyfit(x, y, 1)\np = np.poly1d(z)\nplt.plot(x,p(x),\"r--\")\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:56.305678Z","iopub.execute_input":"2021-12-02T14:15:56.306093Z","iopub.status.idle":"2021-12-02T14:15:56.590342Z","shell.execute_reply.started":"2021-12-02T14:15:56.306058Z","shell.execute_reply":"2021-12-02T14:15:56.589353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Test data","metadata":{}},{"cell_type":"code","source":"example_test.shape","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:56.591481Z","iopub.execute_input":"2021-12-02T14:15:56.59169Z","iopub.status.idle":"2021-12-02T14:15:56.599172Z","shell.execute_reply.started":"2021-12-02T14:15:56.591665Z","shell.execute_reply":"2021-12-02T14:15:56.598042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"example_test.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:56.600297Z","iopub.execute_input":"2021-12-02T14:15:56.600503Z","iopub.status.idle":"2021-12-02T14:15:56.624636Z","shell.execute_reply.started":"2021-12-02T14:15:56.600479Z","shell.execute_reply":"2021-12-02T14:15:56.624056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batches_test = set(list(example_test.task_container_id.unique()))\nbatches_train = set(list(train.task_container_id.unique()))\nprint(f'All batches in example_test are also in train is {batches_test.issubset(batches_train)}.')","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:56.625632Z","iopub.execute_input":"2021-12-02T14:15:56.625959Z","iopub.status.idle":"2021-12-02T14:15:57.334883Z","shell.execute_reply.started":"2021-12-02T14:15:56.625931Z","shell.execute_reply":"2021-12-02T14:15:57.334015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"user_test = set(list(example_test.user_id.unique()))\nuser_train = set(list(train.user_id.unique()))\n\nprint(f'User_ids in example_test but not in train: {user_test - user_train}.')","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:57.338672Z","iopub.execute_input":"2021-12-02T14:15:57.33894Z","iopub.status.idle":"2021-12-02T14:15:57.957362Z","shell.execute_reply.started":"2021-12-02T14:15:57.33891Z","shell.execute_reply":"2021-12-02T14:15:57.956484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Baseline model","metadata":{}},{"cell_type":"code","source":"#this clears everything loaded in RAM, including the libraries\n%reset -f","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:57.958457Z","iopub.execute_input":"2021-12-02T14:15:57.958691Z","iopub.status.idle":"2021-12-02T14:15:58.448068Z","shell.execute_reply.started":"2021-12-02T14:15:57.958656Z","shell.execute_reply":"2021-12-02T14:15:58.446168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:58.449865Z","iopub.execute_input":"2021-12-02T14:15:58.450886Z","iopub.status.idle":"2021-12-02T14:15:58.471798Z","shell.execute_reply.started":"2021-12-02T14:15:58.45075Z","shell.execute_reply":"2021-12-02T14:15:58.470397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nsys.path.append('/kaggle/input/riiid-test-answer-prediction')\nimport riiideducation","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:58.473011Z","iopub.execute_input":"2021-12-02T14:15:58.474377Z","iopub.status.idle":"2021-12-02T14:15:58.511592Z","shell.execute_reply.started":"2021-12-02T14:15:58.474333Z","shell.execute_reply":"2021-12-02T14:15:58.510643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport matplotlib.style as style\nstyle.use('fivethirtyeight')\nimport seaborn as sns\n\nimport lightgbm as lgb\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.preprocessing import LabelEncoder\n\nimport gc\nimport sys\npd.set_option('display.max_rows', None)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:58.513223Z","iopub.execute_input":"2021-12-02T14:15:58.513805Z","iopub.status.idle":"2021-12-02T14:15:59.666125Z","shell.execute_reply.started":"2021-12-02T14:15:58.513724Z","shell.execute_reply":"2021-12-02T14:15:59.665157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ncols_to_load = ['row_id', 'user_id', 'answered_correctly', 'content_id', 'prior_question_had_explanation', 'prior_question_elapsed_time']\ntrain = pd.read_pickle(\"../input/riiid-train-data-multiple-formats/riiid_train.pkl.gzip\")[cols_to_load]\ntrain['prior_question_had_explanation'] = train['prior_question_had_explanation'].astype('boolean')\n\nprint(\"Train size:\", train.shape)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:15:59.667678Z","iopub.execute_input":"2021-12-02T14:15:59.667935Z","iopub.status.idle":"2021-12-02T14:16:56.058065Z","shell.execute_reply.started":"2021-12-02T14:15:59.667906Z","shell.execute_reply":"2021-12-02T14:16:56.056949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nquestions = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv')\nlectures = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv')\nexample_test = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_test.csv')\nexample_sample_submission = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:16:56.059391Z","iopub.execute_input":"2021-12-02T14:16:56.059629Z","iopub.status.idle":"2021-12-02T14:16:56.094123Z","shell.execute_reply.started":"2021-12-02T14:16:56.059601Z","shell.execute_reply":"2021-12-02T14:16:56.09348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train.shape)\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:16:56.095371Z","iopub.execute_input":"2021-12-02T14:16:56.095716Z","iopub.status.idle":"2021-12-02T14:16:56.109356Z","shell.execute_reply.started":"2021-12-02T14:16:56.095686Z","shell.execute_reply":"2021-12-02T14:16:56.108273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n#adding user features\nuser_df = train[train.answered_correctly != -1].groupby('user_id').agg({'answered_correctly': ['count', 'mean']}).reset_index()\nuser_df.columns = ['user_id', 'user_questions', 'user_mean']\n\nuser_lect = train.groupby([\"user_id\", \"answered_correctly\"]).size().unstack()\nuser_lect.columns = ['Lecture', 'Wrong', 'Right']\nuser_lect = user_lect[['Lecture']].fillna(0).astype('int8')\n#user_lect = user_lect.astype('int8')\nuser_lect['watches_lecture'] = np.where(user_lect.Lecture > 0, 1, 0)\nuser_lect = user_lect.reset_index()\nuser_lect = user_lect[['user_id', 'watches_lecture']]\n\nuser_df = user_df.merge(user_lect, on = \"user_id\", how = \"left\")\ndel user_lect\nuser_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:16:56.11097Z","iopub.execute_input":"2021-12-02T14:16:56.111403Z","iopub.status.idle":"2021-12-02T14:17:12.63275Z","shell.execute_reply.started":"2021-12-02T14:16:56.111358Z","shell.execute_reply":"2021-12-02T14:17:12.632156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n#adding content features\ncontent_df = train[train.answered_correctly != -1].groupby('content_id').agg({'answered_correctly': ['count', 'mean']}).reset_index()\ncontent_df.columns = ['content_id', 'content_questions', 'content_mean']\ncontent_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:17:12.634074Z","iopub.execute_input":"2021-12-02T14:17:12.634487Z","iopub.status.idle":"2021-12-02T14:17:20.808881Z","shell.execute_reply.started":"2021-12-02T14:17:12.634456Z","shell.execute_reply":"2021-12-02T14:17:20.808073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n#using one of the validation sets composed by tito\ncv2_train = pd.read_pickle(\"../input/riiid-cross-validation-files/cv2_train.pickle\")['row_id']\ncv2_valid = pd.read_pickle(\"../input/riiid-cross-validation-files/cv2_valid.pickle\")['row_id']","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:17:20.810282Z","iopub.execute_input":"2021-12-02T14:17:20.810905Z","iopub.status.idle":"2021-12-02T14:18:11.48984Z","shell.execute_reply.started":"2021-12-02T14:17:20.810859Z","shell.execute_reply":"2021-12-02T14:18:11.488931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train[train.answered_correctly != -1]\n\n#save mean before splitting\n#please be aware that there is an issues with train.prior_question_elapsed_time.mean()\n#see https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/195032\nmean_prior = train.prior_question_elapsed_time.astype(\"float64\").mean()\n\nvalidation = train[train.row_id.isin(cv2_valid)]\ntrain = train[train.row_id.isin(cv2_train)]\n\nvalidation = validation.drop(columns = \"row_id\")\ntrain = train.drop(columns = \"row_id\")\n\ndel cv2_train, cv2_valid\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:18:11.491686Z","iopub.execute_input":"2021-12-02T14:18:11.492271Z","iopub.status.idle":"2021-12-02T14:18:38.123022Z","shell.execute_reply.started":"2021-12-02T14:18:11.492225Z","shell.execute_reply":"2021-12-02T14:18:38.121949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def data_pipeline(df, user_df, content_df, label_enc, mode='train'):\n    df = df.merge(user_df, on = \"user_id\", how = \"left\")\n    df = df.merge(content_df, on = \"content_id\", how = \"left\")\n    df['content_questions'].fillna(0, inplace = True)\n    df['content_mean'].fillna(0.5, inplace = True)\n    df['watches_lecture'].fillna(0, inplace = True)\n    df['user_questions'].fillna(0, inplace = True)\n    df['user_mean'].fillna(0.5, inplace = True)\n    df['prior_question_elapsed_time'].fillna(mean_prior, inplace = True)\n    df['prior_question_had_explanation'].fillna(False, inplace = True)\n\n    if mode =='train':\n        label_enc.fit(df['prior_question_had_explanation'])\n\n    df['prior_question_had_explanation'] = label_enc.transform(df['prior_question_had_explanation'])    \n    df[['content_questions', 'user_questions']] = df[['content_questions', 'user_questions']].astype(int)\n    return df, label_enc","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:18:38.124629Z","iopub.execute_input":"2021-12-02T14:18:38.125112Z","iopub.status.idle":"2021-12-02T14:18:38.134515Z","shell.execute_reply.started":"2021-12-02T14:18:38.125077Z","shell.execute_reply":"2021-12-02T14:18:38.133619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Your notebook tried to allocate more memory than is available. It has restarted.\n#label_enc = LabelEncoder()\n#train, label_enc = data_pipeline(train, user_df, content_df, label_enc, mode='train')\n#train.sample(5)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:18:38.136026Z","iopub.execute_input":"2021-12-02T14:18:38.136898Z","iopub.status.idle":"2021-12-02T14:18:38.15237Z","shell.execute_reply.started":"2021-12-02T14:18:38.136839Z","shell.execute_reply":"2021-12-02T14:18:38.151404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#validation, _ = data_pipeline(validation, user_df, content_df, label_enc, mode='validation')\n#validation.sample(5)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:18:38.153416Z","iopub.execute_input":"2021-12-02T14:18:38.153948Z","iopub.status.idle":"2021-12-02T14:18:38.163609Z","shell.execute_reply.started":"2021-12-02T14:18:38.153916Z","shell.execute_reply":"2021-12-02T14:18:38.163027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"label_enc = LabelEncoder()\n\ntrain = train.merge(user_df, on = \"user_id\", how = \"left\")\ntrain = train.merge(content_df, on = \"content_id\", how = \"left\")\ntrain['content_questions'].fillna(0, inplace = True)\ntrain['content_mean'].fillna(0.5, inplace = True)\ntrain['watches_lecture'].fillna(0, inplace = True)\ntrain['user_questions'].fillna(0, inplace = True)\ntrain['user_mean'].fillna(0.5, inplace = True)\ntrain['prior_question_elapsed_time'].fillna(mean_prior, inplace = True)\ntrain['prior_question_had_explanation'].fillna(False, inplace = True)\n\nlabel_enc.fit(train['prior_question_had_explanation'])\n\ntrain['prior_question_had_explanation'] = label_enc.transform(train['prior_question_had_explanation'])\ntrain[['content_questions', 'user_questions']] = train[['content_questions', 'user_questions']].astype(int)\ntrain.sample(5)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:18:38.164578Z","iopub.execute_input":"2021-12-02T14:18:38.165239Z","iopub.status.idle":"2021-12-02T14:20:06.111982Z","shell.execute_reply.started":"2021-12-02T14:18:38.165191Z","shell.execute_reply":"2021-12-02T14:20:06.111348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"validation = validation.merge(user_df, on = \"user_id\", how = \"left\")\nvalidation = validation.merge(content_df, on = \"content_id\", how = \"left\")\nvalidation['content_questions'].fillna(0, inplace = True)\nvalidation['content_mean'].fillna(0.5, inplace = True)\nvalidation['watches_lecture'].fillna(0, inplace = True)\nvalidation['user_questions'].fillna(0, inplace = True)\nvalidation['user_mean'].fillna(0.5, inplace = True)\nvalidation['prior_question_elapsed_time'].fillna(mean_prior, inplace = True)\nvalidation['prior_question_had_explanation'].fillna(False, inplace = True)\n\nvalidation['prior_question_had_explanation'] = label_enc.transform(validation['prior_question_had_explanation'])\nvalidation[['content_questions', 'user_questions']] = validation[['content_questions', 'user_questions']].astype(int)\nvalidation.sample(5)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:20:06.116054Z","iopub.execute_input":"2021-12-02T14:20:06.116432Z","iopub.status.idle":"2021-12-02T14:20:07.934524Z","shell.execute_reply.started":"2021-12-02T14:20:06.116388Z","shell.execute_reply":"2021-12-02T14:20:07.933716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# features = ['user_questions', 'user_mean', 'content_questions', 'content_mean', 'watches_lecture',\n#             'prior_question_elapsed_time', 'prior_question_had_explanation']\n\nfeatures = ['user_questions', 'user_mean', 'content_questions', 'content_mean', 'prior_question_elapsed_time']\n\n\n#for now just taking 10.000.000 rows for training\ntrain = train.sample(n=10_000_000, random_state = 1)\n\ny_train = train['answered_correctly']\ntrain = train[features]\n\ny_val = validation['answered_correctly']\nvalidation = validation[features]","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:20:07.935701Z","iopub.execute_input":"2021-12-02T14:20:07.935958Z","iopub.status.idle":"2021-12-02T14:20:16.292778Z","shell.execute_reply.started":"2021-12-02T14:20:07.93593Z","shell.execute_reply":"2021-12-02T14:20:16.291921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"params = {'objective': 'binary',\n          'metric': 'auc',\n          'seed': 2020,\n          'learning_rate': 0.1, #default\n          \"boosting_type\": \"gbdt\" #default\n         }","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:20:16.293923Z","iopub.execute_input":"2021-12-02T14:20:16.294141Z","iopub.status.idle":"2021-12-02T14:20:16.298783Z","shell.execute_reply.started":"2021-12-02T14:20:16.294116Z","shell.execute_reply":"2021-12-02T14:20:16.29803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgb_train = lgb.Dataset(train, y_train, categorical_feature = None)\nlgb_eval = lgb.Dataset(validation, y_val, categorical_feature = None)\n\ndel train, y_train, validation, y_val\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:20:16.300165Z","iopub.execute_input":"2021-12-02T14:20:16.300623Z","iopub.status.idle":"2021-12-02T14:20:16.439628Z","shell.execute_reply.started":"2021-12-02T14:20:16.300571Z","shell.execute_reply":"2021-12-02T14:20:16.438978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nmodel = lgb.train(\n    params, lgb_train,\n    valid_sets=[lgb_train, lgb_eval],\n    verbose_eval=50,\n    num_boost_round=10000,\n    early_stopping_rounds=8\n)","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:20:16.441117Z","iopub.execute_input":"2021-12-02T14:20:16.44138Z","iopub.status.idle":"2021-12-02T14:39:24.470703Z","shell.execute_reply.started":"2021-12-02T14:20:16.441351Z","shell.execute_reply":"2021-12-02T14:39:24.469666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgb.plot_importance(model)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:39:24.473467Z","iopub.execute_input":"2021-12-02T14:39:24.474124Z","iopub.status.idle":"2021-12-02T14:39:24.736843Z","shell.execute_reply.started":"2021-12-02T14:39:24.474054Z","shell.execute_reply":"2021-12-02T14:39:24.735923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Lets use the Competition API to bring in the test data and make a submission\nhttps://www.kaggle.com/sohier/competition-api-detailed-introduction/notebook","metadata":{}},{"cell_type":"code","source":"env = riiideducation.make_env()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:39:24.738531Z","iopub.execute_input":"2021-12-02T14:39:24.738973Z","iopub.status.idle":"2021-12-02T14:39:24.743722Z","shell.execute_reply.started":"2021-12-02T14:39:24.738934Z","shell.execute_reply":"2021-12-02T14:39:24.742959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"iter_test = env.iter_test()","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:39:24.745436Z","iopub.execute_input":"2021-12-02T14:39:24.746153Z","iopub.status.idle":"2021-12-02T14:39:24.757946Z","shell.execute_reply.started":"2021-12-02T14:39:24.746106Z","shell.execute_reply":"2021-12-02T14:39:24.757244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n    test_df = test_df.merge(user_df, on = \"user_id\", how = \"left\")\n    test_df = test_df.merge(content_df, on = \"content_id\", how = \"left\")\n    test_df['content_questions'].fillna(0, inplace = True)\n    test_df['content_mean'].fillna(0.5, inplace = True)\n    test_df['watches_lecture'].fillna(0, inplace = True)\n    test_df['user_questions'].fillna(0, inplace = True)\n    test_df['user_mean'].fillna(0.5, inplace = True)\n    test_df['prior_question_elapsed_time'].fillna(mean_prior, inplace = True)\n    test_df['prior_question_had_explanation'].fillna(False, inplace = True)\n    test_df['prior_question_had_explanation'] = label_enc.transform(test_df['prior_question_had_explanation'])\n    test_df[['content_questions', 'user_questions']] = test_df[['content_questions', 'user_questions']].astype(int)\n    test_df['answered_correctly'] =  model.predict(test_df[features])\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:39:24.759411Z","iopub.execute_input":"2021-12-02T14:39:24.760316Z","iopub.status.idle":"2021-12-02T14:39:25.097807Z","shell.execute_reply.started":"2021-12-02T14:39:24.760255Z","shell.execute_reply":"2021-12-02T14:39:25.096876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#last batch overview\n#submission_test_df = test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']]\n#submission_test_df.to_csv('submission.csv')","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:39:25.099205Z","iopub.execute_input":"2021-12-02T14:39:25.100087Z","iopub.status.idle":"2021-12-02T14:39:25.107659Z","shell.execute_reply.started":"2021-12-02T14:39:25.100042Z","shell.execute_reply":"2021-12-02T14:39:25.106601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#submission_test_df.shape","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:39:25.109292Z","iopub.execute_input":"2021-12-02T14:39:25.109905Z","iopub.status.idle":"2021-12-02T14:39:25.122976Z","shell.execute_reply.started":"2021-12-02T14:39:25.109857Z","shell.execute_reply":"2021-12-02T14:39:25.122054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#submission_test_df","metadata":{"execution":{"iopub.status.busy":"2021-12-02T14:39:25.130937Z","iopub.execute_input":"2021-12-02T14:39:25.132045Z","iopub.status.idle":"2021-12-02T14:39:25.145944Z","shell.execute_reply.started":"2021-12-02T14:39:25.131996Z","shell.execute_reply":"2021-12-02T14:39:25.144817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}