{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Dealing with Multiple Games in the Data\n\nThere are a few errors and problems in the data, like **error in the index and elapsed time columns**, **multiple games being present in the same session id**, etc. For the index column, you can refer to [this][1] amazing notebook by @abaojiang. He provides a simple workaround to fix the index column and also gives some other important insights.\n\nIn this notebook I'll try to remove the extra games present in some sessions.\n\nI hope this helps :)\n\n**Version Updates:**\n\nVersion 2: \n\nUpdated with the new data. \n\n[1]: https://www.kaggle.com/code/abaojiang/eda-on-game-progress?scriptVersionId=120133716","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport warnings\nwarnings.filterwarnings('ignore')\n\n#Import the data\n\ndtypes={'session_id':'int', \n'elapsed_time':np.int32,\n    'event_name':'category',\n    'name':'category',\n    'level':np.int32,\n    'page':'category',\n    'room_coor_x':np.float32,\n    'room_coor_y':np.float32,\n    'screen_coor_x':np.float32,\n    'screen_coor_y':np.float32,\n    'hover_duration':np.float32,\n     'text':'category',\n     'fqid':'category',\n     'room_fqid':'category',\n     'text_fqid':'category',\n     'fullscreen':'category',\n     'hq':'category',\n     'music':'category',\n     'level_group':'category'}\n\ntrain = pd.read_csv('/kaggle/input/predict-student-performance-from-game-play/train.csv', dtype=dtypes)\ntrain.drop(['fullscreen','hq','music','index'], axis=1, inplace=True)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-03T03:59:00.656202Z","iopub.execute_input":"2023-04-03T03:59:00.656746Z","iopub.status.idle":"2023-04-03T04:00:47.850939Z","shell.execute_reply.started":"2023-04-03T03:59:00.656711Z","shell.execute_reply":"2023-04-03T04:00:47.849837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Removing the Extra Games\n\nIn some sessions in the data, multiple games are present. This means the student completed the game once and then played again either from the start or from somewhere in between. The hosts were asked how the Kaggle API handles these users by @cdeotte [here][1]. But we've still got no reply. \n\nI assume that multiple games will not be present in the test data. So the best fix will be to remove the extra games as they provide information about the student from the future and hence can give us a higher and unrealistic validation score.\n\n[1]: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479#2149959","metadata":{}},{"cell_type":"code","source":"# There are some sessions which don't start with the 0 index due to the error in this column, so we make a new temporary index\ntrain['cons'] = 1\ntrain['index1'] = train.groupby('session_id')['cons'].agg('cumsum') - 1\ntrain.drop(['cons'], axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2023-04-03T04:01:15.929124Z","iopub.execute_input":"2023-04-03T04:01:15.929499Z","iopub.status.idle":"2023-04-03T04:01:17.468245Z","shell.execute_reply.started":"2023-04-03T04:01:15.929465Z","shell.execute_reply":"2023-04-03T04:01:17.467235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First we calculate a column with level differences to know at which point the level in the data changes for each session id. The level goes from 0 to 22, so the values of level difference should normally be 0,1 and -22. (-22 for the point where a session ends and a new session starts (0-22 = -22))","metadata":{}},{"cell_type":"code","source":"train['level_diff'] = train.level.diff() #Calculate the level differences\ntrain['level_diff'] = train['level_diff'].replace(0, np.nan) #Replace the level_diff of the successive rows with same level with nan\ntrain.loc[train['index1']==0,'level_diff'] = 0 #Set the starting level_diff of each session to 0 to get rid of -22 values\ntrain['level_diff'] = train['level_diff'].ffill() #Forward fill the nan values\ntrain['level_diff'] = train['level_diff'].fillna(0) #Fill the remaining nans with 0","metadata":{"execution":{"iopub.status.busy":"2023-04-03T04:01:19.591729Z","iopub.execute_input":"2023-04-03T04:01:19.592752Z","iopub.status.idle":"2023-04-03T04:01:20.724478Z","shell.execute_reply.started":"2023-04-03T04:01:19.592713Z","shell.execute_reply":"2023-04-03T04:01:20.723450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Visualize the level_diff column\ntrain.loc[0:5,['session_id','level','level_diff']]","metadata":{"execution":{"iopub.status.busy":"2023-04-03T04:01:22.754153Z","iopub.execute_input":"2023-04-03T04:01:22.754893Z","iopub.status.idle":"2023-04-03T04:01:22.771732Z","shell.execute_reply.started":"2023-04-03T04:01:22.754852Z","shell.execute_reply":"2023-04-03T04:01:22.770682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Visualize More\ntrain.loc[162:168,['session_id','level','level_diff']]","metadata":{"execution":{"iopub.status.busy":"2023-04-03T04:01:31.118206Z","iopub.execute_input":"2023-04-03T04:01:31.118951Z","iopub.status.idle":"2023-04-03T04:01:31.130384Z","shell.execute_reply.started":"2023-04-03T04:01:31.118902Z","shell.execute_reply":"2023-04-03T04:01:31.129266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So now, the only normal values that remain for the level_diff column are 0s and 1s (As we removed -22 by replacing it with 0).","metadata":{}},{"cell_type":"code","source":"#Different values of level_diff column\ntrain.level_diff.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-04-03T04:01:35.685420Z","iopub.execute_input":"2023-04-03T04:01:35.686157Z","iopub.status.idle":"2023-04-03T04:01:36.047783Z","shell.execute_reply.started":"2023-04-03T04:01:35.686116Z","shell.execute_reply":"2023-04-03T04:01:36.046672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note: All the negative values are the level_diff of rows which belong to extra games. We'll drop all the these rows.\n\nBut as you can see, there is also a value 2 here. It represents some sessions which skip a level. I think this is also an error in the data. I talk about this [here.][1] You can either drop these sessions or just leave them be. It's totally upto you. I'll just ignore them for now.\n\n[1]: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/390339","metadata":{}},{"cell_type":"code","source":"#Now we start dropping the extra games\ntrain = train[(train['level_diff']==0) | (train['level_diff']==1) | (train['level_diff']==2)]","metadata":{"execution":{"iopub.status.busy":"2023-04-03T04:01:46.954363Z","iopub.execute_input":"2023-04-03T04:01:46.955070Z","iopub.status.idle":"2023-04-03T04:01:49.394518Z","shell.execute_reply.started":"2023-04-03T04:01:46.955031Z","shell.execute_reply":"2023-04-03T04:01:49.393500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Calculate level_diff again\ntrain['level_diff'] = train.level.diff()\ntrain['level_diff'] = train['level_diff'].replace(0, np.nan)\ntrain.loc[train['index1']==0,'level_diff'] = 0\ntrain['level_diff'] = train['level_diff'].ffill()\ntrain['level_diff'] = train['level_diff'].fillna(0)","metadata":{"execution":{"iopub.status.busy":"2023-04-03T04:01:49.720628Z","iopub.execute_input":"2023-04-03T04:01:49.721426Z","iopub.status.idle":"2023-04-03T04:01:50.857529Z","shell.execute_reply.started":"2023-04-03T04:01:49.721388Z","shell.execute_reply":"2023-04-03T04:01:50.856511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.level_diff.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-04-03T04:01:50.859465Z","iopub.execute_input":"2023-04-03T04:01:50.859916Z","iopub.status.idle":"2023-04-03T04:01:51.212652Z","shell.execute_reply.started":"2023-04-03T04:01:50.859871Z","shell.execute_reply":"2023-04-03T04:01:51.211464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we just continue the same process until we are left with three nuniques (0,1 and 2) in the level_diff column.","metadata":{}},{"cell_type":"code","source":"iterr = 1\n\nwhile train.level_diff.nunique() !=3:\n    \n    train = train[(train['level_diff']==0) | (train['level_diff']==1) | (train['level_diff']==2)]\n    \n    train['level_diff'] = train.level.diff()\n    train['level_diff'] = train['level_diff'].replace(0, np.nan)\n    train.loc[train['index1']==0,'level_diff'] = 0\n    train['level_diff'] = train['level_diff'].ffill()\n    train['level_diff'] = train['level_diff'].fillna(0)\n    \n    print(f'(Iter{iterr})', f'Nuniques:{train.level_diff.nunique()}, ', end='')\n    iterr += 1","metadata":{"execution":{"iopub.status.busy":"2023-04-03T04:01:54.031722Z","iopub.execute_input":"2023-04-03T04:01:54.032096Z","iopub.status.idle":"2023-04-03T04:03:18.808418Z","shell.execute_reply.started":"2023-04-03T04:01:54.032062Z","shell.execute_reply":"2023-04-03T04:03:18.807408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.level_diff.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-04-03T04:03:18.810597Z","iopub.execute_input":"2023-04-03T04:03:18.810999Z","iopub.status.idle":"2023-04-03T04:03:19.162352Z","shell.execute_reply.started":"2023-04-03T04:03:18.810960Z","shell.execute_reply":"2023-04-03T04:03:19.161104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wohoo!!! We're done. Now we can save the dataset and use it to build some amazing models!\n\nAll the best for the competition!","metadata":{}},{"cell_type":"code","source":"#Save the dataset\ntrain.to_csv('train.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-04-03T04:03:51.398514Z","iopub.execute_input":"2023-04-03T04:03:51.399219Z"},"trusted":true},"execution_count":null,"outputs":[]}]}