{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Intro","metadata":{}},{"cell_type":"markdown","source":"The notebook is used for simple data analysis and will start on the FE & ML part later. Please feel free to leave a comment if there is any suggestions & improvement. Thanks for viewing!!!\n\nWill use train_0 for notebook only since the memory in Kaggle ram is not avaliable.","metadata":{}},{"cell_type":"markdown","source":"## Basic Setting","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport math\nimport seaborn as sns","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:37:25.764119Z","iopub.execute_input":"2022-10-16T13:37:25.764542Z","iopub.status.idle":"2022-10-16T13:37:26.504829Z","shell.execute_reply.started":"2022-10-16T13:37:25.764505Z","shell.execute_reply":"2022-10-16T13:37:26.503841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Loading part","metadata":{}},{"cell_type":"markdown","source":"Following the guidelline from competition page","metadata":{}},{"cell_type":"markdown","source":"### Training data set","metadata":{}},{"cell_type":"code","source":"# read the training data type\ndtypes_df = pd.read_csv('../input/tabular-playground-series-oct-2022/train_dtypes.csv')\n\n# assign a dict as an input for read_csv\ndtypes = {k : v for (k, v) in zip(dtypes_df['column'], dtypes_df['dtype'])}\n\n# create an empty dateframe and concat for each dataset\n# raw_df = pd.DataFrame()\n# for i in range(10):\n#    temp_df = pd.read_csv(f'../input/tabular-playground-series-oct-2022/train_{i}.csv', dtype = dtypes)\n#    raw_df = pd.concat([raw_df, temp_df], ignore_index = True)\nraw_df = pd.read_csv('../input/tabular-playground-series-oct-2022/train_0.csv', dtype = dtypes)","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:37:28.137707Z","iopub.execute_input":"2022-10-16T13:37:28.138134Z","iopub.status.idle":"2022-10-16T13:38:17.676263Z","shell.execute_reply.started":"2022-10-16T13:37:28.138098Z","shell.execute_reply":"2022-10-16T13:38:17.674733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check the columns in whole dataset\nraw_df.columns","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:39:15.753840Z","iopub.execute_input":"2022-10-16T13:39:15.754484Z","iopub.status.idle":"2022-10-16T13:39:15.767213Z","shell.execute_reply.started":"2022-10-16T13:39:15.754444Z","shell.execute_reply":"2022-10-16T13:39:15.766253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Base on the instructure of competitoin data page, these columns are for training set only.","metadata":{}},{"cell_type":"markdown","source":"- game_num, event_id -> These are the unique identifier in training set\n\n- event_time -> column that the time before the event ended\n\n- player_scoring_next -> which player scores at the end of the current event [0,6) or -1 if the event does not end in a goal\n\n- team_scoring_next -> team scores at the end of current event (A or B)\n\n- team_[A|B]_scoring_within_10sec -> [Target Column], 1 if term_scoring_next == [A|B] & time_before_event is in [-10, 0], else 0\n\nFor game_num & event_id, it's a unique identifier and it won't provide any information for model to learn as it should not be any relationship between the identifier & the scroing within 10sec. -> These can be eliminated.\n\nFor event_time, player_scoring_next, team_scoring_next & team_[A|B]_scoring_within_10sec, these are only in the training set and we cannot found it in testing set. These could provide some extra information for model to learn and the approch will be provide some weighting for other columns such as team_[A|B]_scoring_within_10sec, team_scoring_next. These are the column related to the scoring infomation. We can keep it and check is there any relationship later.","metadata":{}},{"cell_type":"code","source":"# copy the training set from the raw_df\ntrain_df = raw_df.copy()\n\n# eliminate the unique identifier columns\ntrain_df = train_df.drop(['game_num', 'event_id'], axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:39:19.976530Z","iopub.execute_input":"2022-10-16T13:39:19.977043Z","iopub.status.idle":"2022-10-16T13:39:20.393843Z","shell.execute_reply.started":"2022-10-16T13:39:19.977011Z","shell.execute_reply":"2022-10-16T13:39:20.392275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check the shape for training data\nprint(train_df.shape)\n\n# check the missing value in the training set\nfor column in train_df.columns:\n    print(f\"Missing value for {column}: {train_df[column].isna().sum()}\")","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:39:21.608296Z","iopub.execute_input":"2022-10-16T13:39:21.608718Z","iopub.status.idle":"2022-10-16T13:39:22.033046Z","shell.execute_reply.started":"2022-10-16T13:39:21.608685Z","shell.execute_reply":"2022-10-16T13:39:22.031646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Testing data set","metadata":{}},{"cell_type":"code","source":"# read the testing data type\ndtypes_df = pd.read_csv('../input/tabular-playground-series-oct-2022/test_dtypes.csv')\n\n# assign the testing date type for read csv function\ndtypes = {k : v for (k, v) in zip(dtypes_df['column'], dtypes_df['dtype'])}\n\n# read the data with defined data column type\nraw_test_df = pd.read_csv('../input/tabular-playground-series-oct-2022/test.csv', dtype = dtypes, index_col = 'id')\n\n# copy the testing set from the raw data\ntest_df = raw_test_df.copy()","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:39:29.335152Z","iopub.execute_input":"2022-10-16T13:39:29.335986Z","iopub.status.idle":"2022-10-16T13:39:43.570579Z","shell.execute_reply.started":"2022-10-16T13:39:29.335938Z","shell.execute_reply":"2022-10-16T13:39:43.569255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check the columns in whole testing dataset\ntest_df.columns","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:39:43.572777Z","iopub.execute_input":"2022-10-16T13:39:43.573955Z","iopub.status.idle":"2022-10-16T13:39:43.583635Z","shell.execute_reply.started":"2022-10-16T13:39:43.573878Z","shell.execute_reply":"2022-10-16T13:39:43.581997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check the shape for testing data\nprint(test_df.shape)\n\n# check the missing value in the testing set\nfor column in test_df.columns:\n    print(f\"Missing value for {column}: {test_df[column].isna().sum()}\")","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:39:43.585048Z","iopub.execute_input":"2022-10-16T13:39:43.585446Z","iopub.status.idle":"2022-10-16T13:39:43.704324Z","shell.execute_reply.started":"2022-10-16T13:39:43.585412Z","shell.execute_reply":"2022-10-16T13:39:43.703043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check the portion of missing data for both set\n# getthing the second large missing count which is reasonable\ntrain_second_large_missing = [train_df[column].isna().sum() for column in train_df.columns]\ntrain_second_large_missing.sort()\n\n# Display the missing portion exclude the team_scoring_next value which is not include in testing set\nprint(f\"Training set: {train_second_large_missing[-2]/train_df.shape[0]}\")\nprint(f\"Testing set: {max([test_df[column].isna().sum() for column in test_df.columns])/ test_df.shape[0]}\")\n\n# Calculate the differences\ndifferences = (train_second_large_missing[-2]/train_df.shape[0]) - (max([test_df[column].isna().sum() for column in test_df.columns])/ test_df.shape[0])\nprint(f\"Differences: {abs(differences)}\")","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:39:43.707431Z","iopub.execute_input":"2022-10-16T13:39:43.708785Z","iopub.status.idle":"2022-10-16T13:39:44.299485Z","shell.execute_reply.started":"2022-10-16T13:39:43.708737Z","shell.execute_reply":"2022-10-16T13:39:44.298337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The different for the missing part between training & testing set is not that much for the largest column without team_scoring_next.","metadata":{}},{"cell_type":"code","source":"# drop the row with no team_scoring_next value and check again the missing row\ndrop_train_df = train_df.drop(train_df[train_df['team_scoring_next'].isna()].index)\nfor column in drop_train_df.columns:\n    print(f\"Missing value for {column}: {drop_train_df[column].isna().sum()}\")","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:39:46.697880Z","iopub.execute_input":"2022-10-16T13:39:46.698270Z","iopub.status.idle":"2022-10-16T13:39:47.886474Z","shell.execute_reply.started":"2022-10-16T13:39:46.698237Z","shell.execute_reply":"2022-10-16T13:39:47.885102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After removing the missing team_scoring_next data row, there is still some number for thr player position is missing. \n\nThat's mean these rows still got the information about the team scroing next no matter it is A team or B team. We should keep it or using other inforamtion to indicate the position of existing player for the game & the team_scoring_next column will be dropped since we cant find it in the testing set or it can help to give a weighting for that row of data.","metadata":{}},{"cell_type":"markdown","source":"## Data analysis","metadata":{}},{"cell_type":"code","source":"# drop the event_time column as it shoule not be included in the analysis\ntrain_df.drop('event_time', axis = 1, inplace = True)","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:39:50.980787Z","iopub.execute_input":"2022-10-16T13:39:50.981583Z","iopub.status.idle":"2022-10-16T13:39:51.223647Z","shell.execute_reply.started":"2022-10-16T13:39:50.981546Z","shell.execute_reply":"2022-10-16T13:39:51.222545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training set type assignment","metadata":{}},{"cell_type":"code","source":"# Let assign differnt type of training set column for statistic analysis\nint_list = []\nfloat_list = []\nobject_list = []\nfor i in train_df.columns:\n    if 'int' in str(train_df[i].dtypes):\n        int_list.append(i)\n    elif 'float' in str(train_df[i].dtypes):\n        float_list.append(i)\n    else:\n        object_list.append(i)","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:39:53.472955Z","iopub.execute_input":"2022-10-16T13:39:53.473357Z","iopub.status.idle":"2022-10-16T13:39:53.485209Z","shell.execute_reply.started":"2022-10-16T13:39:53.473325Z","shell.execute_reply":"2022-10-16T13:39:53.483876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"int type features: {int_list}\")\nprint(f\"float type features: {float_list}\")\nprint(f\"object type features: {object_list}\")","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:39:55.637766Z","iopub.execute_input":"2022-10-16T13:39:55.638152Z","iopub.status.idle":"2022-10-16T13:39:55.645664Z","shell.execute_reply.started":"2022-10-16T13:39:55.638122Z","shell.execute_reply":"2022-10-16T13:39:55.644042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Testing set type assignment","metadata":{}},{"cell_type":"code","source":"# Let assign differnt type of training set column for statistic analysis\nint_list_test = []\nfloat_list_test = []\nobject_list_test = []\nfor i in test_df.columns:\n    if 'int' in str(test_df[i].dtypes):\n        int_list_test.append(i)\n    elif 'float' in str(test_df[i].dtypes):\n        float_list_test.append(i)\n    else:\n        object_list_test.append(i)","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:39:58.265369Z","iopub.execute_input":"2022-10-16T13:39:58.265771Z","iopub.status.idle":"2022-10-16T13:39:58.274432Z","shell.execute_reply.started":"2022-10-16T13:39:58.265739Z","shell.execute_reply":"2022-10-16T13:39:58.272968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"int type features: {int_list_test}\")\nprint(f\"float type features: {float_list_test}\")\nprint(f\"object type features: {object_list_test}\")","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:40:01.204896Z","iopub.execute_input":"2022-10-16T13:40:01.205315Z","iopub.status.idle":"2022-10-16T13:40:01.211703Z","shell.execute_reply.started":"2022-10-16T13:40:01.205281Z","shell.execute_reply":"2022-10-16T13:40:01.210148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### int type","metadata":{}},{"cell_type":"markdown","source":"There is only the int type for the targeting columns & player_scoring_next column.","metadata":{}},{"cell_type":"code","source":"# checking the range for the int_type feature in test set\nfor i in int_list:\n    temp_list = list(train_df[i].unique())\n    temp_list.sort()\n    print(f\"{i}, min: {temp_list[0]}, max: {temp_list[-1]}, number of value: {len(temp_list)}\")","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:40:03.322018Z","iopub.execute_input":"2022-10-16T13:40:03.322458Z","iopub.status.idle":"2022-10-16T13:40:03.379701Z","shell.execute_reply.started":"2022-10-16T13:40:03.322424Z","shell.execute_reply":"2022-10-16T13:40:03.378422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For the target columns, it is boolean value and it will be correlated with other columns as the aim for the analysis.\n\nFor player_scoring_next column, it was not included in the testing set as well but it can be used as a weighting for other parameter for feature engieering.","metadata":{}},{"cell_type":"code","source":"# train set\n# check with the distribution for the int type features\n# melt the data and build a counts column for visualisation\nf = pd.melt(train_df, value_vars = int_list)\nf['counts'] = 1\nf = f.groupby(['variable','value']).sum()\nncols = 3\nnrows = round(len(int_list) / ncols)\nfig, axes = plt.subplots(nrows, ncols, figsize=(16, round(nrows*16/ncols)))\nax = axes.ravel()\nfor i in range(len(int_list)):\n    ax[i].bar(data = f.loc[int_list[i]], x = f.loc[int_list[i]].index, height = 'counts')\n    ax[i].set_title(int_list[i])","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:40:26.482177Z","iopub.execute_input":"2022-10-16T13:40:26.482649Z","iopub.status.idle":"2022-10-16T13:40:28.331753Z","shell.execute_reply.started":"2022-10-16T13:40:26.482614Z","shell.execute_reply":"2022-10-16T13:40:28.330013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df['team_A_scoring_within_10sec'].unique()","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:40:31.513953Z","iopub.execute_input":"2022-10-16T13:40:31.514887Z","iopub.status.idle":"2022-10-16T13:40:31.537920Z","shell.execute_reply.started":"2022-10-16T13:40:31.514826Z","shell.execute_reply":"2022-10-16T13:40:31.536988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the bar chart above, it's mostly -1 for the scoring_within_10sec no matter for team A & team B. \n\nThe distribution for the player score is quite similar. It is possible that we can try to summarize the player position as a one parameter instead of separate in to 6 different players as input. Just an imagination as the player location between the player should be the key point to score as an operation in a match. ","metadata":{}},{"cell_type":"markdown","source":"There is no integer type in the testing set and we can just skip it at this moment.","metadata":{}},{"cell_type":"markdown","source":"### Float type","metadata":{}},{"cell_type":"markdown","source":"Using the describe function to check about the floating column","metadata":{}},{"cell_type":"code","source":"train_df[float_list].describe()","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:40:34.209891Z","iopub.execute_input":"2022-10-16T13:40:34.210320Z","iopub.status.idle":"2022-10-16T13:40:41.193373Z","shell.execute_reply.started":"2022-10-16T13:40:34.210288Z","shell.execute_reply":"2022-10-16T13:40:41.192222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df[float_list_test].describe()","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:41:30.575644Z","iopub.execute_input":"2022-10-16T13:41:30.576056Z","iopub.status.idle":"2022-10-16T13:41:33.522449Z","shell.execute_reply.started":"2022-10-16T13:41:30.576025Z","shell.execute_reply":"2022-10-16T13:41:33.520718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# comparing the max and min for both set by it's ratio\ndf_comparing = pd.DataFrame()\ndf_comparing['max_ratio'] = train_df[float_list].describe().T['max'] / test_df[float_list_test].describe().T['max']\ndf_comparing['min_ratio'] = train_df[float_list].describe().T['min'] / test_df[float_list_test].describe().T['min']\ndf_comparing","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:40:41.212635Z","iopub.execute_input":"2022-10-16T13:40:41.213335Z","iopub.status.idle":"2022-10-16T13:41:01.089605Z","shell.execute_reply.started":"2022-10-16T13:40:41.213278Z","shell.execute_reply":"2022-10-16T13:41:01.088250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Based on the calculation of minimum & maximum part (ignoring the booster part), there is a big contrast for the position z no matter it's the ball or player position.","metadata":{}},{"cell_type":"markdown","source":"#### Simple visualization of float data","metadata":{}},{"cell_type":"code","source":"# let's do the kernal density estimation plot for distribution check\n\n# Get rid of the booster column as sd / variance cannnot be determined\nfloat_list_dis = [i for i in float_list if 'boost' not in i]\nfloat_list_dis_test = [i for i in float_list_test if 'boost' not in i]\n\n# train set\nncols = 3\nnrows = math.ceil(len(float_list_dis) / ncols)\nfig, axes = plt.subplots(nrows, ncols, figsize=(16, round(nrows*16/ncols)))\nax = axes.ravel()\nfor i in range(len(float_list_dis)):\n    # plot the distribution for both train and test set\n    ax[i] = sns.kdeplot(data = train_df, x = float_list_dis[i], label = 'training', color = 'b', fill = False, ax = ax[i])\n    ax[i] = sns.kdeplot(data = test_df,x = float_list_dis_test[i], label = 'testing', color= 'r', fill = False, ax = ax[i])\n    # show the legend for the labels\n    ax[i].legend()\n    ax[i].set_title(float_list_dis[i])","metadata":{"execution":{"iopub.status.busy":"2022-10-16T13:41:54.139408Z","iopub.execute_input":"2022-10-16T13:41:54.139781Z","iopub.status.idle":"2022-10-16T13:49:51.522916Z","shell.execute_reply.started":"2022-10-16T13:41:54.139752Z","shell.execute_reply":"2022-10-16T13:49:51.521968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Based on the visuals, the distribution for both training and testing set are similar but it's not a normal distribution for those features. Feature engineering is very important to work on the data and feed to the model. It will be the next step for my work!","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}