{"cells":[{"metadata":{"_uuid":"5e1444ad03cf113b4b7c77c012dac45fe14a748e"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"e63d5005e39954bdd8ce20c90b4c6b1e5112fe1e"},"cell_type":"markdown","source":"# PUBG: Exploratory analysis and predictions\n#### by Kristofer Söderström\n___\n*This notebook showcases data exploration, manipulation and predictions with a video-game database*\n### Contents\n1.  Database Description\n1. Exploratory Analysis\n1.  Baseline Model\n1.  Feature Engineering\n\n**Keywords:** Descriptive statistics, feature engineering, prediction. "},{"metadata":{"_uuid":"b693960efa172b1fee110216cc2d144a424a0f33"},"cell_type":"markdown","source":"### 1. Database Description\n\nIn a PUBG game, up to 100 players start in each match (matchId). Players can be on teams (groupId) which get ranked at the end of the game (winPlacePerc) based on how many other teams are still alive when they are eliminated. In game, players can pick up different munitions, revive downed-but-not-out (knocked) teammates, drive vehicles, swim, run, shoot, and experience all of the consequences -- such as falling too far or running themselves over and eliminating themselves.\nYou are provided with a large number of anonymized PUBG game stats, formatted so that each row contains one player's post-game stats. The data comes from matches of all types: solos, duos, squads, and custom; there is no guarantee of there being 100 players per match, nor at most 4 player per group.\n\n**Objective** : Predict a players finishing placement based on their stats.\n\n**Data fields**\n* DBNOs - Number of enemy players knocked.\n* assists - Number of enemy players this player damaged that were killed by teammates.\n* boosts - Number of boost items used.\n* damageDealt - Total damage dealt. Note: Self inflicted damage is subtracted.\n* headshotKills - Number of enemy players killed with headshots.\n* heals - Number of healing items used.\n* Id - Player’s Id\n* killPlace - Ranking in match of number of enemy players killed.\n* killPoints - Kills-based external ranking of player. (Think of this as an Elo ranking where only kills matter.) If there is a value other than -1 in rankPoints, then any 0 in killPoints should be treated as a “None”.\n* killStreaks - Max number of enemy players killed in a short amount of time.\n* kills - Number of enemy players killed.\n* longestKill - Longest distance between player and player killed at time of death. This may be misleading, as downing a player and driving away may lead to a large longestKill stat.\n* matchDuration - Duration of match in seconds.\n* matchId - ID to identify match. There are no matches that are in both the training and testing set.\n* matchType - String identifying the game mode that the data comes from. The standard modes are “solo”, “duo”, “squad”, “solo-fpp”, “duo-fpp”, and “squad-fpp”; other modes are from events or custom matches.\n* rankPoints - Elo-like ranking of player. This ranking is inconsistent and is being deprecated in the API’s next version, so use with caution. Value of -1 takes place of “None”.\n* revives - Number of times this player revived teammates.\n* rideDistance - Total distance traveled in vehicles measured in meters.\n* roadKills - Number of kills while in a vehicle.\n* swimDistance - Total distance traveled by swimming measured in meters.\n* teamKills - Number of times this player killed a teammate.\n* vehicleDestroys - Number of vehicles destroyed.\n* walkDistance - Total distance traveled on foot measured in meters.\n* weaponsAcquired - Number of weapons picked up.\n* winPoints - Win-based external ranking of player. (Think of this as an Elo ranking where only winning matters.) If there is a value other than -1 in rankPoints, then any 0 in winPoints should be treated as a “None”.\n* groupId - ID to identify a group within a match. If the same group of players plays in different matches, they will have a different groupId each time.\n* numGroups - Number of groups we have data for in the match.\n* maxPlace - Worst placement we have data for in the match. This may not match with numGroups, as sometimes the data skips over placements.\n* winPlacePerc - The target of prediction. This is a percentile winning placement, where 1 corresponds to 1st place, and 0 corresponds to last place in the match. It is calculated off of maxPlace, not numGroups, so it is possible to have missing chunks in a match."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\nimport os\nprint(os.listdir(\"../input\"))\n#loading additional dependencies \nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt #plots and graphs\nimport seaborn as sns #additional functionality and visualization\nfrom math import sqrt\nimport random\nrandom.seed(30) #seed for reproducibility\n#dependencies for preprocessing and modelling data\nfrom sklearn import preprocessing \nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error,r2_score\nimport xgboost as xgb\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a730ea2b56473e5917a49ebbec29cad26eb75457"},"cell_type":"markdown","source":"### 2. Exploratory Analysis\n\nEDA for short helps us understand the data better by summarizing main characteristics. This also helps formulating hypotheses for more data colleciton or experiments. "},{"metadata":{"trusted":true,"_uuid":"6188042642f8d5090fdd95c096b526dbd6f95cbc"},"cell_type":"code","source":"#load data and create dataframe \ntrain_data = pd.read_csv('../input/train_V2.csv')\n#summarize information \ntrain_data.info()\nprint(\"database shape:\",train_data.shape)\nbefore = train_data.shape\nprint(\"missing data?\",train_data.isnull().values.any())\nprint(\"deleting missing values...\")# dataframe has missing values, we will drop them because of time constraints. Usually not desirable since missing information can actually provide with important insights.\ntrain_data = train_data.dropna()\nprint(\"missing data?\",train_data.isnull().values.any())\nafter = train_data.shape\n#print(\"using random sample (1% of data) to speed up computation...\")\n#train_data = train_data.sample(n=None, frac=0.01, replace=False, weights=None, random_state=None, axis=None)\nprint(\"database shape:\",train_data.shape)\nprint(\"Dropped rows:\",before[0]-after[0])\ntrain_data.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"923d890bb47fa16820f980cdceef783ab2d5cc35"},"cell_type":"markdown","source":"* There is a mix of object, interger, and float data.\n* Around 4.5 million rows.\n* 29 columns \n* The **target** or **dependent variable** is continuous ranging from 0 (last place) to 1 (first place). \n* Most data is numerical. Id, groupId, matchId and matchType are object data"},{"metadata":{"_uuid":"db004b56474f23b1d0d31a07612d65daa26f9398"},"cell_type":"markdown","source":"A **heatmap** is a good way to start visualization. It will allow a bird's-eye view of the dataset and identifying correlation between features."},{"metadata":{"trusted":true,"_uuid":"894bde2d4213c0306b61dd064b588af67577cb8f"},"cell_type":"code","source":"#developing a heatmap with example from https://seaborn.pydata.org/examples/many_pairwise_correlations.html\nsns.set(style=\"white\") # set a style for the graph\ncorr = train_data.corr() # compute correlation matrix\nf, ax = plt.subplots(figsize=(15,15)) #set size\ncmap = sns.diverging_palette(220,10,as_cmap=True) #define a custom color palette\nsns.heatmap(corr,annot=False,cmap=cmap,square=True,linewidths=0.5) #draw graph\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bc1326caf1152f70943156ed8d835a996e128822"},"cell_type":"markdown","source":"It's already possible to make out some insights from the data.The upper left quadrant of the heatmap let's us know that some of the features (independent variables) are highly correlated, for example kills and damage dealt, or heals and boost. In econometrics, this is called multicollinearity. It is not as big of a problem for prediction models (in the sample dataset) than it is for explanatory models. However, there are some ways to deal with this problem in machine learning, like feature engineering by hand or principal component analysis (PCA).\n\nThe objective is to predict the finishing placement (winPlacePerc),and we can see that some variables are highly correlated. However, it is also interesting to see what variables have almost zero or inverse relationship with the target, such as:\n* walkDistance, high correlation\n* boosts, high correlation\n* damageDealt, high correlation\n* roaDkills, low correlation\n* matchDuration, low correlation\n* killPlace, negative correlation.\n    * This feature refers to how many enemy players someone kils during a match. Perhaps implying a strategy for winning players. \n    \nWe can zoom in on those to get a better understanding of the potential predictors."},{"metadata":{"trusted":true,"_uuid":"76accb5c7f222fa5561c1df739b5fe336958c192"},"cell_type":"code","source":"n = 10\nf, ax = plt.subplots(figsize=(15,15))\ncols = train_data.corr().nlargest(n,\"winPlacePerc\")[\"winPlacePerc\"].index\ncorr = np.corrcoef(train_data[cols].values.T)\nsns.heatmap(corr, annot=True,cmap=cmap,square=True,linewidths=0.5,xticklabels=cols.values,yticklabels=cols.values)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f49c5dacd9d4d0cef5a7cfbea390730543f85c9"},"cell_type":"markdown","source":"Walking distance is the feature with the highest correlation. This makes sense if we think of walking distance as a proxy for persistence in the game. This is a good example of correlation vs causation. Obviously, running is not going to make you win first place in a match but it does give a signal that will be useful for prediction. "},{"metadata":{"_kg_hide-output":false,"trusted":true,"scrolled":true,"_uuid":"624dd54bb56f4091b6c4a9ec4e7c75ac83e85792"},"cell_type":"code","source":"sns.set_style(\"white\")\nf, ax = plt.subplots(figsize=(8,5))\nsns.scatterplot(x=\"walkDistance\",y=\"winPlacePerc\",data=train_data)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eb41280ef9fa19f1458975cb31267ed4fd4a09a0"},"cell_type":"markdown","source":"There are some observations to  be made here: \n1. There is a bias towards zero\n    1. There are players that appear to win the match without moving, which might indicate some sort of cheating in the game, although more thorough examination must be done to confirm this. \n1. There are a number of outliers skewing the graph. \n\nIt's easier (but computationally more expensive) to see the relationship when using a line plot\n"},{"metadata":{"trusted":true,"_uuid":"554fb0408989952e9f7513677b633926b53c93f5"},"cell_type":"code","source":"sns.set_style(\"white\")\nf, ax = plt.subplots(figsize=(8,5))\nsns.lineplot(x=\"walkDistance\", y=\"winPlacePerc\", data=train_data)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"abbfd94009214e1d46facd7816bb2fc810a04adf"},"cell_type":"markdown","source":"There is an interesting logarithmic trend seen in the data that plateaus around 4000 meters. There is a steady trend between the range of 2000 to 4000 meteres travelled that seems to be associated with finishing placement. Perhaps this signifies the walking distance from the outer to inner-most rings during the matches."},{"metadata":{"trusted":true,"_uuid":"2e74ce638ead1efefa564ed40bfe3c44045c24ed"},"cell_type":"code","source":"sns.set(style=\"white\")\ncols = ['winPlacePerc','walkDistance', \n        'boosts','weaponsAcquired',\n        'damageDealt','heals']\nsns.pairplot(train_data[cols])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bccc51f0752a297ac5389ff8bfcb886a44189e73"},"cell_type":"markdown","source":"As expected, based on their correlation coefficient, it is possible to see a strong positive association between the features and the target variable. It is also possible to see the multicollinearity between some of the features. \nThe diagonal graphs show the histogram of each variable, here the skewness towards zero is even more evident. This will warrant some sort of normalization in the data to achieve more accurate results. \nUsually, data exploration would be performed deeper to better understand the data. For the purposes of this demo, we will jump ahead to a baseline model. "},{"metadata":{"_uuid":"74ebe0c96bd7eab85a0b8d075de21432cd266bf8"},"cell_type":"markdown","source":"### 3. Baseline Model\n#### Linear Regression\n\nThe baseline model will be the most simplistic model possible from which we can compare and gauge our results in more complex modelling. It is a good starting point where not much change is implemented to the original dataset. Before we do this, we need to separate features and targets, as well as transforming matchType to categorical values, since it might include some useful information for the model. For now, we drop id, groupId and matchId"},{"metadata":{"trusted":true,"_uuid":"0eaa06b8be4ff90ae59690dab2f535956799824d"},"cell_type":"code","source":"train_data[\"matchType\"].unique() # we can see the different types of matches in the game","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"663b05abeec22eec24e80cf02c73e91e4d346dfb"},"cell_type":"code","source":"#we start by diving the training data between features and targets\nx_train = train_data.iloc[:,3:-1]\ny_train = train_data.iloc[:,-1].values\nprint(\"traning data has the shape:\",x_train.shape)\n#one hot encoding matchType to include in analysis, it has 16 different types which might reflect specific characteristics between the match types\nx_train = pd.get_dummies(x_train, prefix = [\"matchType\"])\nprint(\"x_train shape after one hot encoding\",x_train.shape)\nprint(\"y_train shape\",y_train.shape)\n\n#we will normalize data to facilitate learning\nprint(\"normalizing data...\")\n#x_train = preprocessing.StandardScaler().fit_transform(x_train)\nx_train = preprocessing.scale(x_train)\n\nprint(\"validation split to 20%\")#\nX_train,X_val,y_train,y_val = train_test_split(x_train,y_train,\n                                               test_size=0.2,\n                                               random_state=30)\nprint(\"fitting linear regression...\")\nreg = LinearRegression()\nfit = reg.fit(X_train,y_train)\ny_predicted = reg.predict(X_val)\nprint('Train R2',reg.score(X_train,y_train))\nprint('Val R2', r2_score(y_val,y_predicted))\nprint('Val RMSE', sqrt(mean_squared_error(y_val, y_predicted)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1a08fa73d58dbc093b188555113d2a84c4492452"},"cell_type":"markdown","source":"Our baseline model is an out-of-the-box Linear Regression. It is adequated for regression problems where we have abundance of data and are not concerned with selecting only a few features. It should be noted that normalization was done to the features to facilitate learning by the model.\n1. Training and validation is around 84%, indicating robusteness and a generous fit out of the box. \n2. The value of the RMSE implies a 12.3% mean error (according to the scale of the target) between the predicted and validation data. Compared to more advanced models, this is probably high. Let's try a more complex model."},{"metadata":{"_uuid":"5a7a1b3545afbeac98f73d25b9a390f13cce8b69"},"cell_type":"markdown","source":"#### XGBoost: A more complex model\nThis is a powerful algorithm for machine learning which usually outperforms every other complex algorithm in terms of ease of use and speed. "},{"metadata":{"trusted":true,"_uuid":"c9dde50883bc04d06ebf5bf2f6c9c824f31d58b1"},"cell_type":"code","source":"reg = xgb.XGBRegressor()                        \nprint(\"fitting xgboost regression\")\nfit = reg.fit(X_train,y_train)\ny_predicted = reg.predict(X_val)\nprint('Train R2',reg.score(X_train,y_train))\nprint('Val R2', r2_score(y_val,y_predicted))\nprint('Val RMSE', sqrt(mean_squared_error(y_val, y_predicted)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b340fd7dcd90843bb412f783cfcd2995bce117a9"},"cell_type":"markdown","source":"The default XGBoost model outpeformed Linear Regression in all fronts. Increasing training and validation R2 to around 90% and reducing the error to 9.8%. However, computation time was significantly higher.\nFor now we can continue to feature engineering."},{"metadata":{"_uuid":"a7332e03e3f06d00f41aff3717f745fbefac7185"},"cell_type":"markdown","source":"### 4. Feautre Engineering\nThe process of feature engineering requires domain knowledge to create features, it is one of the main components of applied machine learning and very time consuming. \nI have never played this particular game but I am familiar with battle royale style games and gaming in general, which might help with the objective of predicting final placement.\n#### Feature creation and selection\nBased on our correlation matrix earlier, it is possible to see some variables that do not seem to be strong predictors for the taget. Whether they should be dropped from the model is not easy to say at first glance. However, it's possible to combine some features that are already correlated with each other to reduce multicollinearity and increasing robustness. \n* Boosts and heals are items that might increase a players **passive capabilities**, instead of actively, like a weapon would. "},{"metadata":{"trusted":true,"_uuid":"eb70e745af25cc81a477f45decf4e833c147c32e"},"cell_type":"code","source":"#boosts and heals \nprint(\"correlation between passive items and finishing placement:\")\ntrain_data[\"_passiveItems\"] = train_data[\"boosts\"]+train_data[\"heals\"]\nprint(np.corrcoef(train_data[\"_passiveItems\"],train_data[\"winPlacePerc\"])) #corrcoefficient\n#correlation graph\nsns.set_style(\"white\")\nf, ax = plt.subplots(figsize=(8,5))\nsns.scatterplot(x=\"_passiveItems\",y=\"winPlacePerc\",data=train_data,legend=\"full\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2518ff8418f97f4efa4c38fdb40e798f336027ff"},"cell_type":"markdown","source":"* If we think of **distance** as just another measure of persistence in the match, we can go ahead and combine ride, swim and walk distnance into a single feature of total distance travelled. However, we can see a sharp decrease in the correlation coefficient when compared to walkDistance. This might affect results "},{"metadata":{"trusted":true,"_uuid":"a9dc8750c866fd164d1c6e5a3a99154f42ddd096"},"cell_type":"code","source":"#total distance\nprint(\"correlation total distance travelled and finishing placement:\")\ntrain_data[\"_totalDistance\"] = train_data[\"walkDistance\"]+train_data[\"rideDistance\"]+train_data[\"swimDistance\"]\n\nprint(np.corrcoef(train_data[\"_totalDistance\"],train_data[\"winPlacePerc\"])) #corrcoefficient\n#correlation graph\nsns.set_style(\"white\")\nf, ax = plt.subplots(figsize=(8,5))\nsns.lineplot(x=\"_totalDistance\",y=\"winPlacePerc\",data=train_data,legend=\"full\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5199df9fd6de02aade4e51e5ba5d69a352fefd0e"},"cell_type":"markdown","source":"* In econometrics, the use of dummy variables aims to control for unseen characteristics in a model. We might not know how exactly they interact with the target, but we suppose they are there. Similarly, there might be match specific characteristics that ultimately impact the outcome of a match. Any set of characteristics that are particular to the match in progress. "},{"metadata":{"trusted":true,"_uuid":"20bbe71da3a31bad17dfafb6fc41200a6f66e673"},"cell_type":"code","source":"train_data[\"matchId\"].describe()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_uuid":"68dbd88afbe4270bd14ccc72806f5fbe69fb542a"},"cell_type":"markdown","source":"There are 47,964 unique matches in the data. If we tried to do one-hot-encoding, it would add the same amount of columns to the database. In terms of added information vs cost, this might not be worth exploring. We will attempt to do so anyways to see what happens. Deep learning models perform better with more data and perhaps there is a signal here that we would be missing out from. \nUnfortunately, attempting to one hot encode all the unique matches results in a memory error. While using categorical values in one column instead would solve that problem, it would incorrectly assign cardinality to the variable (one match is not necessarily \"better\" or \"worse\" than another). An attempt was to use a random sample of 1% of the data which made the one hot encoding possible. However, even using linear regression would result again in a memory error. "},{"metadata":{"trusted":true,"_uuid":"cab140818f7fd1419629c04106e73b8a2c8d37ff"},"cell_type":"code","source":"#skipped\n#adding matchId\n#x_train = train_data.iloc[:,2:-1]\n#y_train = train_data.iloc[:,-1].values\n#print(\"traning data has the shape:\",x_train.shape)\n\n#x_train = pd.get_dummies(x_train, prefix = [\"matchId\",\"matchType\"])\n#print(\"x_train shape after one hot encoding\", x_train.shape)\n#print(\"y_train shape\", y_train.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1b00ee368c80686302794d57eae529452afccb91"},"cell_type":"markdown","source":"1. Let's see how our constructed features perform on xgboost relative to the base models, we will only drop the features that where used to construct our own and the ID columns"},{"metadata":{"trusted":true,"_uuid":"78c1cd253b1da15b24b2af3ce5dcefcc492bde0e"},"cell_type":"code","source":"train_data_feat = train_data.drop(columns=[\"boosts\",\"heals\",\"walkDistance\",\n                                           \"rideDistance\",\"swimDistance\",\n                                           \"Id\",\"groupId\",\"matchId\"])\n\n#once again we construct our x and y sets, one hot encode matchType and normalize the data\nx_train = train_data_feat.drop(columns=[\"winPlacePerc\"])\ny_train = train_data[[\"winPlacePerc\"]].values\nprint(\"traning data has the shape:\",x_train.shape)\n#one hot encoding matchType to include in analysis\nx_train = pd.get_dummies(x_train, prefix = [\"matchType\"])\nprint(\"x_train shape after one hot encoding\", x_train.shape)\nprint(\"y_train shape\", y_train.shape)\n\n#we will normalize data to facilitate learning\nprint(\"normalizing data...\")\nfrom sklearn import preprocessing \nx_train = preprocessing.scale(x_train)\n\n#training-validation split\nX_train,X_val,y_train,y_val = train_test_split(x_train,y_train,test_size=0.2,random_state=30)\nprint(\"fitting xgboost regression...\")\n\nreg = xgb.XGBRegressor()  \nfit = reg.fit(X_train,y_train)\ny_predicted = reg.predict(X_val)\nprint('Train R2',reg.score(X_train,y_train))\nprint('Val R2', r2_score(y_val,y_predicted))\nprint('Val RMSE', sqrt(mean_squared_error(y_val, y_predicted)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f6319890301556b200e7acd4615d959a571d80c1"},"cell_type":"markdown","source":"It seems that our constructed variables are not helping increasing the prediction capacity of the model, as measured by R2 on the validation data. It might be that since we are decreasing the number of variables instead of increasing it, we are taking away signals from the model. However, the similar results with the previous model might signify some level of robustness in the model. \n\nOur xgboost regression model with not constructed variables was the better performer. Next steps could be trying to create better features, as well as tuning the current model or trying other approaches like neural networks."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}