{"cells":[{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"553b25a99880dc0d721d9b7952661b7e2fb4d0f2"},"cell_type":"code","source":"from IPython.display import HTML\nstyle = \"\"\"\n\n\"\"\"\nHTML(style)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7571a382e7f09662ee57361a612177defdadcf11"},"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport math\nimport numpy as np\n%matplotlib inline\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a9a34e7f2dde4642938ef5b8c2a5ccff1e7695dd"},"cell_type":"markdown","source":"\n\n\nA punt is a kick where the ball must not hit the ground before the kick. It is typically performed in the final down if a team does not believed they can cover the necessary distance to touchdown or to score a field goal. Thus it is used to deny the opposition additional ground on turnover. A punter and a punt reciever are specialised roles on either team who's responsibility it is to kick / recieve the ball and place it as close as possible to the opponents endzone before the turnover\n
\n\nAll code cells have been hidden for the purposes of clarity of findings, you can easily reveal these code cells by clicking (...) throughout this notebook. This notebook details by entire exploration process, if you are judging this competition I highly reccomend looking at the slide deck / presentation to get an overview of my findings before diving into the details below! \n
\n\n\n Below we read in the game_data, video_review and play_information data sets in order to examine links between match data and concussions\n
"},{"metadata":{"trusted":true,"_uuid":"c3d14c926147c91a86802256ff279990a610d6a3"},"cell_type":"code","source":"# Game Data & Video Review Data\ngame_data = pd.read_csv('../input/game_data.csv')\nvideo_review = pd.read_csv('../input/video_review.csv')\nconcussions_and_game_data = video_review.merge(right=game_data,how='inner', on='GameKey')\n\n# Play Information Data Cleaning / Engineering\nplay_information = pd.read_csv('../input/play_information.csv')\nplay_information_punt_only = play_information[play_information['Play_Type'] == 'Punt']\n\nplay_information_punt_only[['TeamYardLine','YardLineDist']] = play_information_punt_only['YardLine'].str.split(' ', expand=True)\nplay_information_punt_only[['HomeTeam','VisitTeam']] = play_information_punt_only['Home_Team_Visit_Team'].str.split('-', expand=True)\nplay_information_punt_only[['ScoreHome','ScoreVisit']] = play_information_punt_only['Score_Home_Visiting'].str.strip().str.split('-', expand=True).astype('int64')\nplay_information_punt_only['YardLineDist'] = play_information_punt_only['YardLineDist'].astype('int64')\n\n# [Assumption] Score difference is more important than the actual scores\n# This is logically what creates a pressured environment on the pitch \n# [Hypothesis] There exists certain score differences where concussions are more likely\nplay_information_punt_only['ScoreDifference'] = play_information_punt_only['ScoreHome']-play_information_punt_only['ScoreVisit']\n\n# Combining play information with data on concussions\nplay_concussion_data = play_information_punt_only.merge(right=concussions_and_game_data,how='inner', on=['GameKey','PlayID'])\n\nhelmet_to_body = play_concussion_data[play_concussion_data['Primary_Impact_Type']=='Helmet-to-body']\nhelmet_to_helmet = play_concussion_data[play_concussion_data['Primary_Impact_Type']=='Helmet-to-helmet']\n\n# Combining play, game_data and player roles durin each concussion\nplayer_role = pd.read_csv('../input/play_player_role_data.csv')\nplay_concussion_data_role = play_concussion_data.merge(right=player_role,how='inner',on=['Season_Year','GameKey','PlayID','GSISID'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8dc47314915795e15eed9b0895ded660200f7a24"},"cell_type":"markdown","source":"\n\n In this section we will explore some of the following key identified features from the dataset.\n
"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"dce1f67b9d041430b4fa2545ba1c8abfa109809c"},"cell_type":"code","source":"plt.figure(figsize=(12,5))\nax0 = plt.subplot2grid((1,2),(0,0))\nax1 = plt.subplot2grid((1,2),(0,1))\nax0.set_title('Frequency of concussions by Type')\nax1.set_title('Frequency of concussions by Player Role')\n\nvideo_review.groupby(by='Primary_Impact_Type').count()['Season_Year'].plot(kind='barh', ax=ax0)\nplay_concussion_data_role.pivot_table(index='Role', columns='Quarter', aggfunc='count', values='GameKey', fill_value=0).sort_values(by=[1,2,3,4], ascending=False).plot(kind='barh', stacked=True, ax=ax1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dab48a5160dffb9acb0cb2cdd27669563cfb8140"},"cell_type":"markdown","source":"\n\n
\n \n
\n The first thing to do is to examine how frequently each type of concussion occurs. As you can see above almost all concussions fall into either helmet to helmet or helmet to body categories. This makes sense as it is difficult to see how a concussion could occur if the first contact was not with the body / helmet! We will focus specifically on these two types during initial data exploration.\n
\n
\n
\n \n
\n Above the frequency of concussion is shown by player role (for all roles which recieved concussions) it is immediately evident that some roles such as punt reciever, left guard and right guard recieve more concussions than any other role, this is clearly an important statistic as it suggest some roles are potentially more vulnerable to concussions or have a play style which leads to these events with higher probability.\n
\n
\n
\n There is evidence to suggest a realtionship between player role and concussions, this feature will likely be inlcuded in any model built.\n
"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"0baa95eb1f38b4ba1ddec62d3056d75d23360060"},"cell_type":"code","source":"plt.figure(figsize=(12,5))\nplt.suptitle('Frequency of concussions (and types thereof) over score difference')\nax0 = plt.subplot2grid((1,2),(0,0))\nax1 = plt.subplot2grid((1,2),(0,1))\n\n# Score Difference Between Teams | Concussions\nsns.distplot(play_concussion_data['ScoreDifference'], bins=6, ax=ax0)\nax0.legend(['Concussions'])\n# ax0.set_title('Score Difference and frequency of concussion impacts')\nax0.set_xlim((-40,40))\n\n# Score Difference Between Teams | Concussions [Type]\nsns.distplot(helmet_to_body['ScoreDifference'], bins=6, ax=ax1)\nsns.distplot(helmet_to_helmet['ScoreDifference'], bins=6, ax=ax1)\nax1.legend(['Helmet to Body','Helmet to helmet'])\n# ax1.set_title('Score Difference and frequency of concussion impacts')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6b6d191cecdb7744853f067f658f8a16e7a41e6d"},"cell_type":"markdown","source":"\n\n The above graphs show the relative frequency of concussions over score differences. Score differences are calculated as the Home Team Score - Visit Team Score. I do not take absolute values in the event that there is skew towards home / away team.\n
\n\n\n
\n \n
\n Overall concussions seem to be concentrated around a zero score difference with slight negative skew. At a glance this would suggest that plays with narrower score differences are likely to have more concussions. However we must also consider the possibility that there are fewer games with wider score differences.\n
\n
\n
\n \n
\n Helmet to helmet has a non zero mean where as helmet to body seems to have a zero mean and negative skew. This would suggest that the conditions under which the two major types of collisions occur may be different. Notice that there are two peaks for helemet to helmet concussions, that appear to be closer to +- 20 than to 0 score difference, this would suggest unlike our previous conclusion of zero score centred concussions that their may indeed be certain critical score differences, which put additional mental stress on players and leads to increased probabilities of concussion.\n
\n
\n
\n There is evidence to suggest a realtionship between score difference and concussions, this feature will likely be inlcuded in any model built\n
"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"28edab331334e399d075956e9e015781ada913c5"},"cell_type":"code","source":"plt.figure(figsize=(12,5))\nplt.suptitle('Frequency of concussions (and types thereof) over distace from endzone (in yards)')\nax0 = plt.subplot2grid((1,2),(0,0))\nax1 = plt.subplot2grid((1,2),(0,1))\n\n# Yard Distance From Endzone | Concussions\nsns.distplot(play_concussion_data['YardLineDist'], ax=ax0)\nax0.legend(['Concussions'])\nax0.set_xlim((0,50))\n\n# Yard Distance From Endzone | Concussions [Type]\nsns.distplot(helmet_to_body['YardLineDist'], ax=ax1)\nsns.distplot(helmet_to_helmet['YardLineDist'], ax=ax1)\nax1.legend(['Helmet to Body','Helmet to helmet'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c57981425cd71e8e123d6c86cdb13ef8ff4b47e7"},"cell_type":"markdown","source":"\n\n The above graphs show the relative frequency of concussions over yard line distances. Yard line distance has been extracted from the play_information dataset through the data cleaning process in the code above. It measures the distance from the teams respective endzone in which punt play began (ie line of scrimmage). Unfortunately since the data available for individual players does not indicate which team they reside on, we are unable to find out which end zone the yard line distance refers to with respect to the concussed player.\n
\n\n\n
\n \n
\n Overall concussions seem to be concentrated between the 20 and 33 yards from the endzone, with smaller peaks at 10 and 50 yards. This makes sense as punts are often reciever between 20 - 40 yards and then the punt reciever either advances forwards towards the 50 yard line or is forced backwards by offensive team towards his own endzone, explaining the peak at the 10 yard mark. \n
\n
\n
\n \n
\n There are several clear peaks in concussions at the 10, 25, 47 yards from the endzone. Similarly to score difference we see a gap between the peaks for the two types of concussions, further reinforcing the hypothesis that these two types of concussions have different underlying mechanics driving their occurences.\n
\n
\n
\n There is evidence to suggest a realtionship between yard line distance and concussions, clearly there are certain critical regions in which concussions are likely to occur as they represent characteristic turning points in play.\n
"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"774bd4074ff10204e9286b95804c7763dddf296e"},"cell_type":"code","source":"# Quarter Distribution | Concussions [Type]\nax1 = sns.distplot(helmet_to_body['Quarter'])\nax2 = sns.distplot(helmet_to_helmet['Quarter'])\nax1.legend(['Helmet to Body','Helmet to helmet'])\nax1.set_title('Quarter distribution of concussion impacts')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d371e348c9009ff174b54e444798bac3a06ebc6"},"cell_type":"markdown","source":"\n\n The above graphs show the relative frequency of concussions over different quarters of play, Q5 being overtime.\n
\n\n\n
\n \n
\n Both types of concussions peak in the 3rd quarter, but helmet to helmet concussions have a secondary peak in the 2nd quarter. The distribution towards the extremes is quite similar for both types of concussion\n
\n
\n
\n This feature is not as useful as some of the others as it doesn't really have any logical explanation to why more concussions should happen in the second or third half. Considering all it tells us that most concussions happen in 2 out of 4 quarters it doesn't seem particularly helpful to add 4 extra one hot encoded features to a model for limited additional explanatory power.\n
"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"6baf9c7f9497632e9d6989c083ea7fe91e574016"},"cell_type":"code","source":"plt.figure(figsize=(12,5))\nplt.suptitle('Frequency of concussions (and types thereof) over tempeature (F)')\nax0 = plt.subplot2grid((1,2),(0,0))\nax1 = plt.subplot2grid((1,2),(0,1))\n\n# Temperature | Concussions\nsns.distplot(game_data['Temperature'].dropna(), color='b', bins = 20, ax=ax0)\nsns.distplot(concussions_and_game_data['Temperature'], color='r', bins = 10, ax=ax0)\nax0.legend(['All Games', 'Concussions'])\nax0.set_xlim((0,100))\n\n# Temperature | Concussions [Type]\nsns.distplot(helmet_to_body['Temperature'], color='orange', bins=10, ax=ax1)\nsns.distplot(helmet_to_helmet['Temperature'], color='teal', bins=6, ax=ax1)\nax1.legend(['Helmet to body', 'Helmet to helmet'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1aaf64f51ebc60cf250ef747a9cca74cd50799f1"},"cell_type":"markdown","source":"\n\n The above graphs show the relative frequency of concussions over different tempeatures (Farenheit).\n
\n\n\n
\n \n
\n Concussions tend to occur at a rate that exceeds the number of games being played at more extreme temperatures, this logically makes sense as their might be additional stress on the players in colder temperatures.\n We can see this where the red line is above the blue line. The relative rate of concussions tends to fall relative to the number of games being played in normal temperatures (60 - 90F)\n
\n
\n
\n \n
\n Helemet to helmet concussions for some reason tend to occur at lower temperatures than helmet to body collisions, this is evident from the peak in the green line being at a lower temperature than the orange line. This is possibly a spurious correlation.\n
\n
\n
\n There is evidence to suggest a realtionship between temperature and concussions, given the rate of concussions increases relative to games at tempratures less than 60F and greater than 90F. This could be used as a feature in the final model.\n
"},{"metadata":{"_uuid":"2781c6248b509b61e30ee952f25bb8d4ac175c1b"},"cell_type":"markdown","source":"\n\n It would be logical to assume that players which recieve concussions are travelling at high velocity, giving them higher momentumn and colliding with greater force.\n
"},{"metadata":{"_kg_hide-input":false,"trusted":false,"_uuid":"cfee075988de72de1b8011e4bd25737e0fbbfcfe"},"cell_type":"code","source":"# Label player role data by concussion and impact type\nplayers_concussed = video_review.set_index(['Season_Year','GameKey','PlayID','GSISID'])\nplayers_concussed['Concussed'] = 1\nplayers_concussed = pd.DataFrame(players_concussed[['Concussed', 'Primary_Impact_Type']])\n\nplayer_data_NGS = player_role.set_index(['Season_Year','GameKey','PlayID','GSISID'])\nplayer_data_NGS = player_data_NGS.merge(right=players_concussed, how='left', left_index=True, right_index=True)\nplayer_data_NGS['Primary_Impact_Type'] = player_data_NGS['Primary_Impact_Type'].fillna('None')\nplayer_data_NGS = player_data_NGS.fillna(0)\n\n# Load and merge NGS 2017\nNGS_2017_reg_week_1_6 = pd.read_csv('../input/NGS-2017-reg-wk1-6.csv')\nNGS_2017_reg_week_13_17 = pd.read_csv('../input/NGS-2017-reg-wk13-17.csv')\nNGS_2017_reg_week_7_12 = pd.read_csv('../input/NGS-2017-reg-wk7-12.csv')\nNGS_2017_reg = NGS_2017_reg_week_1_6.append(NGS_2017_reg_week_7_12.append(NGS_2017_reg_week_13_17))\n\n# Calculate Velocity 2017\nNGS_2017_reg['Time'] = NGS_2017_reg['Time'].astype('datetime64[ns]')\nNGS_2017_reg = NGS_2017_reg.drop(columns='Event')\n\nNGS_2017_reg['TimeDelta'] = NGS_2017_reg['Time'].diff().dt.total_seconds()\nNGS_2017_reg = NGS_2017_reg[NGS_2017_reg['TimeDelta'] > 0.0]\nNGS_2017_reg['dis_metres'] = NGS_2017_reg['dis'] / 1.094\nNGS_2017_reg['Velocity'] = NGS_2017_reg['dis_metres'] / NGS_2017_reg['TimeDelta']\n\n#Load and merge NGS 2016\nNGS_2016_reg_week_7_12 = pd.read_csv('../input/NGS-2016-reg-wk7-12.csv')\nNGS_2016_reg_week_1_6 = pd.read_csv('../input/NGS-2016-reg-wk1-6.csv')\nNGS_2016_reg_week_13_17 = pd.read_csv('../input/NGS-2016-reg-wk13-17.csv')\nNGS_2016_reg = NGS_2016_reg_week_1_6.append(NGS_2016_reg_week_7_12.append(NGS_2016_reg_week_13_17))\n\n# Calculate Velocity 2016\nNGS_2016_reg['Time'] = NGS_2016_reg['Time'].astype('datetime64[ns]')\nNGS_2016_reg = NGS_2016_reg.drop(columns='Event')\n\nNGS_2016_reg['TimeDelta'] = NGS_2016_reg['Time'].diff().dt.total_seconds()\nNGS_2016_reg = NGS_2016_reg[NGS_2016_reg['TimeDelta'] > 0.0]\nNGS_2016_reg['dis_metres'] = NGS_2016_reg['dis'] / 1.094\nNGS_2016_reg['Velocity'] = NGS_2016_reg['dis_metres'] / NGS_2016_reg['TimeDelta']\n\n# Merge 2016 and 2017\nNGS_reg = NGS_2016_reg.append(NGS_2017_reg)\n\nNGS_group_by_play_player = NGS_reg.groupby(by=['Season_Year','GameKey','PlayID','GSISID'])\n\n# Calculate every players velocity by play and update their Next Gen Stats\nvelocity_by_play = NGS_group_by_play_player.mean()['Velocity']\nplayer_data_NGS['Velocity'] = velocity_by_play\nplayer_data_NGS['Velocity'] = player_data_NGS['Velocity'].fillna(player_data_NGS['Velocity'].mean())\n\n# Calculate the duration of each play and pass that information to player NGS\nduration_by_play = (NGS_group_by_play_player.max() - NGS_group_by_play_player.min())['Time'].dt.total_seconds()\nplayer_data_NGS['Play_Duration'] = duration_by_play\n\n# Clean Player Data Values\nplayer_data_NGS = player_data_NGS.dropna()\nplayer_data_NGS = player_data_NGS[player_data_NGS['Velocity'] < 6]\nplayer_data_NGS = player_data_NGS[player_data_NGS['Velocity'] > 10**-4]\n\n# Transform play information data for NGS format\nplay_data_NGS = play_information_punt_only.set_index(['Season_Year','GameKey','PlayID'])\nplay_data_NGS = play_data_NGS[[c for c in play_data_NGS.columns if c[-2:] != '_y']]\nplay_data_NGS.columns = play_data_NGS.columns.str.replace('_x', '')\n\n# Merge NGS Play Data and NGS Player Data\nplay_player_data_NGS = play_data_NGS.merge(player_data_NGS.reset_index().set_index(\n ['Season_Year','GameKey','PlayID']), how='left', left_index=True,right_index=True)\nplay_player_data_NGS = play_player_data_NGS.reset_index().set_index(['Season_Year','GameKey','PlayID','GSISID'])\nplay_player_data_NGS = play_player_data_NGS.dropna()\n\n# Format Game Data for NGS\ngame_data_NGS = game_data.set_index(['Season_Year','GameKey'])\n\n# Merge Play, Player, Game Data NGS\nmod_play_player_data_NGS = play_player_data_NGS.reset_index().set_index(['Season_Year','GameKey'])\nplay_player_game_data_NGS = mod_play_player_data_NGS.merge(right=game_data_NGS, how='left', left_index=True, right_index=True)\nplay_player_game_data_NGS = play_player_game_data_NGS.reset_index().set_index(['Season_Year','GameKey','PlayID','GSISID'])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"77b6c267e9254ee2bac997b06aeb7835f79ee98b"},"cell_type":"code","source":"plt.figure(figsize=(12,5))\nax0 = plt.subplot2grid((1,2),(0,0))\nax1 = plt.subplot2grid((1,2),(0,1))\nax0.set_title('Frequency of Play / Down Duration')\nax1.set_title('Frequency of Player Avg Velocity')\n\nduration_by_concussion_play = play_player_game_data_NGS[play_player_game_data_NGS['Concussed'] == 1]['Play_Duration']\n\nsns.distplot(duration_by_play, ax=ax0)\nsns.distplot(duration_by_concussion_play, ax=ax0)\nax0.axvline(duration_by_play.mean(), color='blue', linestyle='dashed',linewidth=1.2)\nax0.axvline(duration_by_concussion_play.mean(), color='orange', linestyle='dashed',linewidth=1.2)\nax0.annotate(\"%.2f s\" % round(duration_by_play.mean(),2), (duration_by_play.mean() + 5, 0.06))\nax0.annotate(\"%.2f s\" % round(duration_by_concussion_play.mean(),2), (duration_by_concussion_play.mean() + 5, 0.04))\n\nax0.legend(['Mean Duration','Mean Concussion\\nDuration','All Plays','Concussion Plays'])\nax0.set_xlim((0,175))\nax0.set_xlabel(\"Time / s\")\n\nplay_velocity = play_player_game_data_NGS['Velocity']\nplay_concussed_velocity = play_player_game_data_NGS[play_player_game_data_NGS['Concussed'] == 1]['Velocity']\n\nsns.distplot(play_velocity, ax = ax1)\nsns.distplot(play_concussed_velocity, ax=ax1)\nax1.axvline(play_velocity.mean(), color='blue', linestyle='dashed',linewidth=1.2)\nax1.axvline(play_concussed_velocity.mean(), color='orange', linestyle='dashed',linewidth=1.2)\nax1.legend(['Mean Velocity','Mean Concussion\\nVelocity', 'All Plays','Concussion Plays'])\nax1.annotate(\"%.2f m/s\" % round(play_velocity.mean(),2), (play_velocity.mean() + 0.1, 2.0))\nax1.annotate(\"%.2f m/s\" % round(play_concussed_velocity.mean(),2), (play_concussed_velocity.mean()+ 0.1, 2.5))\nax1.set_xlim((0,6))\nax1.set_xlabel(\"Velocity / m/s\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a84dad76091531ac9b94c9a5e2c9c3ab0f42f5aa"},"cell_type":"markdown","source":"\n\n Velocity is claculated as the average velocity of a player over a single play. Similarly duration is calculated over a single play as the max - min time on the clock. \n
\n\n\n
\n \n
\n Plays involving concussions have a mean play duration 13 seconds greater than that of the average play duration, this is interesting and something we should keep in mind, as perhaps it gives tackling players a chance to building up more speed before impact.\n
\n
\n
\n \n
\n Interestingly on average players who tended to be concussed were travelling at lower speeds than the average player on the pitch. This is especially surprising as you would expect the oppposite. We will study this further when we look at the video replays, it is possibile that their opponent was travelling at high speed.\n
\n
\n
\n There is evidence to suggest that both duration of the play and the velocity of the player are important factors to consider in our model\n
"},{"metadata":{"_kg_hide-input":true,"trusted":false,"_uuid":"3cff40ca7797e53e18406c514da9421b342968c0"},"cell_type":"code","source":"# Select Data for model features\ndata_selected = play_player_game_data_NGS[['YardLineDist','ScoreDifference','Role','Temperature','Quarter','Velocity','Concussed','Primary_Impact_Type','Play_Duration']]\ndata_selected.to_csv('data_selected.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"a13699a63ab23fa0a10c6e8cbe07d1c97cbfd270"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}