{"cells":[{"metadata":{"_uuid":"178095eac8a49316312ec0a217e694e5e661b417"},"cell_type":"markdown","source":"Having watched (and casually played) football for most of my life, plus my love of data analysis, this competition is right up my alley (Go Bills!). My initial hunch about concussion causes is around the team formation, specific to whether the gunners are double teamed or not. Does having two, three or 4 defenders on gunners (\"jammers\") have an effect on concussion rates?\n\nAssumptions:\n - whether the play was a punt, or blocked punt, it's still an attempted punt\n - no-play punts are likely stopped by penalties (dead ball- false starts, delay of game), timeouts, or the end of the quarter/half. Because dead ball plays involve no (or little) movement, and arent really a typical punt play, they are removed from these aggregate stats\n - which season, week, game type (pre-season, regular, playoffs), stadium type (dome, outside), and weather are uncontrollable factors when considering a rule change. Turf may have an effect, which will be analyzed, but ultimately can't be part of a rule change suggestion; it's potentially an alternate suggestion for improvement. \n - Starting yardage is also a non-starter with respect to a rule change, even if it has an effect. Rules varying based on the starting yardline would be disruptive to the game, especially in the context of fake punts working like any other regular 4th down play.\n Therefore, these items (except grass vs turf) will be ignored in this analysis.\n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"\"\"\"\nCreated on Sat Jan  5 13:40:42 2019\n\n@author: Andrew Welsh\n\nSee my powerpoint presentation for a full explanation of the analsys\nThis notebook is more my 'scratchpad' to show my work than the final presentation\n\"\"\"\nimport pandas as pd\nimport numpy as np\nimport tqdm\nimport gc\nimport scipy.stats as scs\nimport matplotlib.pyplot as plt\n\npath ='../input/NFL-Punt-Analytics-Competition/'\nprint('Load data, read CSVs')\n#game level data - stadium, teams, week, location info\nprint('Read Game data')\ngamedata = pd.read_csv(path + 'game_data.csv', encoding='ISO-8859-1')\nstadium_type = gamedata[['GameKey','StadiumType','Turf']]\n\nprint('Read video review concussion data')\n#specific players and play IDs when concussions happened on punts\n#I copied the video_review.csv in order to clean the one instance of \"Unclear\" in the GSISID column\n#for this analysis, nulls and values of 'Unclear' are essentially the same, so to keep the datatype \n#clean, I replaced the 'Unclear' value with a null in the dataset\npath2 = '../input/video-review-cleaned/'\nconcussion_events = pd.read_csv(path2 + 'video_review.csv', encoding='ISO-8859-1')\n# add flag column for indicator of concussions\nconcussion_events['conc_flag'] = 1\nconcussion_events.drop(['Season_Year'], axis=1, inplace=True)\n#impute dummy value of 1 for GSISID for the nulls\nconcussion_events = concussion_events.fillna({'Primary_Partner_GSISID':1})\n#force data type of GSISID for later merges\nconcussion_events['Primary_Partner_GSISID'].astype('float32', inplace=True)\n\nprint('Read Player Role data')\n#the role/position the players were playing on punt plays; independent of their regular roles\npunt_role = pd.read_csv(path + 'play_player_role_data.csv', encoding='ISO-8859-1')\npunt_role.drop(['Season_Year'], axis=1, inplace=True)\n\nprint('Read all punt play data')\n#all punt plays\nall_plays = pd.read_csv(path + 'play_information.csv', encoding='ISO-8859-1')\nall_plays.drop(['Season_Year'], axis=1, inplace=True)\n#find plays that were 'no play', i.e. dead ball penalites, timeouts, or end of quarter\nall_plays['no_play'] = all_plays['PlayDescription'].str.contains('No Play', regex=True)\n#find touchdowns, touchbacks\nall_plays['touchdown'] = all_plays['PlayDescription'].str.contains('TOUCHDOWN', regex=True)\nall_plays['blocked'] = all_plays['PlayDescription'].str.contains('BLOCKED', regex=True)\nall_plays['touchback'] = all_plays['PlayDescription'].str.contains('Touchback', regex=True)\nall_plays['penalty'] = all_plays['PlayDescription'].str.contains('PENALTY', regex=True)\nall_plays['fair_catch'] = all_plays['PlayDescription'].str.contains('fair catch', regex=True)\n\n#list of removed plays, so they can be removed from NGS data\nremoved_plays0 = pd.DataFrame(all_plays[all_plays['no_play'] == True])\nremoved_plays = removed_plays0[['GameKey','PlayID','no_play']]\n#remove no-plays from all punt play data\nall_plays = all_plays[all_plays['no_play'] == False]\nprint('Data cleaned, read NGS files')\n\n\n#%%\n#all NGS data, will be concatenated together.\n\n#force datatypes for NGS files, mostly float32 to save memory\ndtypes = {'Season_Year': 'int16',\n         'GameKey': 'int16',\n         'PlayID': 'int16',\n         'GSISID': 'float32',\n         'Time': 'str',\n         'x': 'float32',\n         'y': 'float32',\n         'dis': 'float32',\n         'o': 'float32',\n         'dir': 'float32',\n         'Event': 'str'}\n\ncol_names = list(dtypes.keys())\n\nngslist = ['NGS-2016-pre.csv',\n           'NGS-2016-reg-wk1-6.csv',\n           'NGS-2016-reg-wk7-12.csv',\n           'NGS-2016-reg-wk13-17.csv',\n           'NGS-2016-post.csv',\n           'NGS-2017-pre.csv',\n           'NGS-2017-reg-wk1-6.csv',\n           'NGS-2017-reg-wk7-12.csv',\n           'NGS-2017-reg-wk13-17.csv',\n           'NGS-2017-post.csv']\n\nprint('Read NGS data')\n#initialize df list object\nngsdf_list = []\n\nfor i in tqdm.tqdm(ngslist):\n    df = pd.read_csv(path+i, usecols=col_names, dtype=dtypes)\n    \n    ngsdf_list.append(df)\n\n#concatenate the NGS CSVs into one file\nngs0 = pd.concat(ngsdf_list)\nprint('NGS data ready')\n#clean up memory usage\ndel ngsdf_list\ndel df\ngc.collect()\n\nngs0['Time'] = pd.to_datetime(ngs0['Time'], format='%Y-%m-%d %H:%M:%S')\n\n#converting yards/sec to mph is 3600 / 1760 = 2.04545... \n#and since each row in NGS data is 0.1s, multiply 2.04545 by 10 = 20.4545\nngs0['mph'] = ngs0['dis']*20.4545\n    \n#add removed_play indicator column to NGS dataset\nngs = pd.merge(ngs0, removed_plays, how='left', on=['GameKey','PlayID'], copy=True)\n\n#clean up memory usage\ndel ngs0\ngc.collect()\n\nprint('Remove non-plays from NGS data')\n#remove no-play plays from NGS before summary stats are calculated\nngs = ngs[ngs['no_play'] != False]\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cb37ecf1c3ecc3bde6b4970ba8a069929f1d6821"},"cell_type":"code","source":"#gather max and mean speed for player during concussion events\nprint('Merge NGS data with concussion event data')\nngs_injuries_player = pd.merge(concussion_events, ngs, how='left', on=['GameKey','PlayID','GSISID'])\nngs_injury_speed_player = ngs_injuries_player.groupby(['GameKey','PlayID','GSISID'], as_index = False)['mph'].agg(\n                {'max_mph_player': max,\n                 'avg_mph_player': np.mean})\n\n#reusing temp1 here\ntemp1 = pd.merge(ngs_injury_speed_player, concussion_events, how='inner', on=['GameKey','PlayID','GSISID'], copy=True)\ntemp1 = pd.merge(temp1, punt_role, how='left', on=['GameKey','PlayID','GSISID'], copy=True)\n\n\n#rename the Primary_Partner_GSISID column to GSISID to make join work properly\n#for whatever reason, joining on the mismatched-names didn't work.. so in the interest of it \n#functioning, I went with this clumsy hack approach\nconcussion_events.columns=['GameKey',\n                           'PlayID',\n                           'GSISID_player',\n                           'Player_Activity_Derived',\n                           'Turnover_Related',\n                           'Primary_Impact_Type',\n                           'GSISID',\n                           'Primary_Partner_Activity_Derived',\n                           'Friendly_Fire',\n                           'conc_flag']\n\nngs_injuries_partner = pd.merge(concussion_events, ngs, how='left', on=['GameKey','PlayID','GSISID'], copy=True)\n\nngs_injury_speed_partner = ngs_injuries_partner.groupby(['GameKey','PlayID','GSISID'], as_index = False)['mph'].agg(\n                {'max_mph_partner': max,\n                 'avg_mph_partner': np.mean})\n\ntemp2 = pd.merge(ngs_injury_speed_partner, concussion_events, how='inner', on=['GameKey','PlayID','GSISID'])\n#get partner positional data\ntemp2 = pd.merge(temp2, punt_role, how='left', on=['GameKey','PlayID','GSISID'], copy=True)\n\ntemp3 = temp2[['GameKey','PlayID','max_mph_partner','avg_mph_partner','Role']]\ntemp3.columns = ['GameKey','PlayID','max_mph_partner','avg_mph_partner','partner_role']\n\n#return column names to original \nconcussion_events.columns=['GameKey',\n                           'PlayID',\n                           'GSISID',\n                           'Player_Activity_Derived',\n                           'Turnover_Related',\n                           'Primary_Impact_Type',\n                           'Primary_Partner_GSISID',\n                           'Primary_Partner_Activity_Derived',\n                           'Friendly_Fire',\n                           'conc_flag']\n\nconcussion_events = pd.merge(temp1, temp3, on=['GameKey','PlayID'], copy=True)\nconcussion_events = concussion_events.fillna({'partner_role':'NA'})\n\n\n#get average, average of the max, and max speeds by punt play role\nprint('Get average velocities of all punt plays by position')\n#need to aggregate the max and avg MPH by player first\npunt_role_ngs0 = ngs.groupby(['GameKey','PlayID','GSISID'], as_index = False)['mph'].agg(\n                {'max_mph': max,\n                 'avg_mph': np.mean})\n\n#create temp df to calculate global stats by role\npunt_role_ngs0 = pd.merge(punt_role_ngs0, punt_role, how='left', on=['GameKey','PlayID','GSISID'])\n\n#take average of max, max of max, and overall average by role\npunt_role_ngs_max = punt_role_ngs0.groupby(['Role'], as_index = False)['max_mph'].agg(\n                {'max_mph_max': max,\n                 'max_mph_avg': np.mean})\n\npunt_role_ngs_avg = punt_role_ngs0.groupby(['Role'], as_index = False)['avg_mph'].agg(\n                {'count','mean'}).reset_index()\n\n#clean up temp df\ndel punt_role_ngs0\ngc.collect()\n\n#summary statistics, by role, for all plays in dataset\npunt_role_ngs = pd.merge(punt_role_ngs_max, punt_role_ngs_avg, how='outer', on=['Role'])\n\n#merge concussion events data to role data\n#combine player roles on punt plays with overall punt play data, reusing temp1\ntemp1 = pd.merge(punt_role, all_plays, how='left', on=['GameKey','PlayID'])\n\n#combine stadium type and turf type data to dataset, to later test whether these are correlated to concussions\ntemp2 = pd.merge(temp1, stadium_type, how='left', on=['GameKey'])\n\n#join concussion event data to the dataset, knowing that for this specific dataset, there are no instances of \n#concussions occurring to 2 players on the same play\npunts = pd.merge(temp2, concussion_events, how='left', on=['GameKey','PlayID','GSISID'])\n\npunts = punts.fillna({'conc_flag':0})\n\n#punts['Turf'].value_counts() shows the data is dirty\n#Grass                    51013\n#Natural Grass            27124\n#Field Turf               13253\n#Artificial               12927\n#FieldTurf                12149\n#UBU Speed Series-S5-M     9079\n#DD GrassMaster            4418\n#A-Turf Titan              4283\n#UBU Sports Speed S5-M     3717\n#Natural grass             2436\n#FieldTurf 360             1406\n#Artifical                  812\n#Natural                    790\n#grass                      593\n#Natrual Grass              462\n#Natural Grass              440\n#FieldTurf360               396\n#UBU Speed Series S5-M      396\n#Field turf                 264\n#Synthetic                  263\n#Naturall Grass             154\n# so we're going to recode it\n\n#decare grass_ind column, to create indicator of natural grass vs not\npunts['grass_ind']=punts['Turf']\n\n#define \"grass_recode\" dictionary based on value_counts from raw data\ngrass_recode = {'Grass':1,\n                'Natural Grass':1,\n                'Field Turf':0,\n                'Artificial':0,\n                'FieldTurf':0,\n                'UBU Speed Series-S5-M':0,\n                'DD GrassMaster':0,\n                'A-Turf Titan':0,\n                'UBU Sports Speed S5-M':0,\n                'Natural grass':1,\n                'FieldTurf 360':0,\n                'Artifical':0,\n                'Natural':1,\n                'grass':1,\n                'Natrual Grass':1,\n                'Natural Grass ':1,\n                'Natural Grass':1,\n                'FieldTurf360':0,\n                'UBU Speed Series S5-M':0,\n                'Field turf':0,\n                'Synthetic':0,\n                'Naturall Grass':1\n               }\n#replace values in grade recode column with values defined in the dictionary array above\npunts = punts.replace(dict(grass_ind=grass_recode))\npunts['grass_ind'].astype('float16')\n\nprint('Test whether natural grass vs turf is related to concussion rate')\ntemp1 = punts.groupby(['grass_ind'])['conc_flag'].agg({'count','sum'}).reset_index()\ntemp1['ratio']= temp1['sum']/temp1['count']\nprint('Concussions by Grass (or not): \\n',temp1)\n#no difference based on turf type. There were two helmet-to-ground concussions in the dataset, too few to draw conclusions\n\n\nprint('Prepare formation data')\n#get counts of players in various positions\nrole_count = pd.pivot_table(punt_role, index=['GameKey', 'PlayID'], columns=['Role'], \n                       aggfunc=lambda x: len(x.unique()))['GSISID'].fillna(0)\n\nrole_count.reset_index(inplace=True)\n#sum of all positional columns, to get count of players encoded\nrole_count['num_players'] = role_count.iloc[:,2:53].sum(axis=1)\n#leave only typical extra-man and one-man-short scenarios, for both teams (20 to 24 players)\nrole_count1 = role_count.loc[(role_count['num_players']>=20) & (role_count['num_players']<=24)]\n\n#knowing there weren't any plays with 2 concussions, copy of just gamekey and playID will be unique identifier\nconcussion_flag = concussion_events[['GameKey', 'PlayID','conc_flag']]\nall_plays_subset = all_plays[['GameKey', 'PlayID','touchdown','blocked','touchback','penalty','fair_catch']]\n\nrole_count1 = pd.merge(role_count1, concussion_flag, how='left', on=['GameKey', 'PlayID'])\nrole_count1 = role_count1.fillna({'conc_flag':0})\nrole_count1['outer_sum'] = (\n        role_count1['VL'] + role_count1['VLi'] + role_count1['VLo'] +\n        role_count1['VR'] + role_count1['VRi'] + role_count1['VRo']\n        )\nrole_count1['outer_sum_grp'] = pd.cut(role_count1['outer_sum'], bins=[-1,2,3,6], labels=['0 to 2','3','4 or more'])\n\nrole_count1['interior_side_overload'] = abs(\n        (role_count1['PDL1'] + role_count1['PDL2'] + role_count1['PDL3'] + \n         role_count1['PDL4'] + role_count1['PDL5'] + role_count1['PDL6'] + \n         role_count1['PLL'])\n        -\n        (role_count1['PDR1'] + role_count1['PDR2'] + role_count1['PDR3'] + \n         role_count1['PDR4'] + role_count1['PDR5'] + role_count1['PDR6'] + \n         role_count1['PLR'])\n        )\nrole_count1['interior_side_overload_grp'] = pd.cut(role_count1['interior_side_overload'], bins=[-1,1,10], labels=['0 or 1','2 or more'])\n\nrole_count1['middle_sum'] = role_count1['PLR'] + role_count1['PLM'] + role_count1['PLL'] + role_count1['PFB']\nrole_count1['middle_sum_grp'] = pd.cut(role_count1['middle_sum'], bins=[-1,1,10], labels=['0 or 1','2 or more'])\n\n#offense interior are players protecting the punter; hypothesis is punting teams expecting only \n#return team to focus on return play, and deliver only a token punt block attempt, will have \n#fewer interior players to protect the punter (PC, PPR, PLW, PRW)\nrole_count1['offense_interior'] = role_count1['PC'] + role_count1['PPR'] + role_count1['PLW'] + role_count1['PRW']\nrole_count1['offense_interior_grp'] = pd.cut(role_count1['offense_interior'], bins=[-1,2,10], labels=['0 to 2','3 or more'])\n\nrole_count1 = pd.merge(role_count1,all_plays_subset, how='inner', on=['GameKey','PlayID'])\n\n#\n#\n#data cleaning and prep complete\n#\n#","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"46260d95cb6ffd0425b1865da65cababf0661232"},"cell_type":"code","source":"#\n#\n#\n# count number of jammers. Typical formations are 2, 3 or 4 defenders on gunners\n#\n#\n#\nconc_by_role1 = pd.pivot_table(role_count1, index=['outer_sum_grp'], columns=['conc_flag'],\n                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\n#run this next line, uncommented, if you want all possible values:\n#conc_by_role1 = pd.pivot_table(role_count1, index=['outer_sum'], columns=['conc_flag'],\n#                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\nconc_by_role1.reset_index(inplace=True)\nconc_by_role1.columns = ['outer_sum','no_conc','conc']\nconc_by_role1['total'] = conc_by_role1['no_conc'] + conc_by_role1['conc']\nconc_by_role1['rate'] = conc_by_role1['conc'] / conc_by_role1['total']\n\n#chi-square test of difference in concussion counts by # of jammers\nchi2, p, dof, expected = scs.chi2_contingency(conc_by_role1[['no_conc','conc']])\nprint('p-value, concussion vs non-concussions by # of jammers')\nprint(p)\n\n#\n#\n# count number of linebacker/fullback defenders. Possibilities are 0 through 4.\n#\n#\n#\n\nconc_by_role2 = pd.pivot_table(role_count1, index=['middle_sum_grp'], columns=['conc_flag'],\n                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\n#run this if you want all possible values\n#conc_by_role2 = pd.pivot_table(role_count1, index=['middle_sum'], columns=['conc_flag'],\n#                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\nconc_by_role2.reset_index(inplace=True)\nconc_by_role2.columns = ['inner_sum','no_conc','conc']\nconc_by_role2['total'] = conc_by_role2['no_conc'] + conc_by_role2['conc']\nconc_by_role2['rate'] = conc_by_role2['conc'] / conc_by_role2['total']\n\nchi2, p, dof, expected = scs.chi2_contingency(conc_by_role2[['no_conc','conc']])\nprint('p-value, concussion vs non-concussions by # of midfield defenders')\nprint(p)\n\n#\n#\n#\n# test if interior left or right side overload (0, 1, 2, etc) is related to concussion rate\n#\n#\n#\n\nconc_by_role3 = pd.pivot_table(role_count1, index=['interior_side_overload_grp'], columns=['conc_flag'],\n                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\n#run this if you want all possible values\n#conc_by_role3 = pd.pivot_table(role_count1, index=['interior_side_overload'], columns=['conc_flag'],\n#                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\nconc_by_role3.reset_index(inplace=True)\nconc_by_role3.columns = ['overload_sum','no_conc','conc']\nconc_by_role3['total'] = conc_by_role3['no_conc'] + conc_by_role3['conc']\nconc_by_role3['rate'] = conc_by_role3['conc'] / conc_by_role3['total']\n\nchi2, p, dof, expected = scs.chi2_contingency(conc_by_role3[['no_conc','conc']])\nprint('p-value, concussion vs non-concussions by # of L or R overloaded defenders')\nprint(p)\n\n#\n#\n# test if number of offensive interior players  is related to concussion rate. \n# Value possibilities ore 0 through 4.\n#\n#\n\nconc_by_role4 = pd.pivot_table(role_count1, index=['offense_interior_grp'], columns=['conc_flag'],\n                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\n#run this if you want all possible values\n#conc_by_role4 = pd.pivot_table(role_count1, index=['offense_interior'], columns=['conc_flag'],\n#                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\nconc_by_role4.reset_index(inplace=True)\nconc_by_role4.columns = ['offense_interior_sum','no_conc','conc']\nconc_by_role4['total'] = conc_by_role4['no_conc'] + conc_by_role4['conc']\nconc_by_role4['rate'] = conc_by_role4['conc'] / conc_by_role4['total']\n\nchi2, p, dof, expected = scs.chi2_contingency(conc_by_role4[['no_conc','conc']])\nprint('p-value, concussion vs non-concussions by # of interior offensive players')\nprint(p)\n\n#plot rates from concussions by role groupings\ny_pos = np.arange(len(conc_by_role1['outer_sum']))\nplt.ylabel('Concussion Rate (per 1000 plays)')\nplt.xlabel('Number of jammers')\nplt.bar(y_pos, conc_by_role1['rate']*1000, tick_label=conc_by_role1['outer_sum'], \n        color=['#1f77b4','r','crimson'])\n\n# number of jammers is strongly associated with concussion rates, confirming my initial hunch\n# however, the results are opposite what I had initially expected. I thought double-teamed gunners\n# would've resulted in fewer concussions, as the gunners were, what I thought anecdotally,the \n# players most likely to crash into someone at high speed.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"958ba1b196755367119818bd070473e384584d4d"},"cell_type":"code","source":"# Now we look at the potential impact a rule change would have on other aspects of the game \n# around punt plays.\n\n# The above analysis shows that the number of concussions are statistically significantly different\n# based on the number of jammers. The difference in concussion rates based on the number of \n# L or R side overload by the interior defenders was almost significant, so for funsies we'll \n# include that here too\n\n# After submitting, I realized I probably should've written a function here for this. I did this\n# analysis iteratively, first only looking at the outer sum, and would've gained nothing at that \n# time writing a function. Lesson learned!\n#\n#\n# jammers and touchdown rates\n#\n#\n\nprint('See how number of jammers ')\nouter_sum_td_rate = pd.pivot_table(role_count1, index=['outer_sum_grp'], columns=['touchdown'],\n                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\nouter_sum_td_rate.reset_index(inplace=True)\nouter_sum_td_rate.columns = ['outer_sum_grp','no_td','td']\nouter_sum_td_rate['total'] = outer_sum_td_rate['no_td'] + outer_sum_td_rate['td']\nouter_sum_td_rate['rate'] = outer_sum_td_rate['td'] / outer_sum_td_rate['total']\n\nchi2, p, dof, expected = scs.chi2_contingency(outer_sum_td_rate[['no_td','td']])\nprint('p-value, # of jammers and touchdown rates')\nprint(p)\n\n#\n#\n# jammers and touchback rates\n#\n#\n\nouter_sum_tb_rate = pd.pivot_table(role_count1, index=['outer_sum_grp'], columns=['touchback'],\n                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\nouter_sum_tb_rate.reset_index(inplace=True)\nouter_sum_tb_rate.columns = ['outer_sum_grp','no_tb','tb']\nouter_sum_tb_rate['total'] = outer_sum_tb_rate['no_tb'] + outer_sum_tb_rate['tb']\nouter_sum_tb_rate['rate'] = outer_sum_tb_rate['tb'] / outer_sum_tb_rate['total']\n\nchi2, p, dof, expected = scs.chi2_contingency(outer_sum_tb_rate[['no_tb','tb']])\nprint('p-value, # of jammers and touchback rates')\nprint(p)\n#touchback rates are higher when there are only 2 jammers\n#may be related to teams not attempting returns when punts are within reach of a touchback\n#analysis of average/median punt team yard line would probably show 2-gunner formations to be \n#across the 50 more often\n#\n#\n# jammers and punt block rates\n#\n#\n\nouter_sum_pb_rate = pd.pivot_table(role_count1, index=['outer_sum_grp'], columns=['blocked'],\n                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\nouter_sum_pb_rate.reset_index(inplace=True)\nouter_sum_pb_rate.columns = ['outer_sum_grp','no_pb','pb']\nouter_sum_pb_rate['total'] = outer_sum_pb_rate['no_pb'] + outer_sum_pb_rate['pb']\nouter_sum_pb_rate['rate'] = outer_sum_pb_rate['pb'] / outer_sum_pb_rate['total']\n\nchi2, p, dof, expected = scs.chi2_contingency(outer_sum_pb_rate[['no_pb','pb']])\nprint('p-value, # of jammers and blocked punt rates')\nprint(p)\n#punt block rates are lower when there are 4 jammers, which follows common sense \n#(fewer rushing, intention of punt defenders is to maximize return yards)\n\n\n#\n#\n# jammers and penalty rates (regardless of on who, type, or whether accepted or declined)\n#\n#\n\nouter_sum_plty_rate = pd.pivot_table(role_count1, index=['outer_sum_grp'], columns=['penalty'],\n                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\nouter_sum_plty_rate.reset_index(inplace=True)\nouter_sum_plty_rate.columns = ['outer_sum_grp','no_plty','plty']\nouter_sum_plty_rate['total'] = outer_sum_plty_rate['no_plty'] + outer_sum_plty_rate['plty']\nouter_sum_plty_rate['rate'] = outer_sum_plty_rate['plty'] / outer_sum_plty_rate['total']\n\nchi2, p, dof, expected = scs.chi2_contingency(outer_sum_plty_rate[['no_plty','plty']])\nprint('p-value, # of jammers and penalty rates')\nprint(p)\n#penalties are significantly less frequent in 0-2 jammer formation\n\n#\n#\n# jammers and fair catch rates \n#\n#\n\nouter_sum_fc_rate = pd.pivot_table(role_count1, index=['outer_sum_grp'], columns=['fair_catch'],\n                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\nouter_sum_fc_rate.reset_index(inplace=True)\nouter_sum_fc_rate.columns = ['outer_sum_grp','no_fc','fc']\nouter_sum_fc_rate['total'] = outer_sum_fc_rate['no_fc'] + outer_sum_fc_rate['fc']\nouter_sum_fc_rate['rate'] = outer_sum_fc_rate['fc'] / outer_sum_fc_rate['total']\n\nchi2, p, dof, expected = scs.chi2_contingency(outer_sum_fc_rate[['no_fc','fc']])\nprint('p-value, # of jammers and penalty rates')\nprint(p)\n#fair catches are significantly more frequent in 0-2 jammer formation\n\n#\n#\n# Interior side overload defenders and touchdown rates\n#\n#\n\ninterior_side_overload_td_rate = pd.pivot_table(role_count1, index=['interior_side_overload_grp'], columns=['touchdown'],\n                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\ninterior_side_overload_td_rate.reset_index(inplace=True)\ninterior_side_overload_td_rate.columns = ['inner_sum','no_conc','conc']\ninterior_side_overload_td_rate['total'] = interior_side_overload_td_rate['no_conc'] + interior_side_overload_td_rate['conc']\ninterior_side_overload_td_rate['rate'] = interior_side_overload_td_rate['conc'] / interior_side_overload_td_rate['total']\n\nchi2, p, dof, expected = scs.chi2_contingency(interior_side_overload_td_rate[['no_conc','conc']])\nprint('p-value, # of Interior side overload defenders and touchdown rates')\nprint(p)\n\n#\n#\n# Interior side overload defenders and touchback rates\n#\n#\n\ninterior_side_overload_tb_rate = pd.pivot_table(role_count1, index=['interior_side_overload_grp'], columns=['touchback'],\n                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\ninterior_side_overload_tb_rate.reset_index(inplace=True)\ninterior_side_overload_tb_rate.columns = ['interior_side_overload_grp','no_conc','conc']\ninterior_side_overload_tb_rate['total'] = interior_side_overload_tb_rate['no_conc'] + interior_side_overload_tb_rate['conc']\ninterior_side_overload_tb_rate['rate'] = interior_side_overload_tb_rate['conc'] / interior_side_overload_tb_rate['total']\n\nchi2, p, dof, expected = scs.chi2_contingency(interior_side_overload_tb_rate[['no_conc','conc']])\nprint('p-value, # of Interior side overload defenders and touchback rates')\nprint(p)\n#\n#\n# Interior side overload defenders and punt block rates\n#\n#\n\ninterior_side_overload_pb_rate = pd.pivot_table(role_count1, index=['interior_side_overload_grp'], columns=['blocked'],\n                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\ninterior_side_overload_pb_rate.reset_index(inplace=True)\ninterior_side_overload_pb_rate.columns = ['interior_side_overload_grp','no_conc','conc']\ninterior_side_overload_pb_rate['total'] = interior_side_overload_pb_rate['no_conc'] + interior_side_overload_pb_rate['conc']\ninterior_side_overload_pb_rate['rate'] = interior_side_overload_pb_rate['conc'] / interior_side_overload_pb_rate['total']\n\nchi2, p, dof, expected = scs.chi2_contingency(interior_side_overload_pb_rate[['no_conc','conc']])\nprint('p-value, # of Interior side overload defenders and blocked punt rates')\nprint(p)\n\n#\n#\n# Interior side overload defenders and penalty rates (regardless of on who, type, or whether accepted or declined)\n#\n#\n\ninterior_side_overload_plty_rate = pd.pivot_table(role_count1, index=['interior_side_overload_grp'], columns=['penalty'],\n                              aggfunc=lambda x: len(x))['PlayID'].fillna(0)\ninterior_side_overload_plty_rate.reset_index(inplace=True)\ninterior_side_overload_plty_rate.columns = ['interior_side_overload_grp','no_conc','conc']\ninterior_side_overload_plty_rate['total'] = interior_side_overload_plty_rate['no_conc'] + interior_side_overload_plty_rate['conc']\ninterior_side_overload_plty_rate['rate'] = interior_side_overload_plty_rate['conc'] / interior_side_overload_plty_rate['total']\n\nchi2, p, dof, expected = scs.chi2_contingency(interior_side_overload_plty_rate[['no_conc','conc']])\nprint('p-value, # of Interior side overload defenders and penalty rates')\nprint(p)\n#none of the results from interior side overload were statistically significant.\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"78c71fdd5d6847ac6ff4fd8053786961ae5cfdf3"},"cell_type":"code","source":"#print contingency tables / crosstabs for visual analysis and inclusion in presentation\nprint('\\n Rate of touchdowns by number of jammers')\nprint(outer_sum_td_rate)\nprint('\\n Rate of touchback by number of jammers')\nprint(outer_sum_tb_rate)\nprint('\\n Rate of penalties by number of jammers')\nprint(outer_sum_plty_rate)\nprint('\\n Rate of fair catches by number of jammers')\nprint(outer_sum_fc_rate)\n\n#get global velocity data by role combined with concussion event velocities by role, to see if there's \n#insights in this part of the dataset, related to the formation\n\n#mean velocities for the concussed player\ntemp1 = concussion_events.groupby(['Role'])['max_mph_player','avg_mph_player'].agg({'mean'}).reset_index()\ntemp1.columns=['Role','max_mph_player_avg','avg_mph_player_avg']\n\n#mean velocities for the partner player, when applicable\ntemp2 = concussion_events.groupby(['partner_role'])['max_mph_partner','avg_mph_partner'].agg({'mean'}).reset_index()\ntemp2.columns=['Role','max_mph_partner_avg','avg_mph_partner_avg']\n\ntemp3 = pd.merge(temp1, temp2, how='outer', on='Role')\n\n#add counts of concussions by player + partner role\ntemp4 = concussion_events.groupby(['Role'])['conc_flag'].agg({'sum'}).reset_index()\ntemp4.columns=['Role','conc_count_player']\n\ntemp5 = concussion_events.groupby(['partner_role'])['conc_flag'].agg({'sum'}).reset_index()\ntemp5.columns=['Role','conc_count_partner']\n\ntemp6 = pd.merge(temp4, temp5, how='outer', on='Role')\ntemp7 = pd.merge(temp3, temp6, how='outer', on='Role')\n\n#merge with velocity data for all punt plays\nconc_speed_compare = pd.merge(punt_role_ngs, temp7, how='outer', on='Role')\n\n#calculate differences. Using overall max speed, which would represent the worst possible collision velocity \n#(even if the concussions didn't happen at the max recorded velocity)\nconc_speed_compare['max_speed_diff_player'] = conc_speed_compare['max_mph_avg'] - conc_speed_compare['max_mph_player_avg']\nconc_speed_compare['max_speed_diff_partner'] = conc_speed_compare['max_mph_avg'] - conc_speed_compare['max_mph_partner_avg']\n\nprint('statistics on the deviation of concussed player velocity compared to global average maximum speed, in MPH')\nprint('max', conc_speed_compare['max_speed_diff_player'].max())\nprint('median', conc_speed_compare['max_speed_diff_player'].median())\nprint('mean', conc_speed_compare['max_speed_diff_player'].mean())\n\nprint('statistics on the deviation of partner player velocity compared to global average maximum speed, in MPH')\nprint('max', conc_speed_compare['max_speed_diff_partner'].max())\nprint('median', conc_speed_compare['max_speed_diff_partner'].median())\nprint('mean', conc_speed_compare['max_speed_diff_partner'].mean())\n\n#this result shows that the concussed players weren't going as fast as players in general\n#further in-depth analysis of player velocities, while interesting from a data science perspective,\n#are of limited probative value, since the formation is shown to have such a strong association\n#with concussion rates, and formations are readily easy to modify via rules. The player behaviors\n#would follow naturally as a result.\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6182c3eaeeabfb8030b56d51d12a86074f2e218e"},"cell_type":"markdown","source":"# Conclusion:\n## The number of jammers in the formation is most strongly related to concussion rates. \n\n## The recommended rule change will be to restrict the number of jammers to a maximum of two. \n## This can be enforced by restricting the number of players within the numbers, or by specifying single coverage.\n\nAn assumption here is that the behavior on punt plays with 0-2 jammers would remain the same if all punt plays were restricted to this formation. Following the KISS principle, the simple simulation of the effect on the game is to simply look at the rates when there are 0-2 jammers. Coaches and players may find creative ways to circumvent the existing rules to gain an edge, but these are difficult/impossible to simulate, and a non-starter, really."},{"metadata":{"trusted":true,"_uuid":"465d19990d65bbf4e2b293431d1d6703793f97e9"},"cell_type":"code","source":"#plot rates from touchdowns by role groupings\ny_pos = np.arange(len(outer_sum_td_rate['outer_sum_grp']))\nplt.ylabel('Touchdowns per 1000 plays')\nplt.xlabel('Number of jammers')\nplt.bar(y_pos, outer_sum_td_rate['rate']*1000, tick_label=outer_sum_td_rate['outer_sum_grp'], \n        color=['g','#1f77b4','#1f77b4'])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e9e45edccdf3e4e7d3a1809b45c836c0e1138966"},"cell_type":"code","source":"#plot rates from touchbacks by role groupings\ny_pos = np.arange(len(outer_sum_td_rate['outer_sum_grp']))\nplt.ylabel('Touchbacks per 1000 plays')\nplt.xlabel('Number of jammers')\nplt.bar(y_pos, outer_sum_tb_rate['rate']*1000, tick_label=outer_sum_tb_rate['outer_sum_grp'], \n        color=['g','#1f77b4','#1f77b4'])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"03da39a443014db9173ca8c5c1c4ed1b6eefd530"},"cell_type":"code","source":"#plot rates from penalties by role groupings\ny_pos = np.arange(len(outer_sum_td_rate['outer_sum_grp']))\nplt.ylabel('Penalties per 1000 plays')\nplt.xlabel('Number of jammers')\nplt.bar(y_pos, outer_sum_plty_rate['rate']*1000, tick_label=outer_sum_plty_rate['outer_sum_grp'], \n        color=['g','#1f77b4','#1f77b4'])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8e7ecf96d4dad048f86d9fb842af74276d4c3077"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}