{"cells":[{"metadata":{"_uuid":"d9f401c78f61fc40aab3b5e2d7ec91bb91e3aa15"},"cell_type":"markdown","source":"# NFL Punt Analytics Competition\n### Andrew Rall"},{"metadata":{"trusted":true,"_uuid":"58eeec7acd91599640209ba9461547270d1a3fdb"},"cell_type":"code","source":"import pandas as pd\nimport re\nimport numpy as np\nimport collections\nimport seaborn as sns\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"38de400c1c38a070e07f78342214155fc43ba0ed"},"cell_type":"markdown","source":"## Introduction and Motivation\n\nPunt returns are often perceived as one of the most dangerous yet also exciting plays in football. First, I explore the data and verify that the first part of this claim has validity. From this exploration, I attempt to come up with a minimally invasive yet highly effective rule change to reduce the occurrence of concussions on punt plays.","attachments":{}},{"metadata":{"_uuid":"634d0e54439711309a3876b6f06bc3f9a24465b6"},"cell_type":"markdown","source":"## Loading the Data"},{"metadata":{"_uuid":"7d3d8624a75d8a409bc17d26d5d65caab5571b68"},"cell_type":"markdown","source":"#### Loading the Next Gen Stats for Punts on Which a Concussion was Sustained"},{"metadata":{"trusted":true,"_uuid":"9f24b9949979f54719ecc1b13f424b7c6bda18f7"},"cell_type":"code","source":"%%capture\nngs_2016_pre_conc = pd.read_csv('../input/punt-data/con_ngs_2016_pre.csv')\nngs_2016_reg_wk1_6 = pd.read_csv('../input/punt-data/con_ngs_2016_wk1_6.csv')\nngs_2016_reg_wk7_12 = pd.read_csv('../input/punt-data/con_ngs_2016_wk7_12.csv')\nngs_2016_reg_wk13_17 = pd.read_csv('../input/punt-data/con_ngs_2016_wk13_17.csv')\nngs_2017_pre_conc = pd.read_csv('../input/punt-data/con_ngs_2017_pre.csv')\nngs_2017_reg_wk1_6 = pd.read_csv('../input/punt-data/con_ngs_2017_wk1_6.csv')\nngs_2017_reg_wk7_12 = pd.read_csv('../input/punt-data/con_ngs_2017_wk7_12.csv')\nngs_2017_reg_wk13_17 = pd.read_csv('../input/punt-data/con_ngs_2017_wk13_17.csv')\n\nngs_list = [ngs_2016_pre_conc, ngs_2016_reg_wk1_6, ngs_2016_reg_wk7_12, ngs_2016_reg_wk13_17, ngs_2017_pre_conc, ngs_2017_reg_wk1_6, ngs_2017_reg_wk7_12, ngs_2017_reg_wk13_17]\nngs_conc = pd.concat(ngs_list)\nngs_conc['unique_id'] = [i + j + k for i, j, k in zip([str(x) for x in ngs_conc.GameKey], [str(x) for x in ngs_conc.PlayID], [str(x) for x in ngs_conc.Season_Year])]\nngs_conc = ngs_conc.drop(['Unnamed: 0', 'Unnamed: 0.1'], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3758d3aa2c64e0546e3b27b31c8e9408bfa2181c"},"cell_type":"code","source":"ngs_conc.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e7482312f535883cf6e5e076cbb7818a218d009a"},"cell_type":"markdown","source":"#### Loading and Combining the Play Information and Injury Data"},{"metadata":{"trusted":true,"_uuid":"993eb9b8cee274140e5324813fae9191eeae97e4"},"cell_type":"code","source":"video_review = pd.read_csv('../input/NFL-Punt-Analytics-Competition/video_review.csv')\nvideo_footage_control = pd.read_csv('../input/NFL-Punt-Analytics-Competition/video_footage-control.csv')\nvideo_footage_injury = pd.read_csv('../input/NFL-Punt-Analytics-Competition/video_footage-injury.csv')\npunt_data = pd.read_csv('../input/NFL-Punt-Analytics-Competition/play_information.csv')\n\nregexp_no_punt = re.compile('(No)\\s(Play)|(Delay)\\s(of)\\s(Game)|(Aborted)|(pass)|(False)\\s(Start)')\nno_punt = [regexp_no_punt.search(x) == None for x in punt_data.PlayDescription]\n\n### There was a penalty on the play so the punt didn't count but there was still a concussion\nno_punt[4018] = True\npunt_data = punt_data[no_punt]\n\nvideo_review['unique_id'] = [i + j + k for i, j, k in zip([str(x) for x in video_review.GameKey], [str(x) for x in video_review.PlayID], [str(x) for x in video_review.Season_Year])]\nvideo_footage_injury['unique_id'] = [i + j + k for i, j, k in zip([str(x) for x in video_footage_injury.gamekey], [str(x) for x in video_footage_injury.playid], [str(x) for x in video_footage_injury.season])]\nvideo_footage_control['unique_id'] = [i + j + k for i, j, k in zip([str(x) for x in video_footage_control.gamekey], [str(x) for x in video_footage_control.playid], [str(x) for x in video_footage_control.season])]\npunt_data['unique_id'] = [i + j + k for i, j, k in zip([str(x) for x in punt_data.GameKey], [str(x) for x in punt_data.PlayID], [str(x) for x in punt_data.Season_Year])]\n\ninjury_data = video_review.merge(video_footage_injury, on='unique_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f4ccbbe09a2c4b3519b0041f454397ba979c1625"},"cell_type":"code","source":"data = punt_data.merge(injury_data, on='unique_id', how='outer', suffixes=('', '_drop'))\ndata = data.iloc[:, np.r_[0:15, 19:25]]\ndata.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f466d213bff09e9f5abf58a3df72f226c7b436f"},"cell_type":"markdown","source":"## Exploring the Data"},{"metadata":{"_uuid":"b260c494d6181d77b807691f77117de4338547b2"},"cell_type":"markdown","source":"First, I categorize the most common outcomes of a punt to gain an understanding of what happens when a punt is executed. A return is by far the most common outcome of a punt, occuring on over 40 percent of punt plays. This is a simple yet very important statistic when considering potential rule changes as it illustrates how important punt returns are to the game. In the plot below, note that the \"muffed\" category includes both punts that were dropped immediately by the returner and punts that were returned but then fumbled on the return. Thus, the \"returned\" category only includes punts that were returned without being muffed or fumbled."},{"metadata":{"trusted":true,"_uuid":"4aba2cb1de18ab553c7ed98d45f894dfb8ee975c"},"cell_type":"code","source":"regexp_fair_catch = re.compile('(fair)\\s(catch)')\nfair_catch = [regexp_fair_catch.search(x) != None for x in punt_data.PlayDescription]\ndata[\"fair_catch\"] = fair_catch\n\nregexp_bounds = re.compile('(out)\\s(of)\\s(bounds)\\.')\nout_of_bounds = [regexp_bounds.search(x) != None for x in punt_data.PlayDescription]\ndata[\"out_of_bounds\"] = out_of_bounds\n\nregexp_touchback = re.compile('(Touchback)\\.')\ntouchback = [regexp_touchback.search(x) != None for x in punt_data.PlayDescription]\ndata[\"touchback\"] = touchback\n\nregexp_muffs = re.compile('(MUFFS)|(FUMBLE)|(Fumble)|(fumble)')\nmuffed = [regexp_muffs.search(x) != None for x in punt_data.PlayDescription]\ndata[\"muffed\"] = muffed\n\nregexp_downed = re.compile('(downed)')\ndowned = [regexp_downed.search(x) != None for x in punt_data.PlayDescription]\ndata[\"downed\"] = downed\n\nregexp_blocked = re.compile('(BLOCKED)')\nblocked = [regexp_blocked.search(x) != None for x in punt_data.PlayDescription]\ndata[\"blocked\"] = blocked\n\nregexp_returned = re.compile(\"[a-zA-Z\\.]*\\sto\\s[A-Z]*\\s[0-9]*\\sfor\\s[-0-9]*\\s|[a-zA-Z\\.]*\\s(pushed)\\sob\\sat\\s[A-Z]*\\s[0-9]*\\sfor\\s[-0-9]*\\s|[a-zA-Z\\.]*\\sto\\s[0-9]*\\sfor\\s[-0-9]*\\s|[a-zA-Z\\.]*\\sto\\s[A-Z]*\\s[0-9]*\\sfor\\sno\\sgain|[a-zA-Z\\.]*\\sran\\sob\\sat\\s[A-Z]*\\s[0-9]*\\sfor|[a-zA-Z\\.]*\\spushed\\sob\\sat\\s[A-Z]*\\s[0-9]*\\sfor|[a-zA-Z\\.]*\\sfor\\s[0-9]*\\syards, TOUCHDOWN|[a-zA-Z\\.]*\\spushed\\sob\\sat\\s[0-9]*\\sfor\\s|[a-zA-Z\\.]*\\sran\\sob\\sat\\s[0-9]*\\sfor\\s\")\nreturned = [regexp_returned.search(x) != None for x in punt_data.PlayDescription]\ndata[\"returned\"] = returned & ~np.array(muffed) & ~np.array(fair_catch) & ~np.array(out_of_bounds) & ~np.array(touchback) & ~np.array(downed) & ~np.array(blocked)\n\n### Manually Change Punts that were challenged\n\ndata.loc[0, 'downed'] = False\ndata.loc[155, 'downed'] = False\ndata.loc[161, 'downed'] = False\ndata.loc[661, 'downed'] = False\ndata.loc[1254, 'downed'] = False\ndata.loc[1415, 'downed'] = False\ndata.loc[1649, 'downed'] = False\ndata.loc[1745, 'downed'] = False\ndata.loc[3796, 'downed'] = False","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"148bdd530285ad3cf263e5bf724b7b39951d6326"},"cell_type":"code","source":"count_data = pd.DataFrame({'Type of Outcome': [\"returned\", \"fair_catch\", \"downed\", \"out_of_bounds\", \"touchback\", \"muffed\", \"blocked\"],\n                         'Proportion of Executed Punts' : np.array([sum(data.returned), sum(data.fair_catch), sum(data.downed), sum(data.out_of_bounds),\n                                    sum(data.touchback), sum(data.muffed), sum(data.blocked)])/len(data)})\n\nfig, ax = plt.subplots(figsize=(15, 10))\nplt.title('Common Outcomes of a Punt')\nsns.barplot(x='Type of Outcome', y='Proportion of Executed Punts', data=count_data);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a56c1600368d9518d7f4f20443b4f6d1e4972a7f"},"cell_type":"markdown","source":"Next, one simple notion that I thought was important to address was the type of contact that resulted in a concussion. Interestingly, there is an even mix of helmet-to-helmet and helmet-to-body contact with a very small amount of hemlet-to-ground contact. This is in line with the new rule instituted in Spring 2018 that prevents players from \"lowering the head to initiate contact\". The effects of this rule change are still to be observed, but the initiation of this rule indicates that a new and separate tackling technique rule is unlikely to practical or effective."},{"metadata":{"trusted":true,"_uuid":"afe780c43be4e39aa745c1b90b7b61efe333bf2e"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15, 10))\nplt.title(\"Type of Contact that Resulted in a Concussion\")\ndata.Primary_Impact_Type.value_counts().plot(kind='bar');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"54c4eff2796d76d88b7ed1e1c36a36ed6e99d522"},"cell_type":"markdown","source":"It is also very important when considering rule changes to reduce the occurence of concussions during punt plays to look at the action the concussed players were taking at the time of the injury. Somewhat counterintuitively, most of the concussed players on punts from 2016-2017 were in the process of tackling another player, while the action with the fewest concussions is a player being tackled. However, it is important to remember that there are 11 players on the field that are attempting to tackle the returner (or get blocked in the process), but only one player at a time that is returning the ball and has the potential to be tackled. Still, this plot raises a new question of whether punt returns are really more dangerous than other outcomes of executed punts. This question will be resolved shortly."},{"metadata":{"trusted":true,"_uuid":"15d75e1574ddb37b7340c0614e0030e2c89719f1"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15, 10))\nplt.title(\"Action Taken by Concussed Player\")\ndata.Player_Activity_Derived.value_counts().plot(kind='bar');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"656e34a2ff7a673178fafddcc9561134668297ba"},"cell_type":"markdown","source":"As expected, the vast majority of concussions are not sustained from friendly fire. This is still important as it eliminates the need to created a rule that addresses friendly fire."},{"metadata":{"trusted":true,"_uuid":"af70585b3a0456d5b24b2f6076011b02b71e4421"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15, 10))\nplt.title(\"Occurence of Friendly Fire Resulting in a Concussion\")\ndata.Friendly_Fire.value_counts().plot(kind='bar');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5a8e4f592d9787a2fcfbd6009a2aecc10f46c28a"},"cell_type":"markdown","source":"Most importantly, and to answer the question posed above, the vast majority of concussions were sustained on punt plays where the ball was returned. So to reiterate, **even though most of the players who sustained a concussion were not returning the ball, the majority of concussions were sustained on returned punts**. This motivates the thought, which was to be expected, that punt returns are dangerous. But as noted earlier, removing them entirely would be too invasive to the game, so how could punt returners be incentivized to call fair catch (on which concussions are very unlikely) instead? "},{"metadata":{"trusted":true,"_uuid":"7d2a826d850aaa68713efa4ae51aef97fd41493c"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15, 10))\nplt.title(\"Outcomes of Punts Plays that Resulted in a Concussion  \")\ndata[data.Player_Activity_Derived.notna()].iloc[:, 21:].sum().plot(kind='bar');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"13840d2c5baff030ac12b8ac2586ba1e83a3b54e"},"cell_type":"markdown","source":"To build off of this thought of incentivizing punt returners to call fair catch, it is first important to get an idea of how long the average punt return is. Obviously the distribution has a strong right skew, resulting in a mean larger than the median."},{"metadata":{"trusted":true,"_uuid":"6064bcd729da93dceeda5129118e1710be73bead"},"cell_type":"code","source":"regexp_ret_dist = re.compile('(for\\sno\\sgain)|(for\\s[-0-9]*\\s(yards|yard))')\nret_dist = np.array([regexp_ret_dist.search(x).group(0) for x in data[data.returned].PlayDescription])\n\nret_dist = np.array([re.search('[-0-9]+|no', x).group(0) for x in ret_dist])\nret_dist[ret_dist == 'no'] = 0\nret_dist = [int(x) for x in ret_dist]\nprint(\"Mean Punt Return Distance (yards): \" + str(np.mean(ret_dist)))\nprint(\"Median Punt Return Distance (yards): \" + str(np.median(ret_dist)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3327bc23b9ab6677f1970d60d3b1ce4adaa37b7f"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15, 10))\nplt.title(\"Distribution of Punt Return Distance (yards)\")\nsns.distplot(ret_dist);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c799466422d72fea3e0103fb0ce2270d2b6d6bce"},"cell_type":"markdown","source":"## Rule Change Proposal: Provide x Additional Yards from the Spot of a Fair Catch"},{"metadata":{"_uuid":"15f747294fd164e0c4509fe08f3bab8295cee0dc"},"cell_type":"markdown","source":"Again, fewer returns and more fair catches will reduce the occurence of concussions during punt plays. The thought process should thus be to incentivize punt returners to call fair catch without eliminating punt returns entirely, which as supported above is a significant part of the game. Removing punt returns entirely would be a far too invasive change to the game.\n\nAkin to moving touchbacks from the 20 to the 25, the first rule change that I am proposing is to give either 5 or 10 free yards to the returning team from the spot that a fair catch is called. Providing either 5 or 10 additional yards could be decided by surveying punt returners and/or special teams coaches to determine how much more \"likely\" they would be to call fair catch at any given point on the field if 5 or 10 additional yards would be given. Five free yards would obviously have a less invasive impact on the game, but might not provide enough of an incentive for punt returners. Obviously one of the most direct potential side effects of this rule change would be an increase in scoring. However, even if 10 free yards were given, the additional points scored per game would be [minimal](https://www.cmusportsanalytics.com/nfl-expected-points-nflscrapr-part-1-introduction-expected-points/) and similar for all teams. Even if there was an unexpected increase in points per game, this would not necessarily be a bad change to the game since higher scoring games are often deemed more entertaining.\n\nSome other potential side effects of this rule change include:\n* fewer penalties (fewer returns should result in fewer holding/illegal block in the back penalites, a postive impact on the quality of game)\n* encouraging punters to kick the ball away from punt returners (also a much safer punt outcome than a return)\n* encouraging punt returners to chase after a ball to call fair catch (maintaining the excitement of decision making in punt plays)\n* a decrease in Net Punting Average (but no change in Gross Punting Average)\n\n    \nThis is a very simple and intuitive change that would require further research to determine it's impact. Next I wil explore the Next Gen Stats to see if either blocking schemes or punt formations have a significant impact on concussion rates."},{"metadata":{"_uuid":"e02a60401f5e5c292da90a321bd5e73f48a33d97"},"cell_type":"markdown","source":"## Exploring the Next Gen Stats"},{"metadata":{"_uuid":"760d3a4f5939b4cd6b060dcef9505ace0fbbb65d"},"cell_type":"markdown","source":"The first image that I produce below is the line set formation paired with the path taken by the concussed player for each of the 37 punt plays on which a concussion was sustained. The second image that I generate is the line set formation for each of the 37 video footage control punt plays. \n\nThere are two different types of formational, tackling, or blocking rule changes that can be made: prior to the snap or during the play. In regard to prior to the snap rule changes, the fundamental question is whether there is an underlying difference between the line set formations of the 37 concussion punts and the 37 control punts. The short answer is no. The more precise answer is that there is not enough data to tell. With there being only 37 punts plays that resulted in a concussion out of over 6000, it is not reasonable or practical to assume that the line set formations for those 37 punts is different than the other 6000+ punts.\n\nIn regard to the during the play rule changes, the fundamental question is whether there is an underlying difference between the paths (including contact) taken by the 37 concussed players and the more than 132,000 (22\\*6000) paths taken by other players. Again we run into the issue of not having enough concussion data. There are nonparametric statistical tests that can be used to address this issue, but other than the obvious helmet-to-helmet or helmet-to-body contact, there is unlikely to be any practical blocking or tackling rule changes that can be made. \n\nThat being said, there is an intuitive benefit to reducing the occurence of high speed collisions. These collisions are more likely to cause concussions in addition to other injuries. However, when looking at the rule changes made to reduce high speed collisions on kickoff returns, these are already characteristics present in punting plays. There is no running start, no wedge blocking, and the returning team blockers are required to run down field \"with\" the coverage team. \n\nBecause of this, I find it unlikely for there to exist an additional and still practical rule change in regard to starting formation, tackling, or blocking. Thus, my final proposal to decrease the occurence of concussions sustained on punting plays is the rule change outlined above. \n"},{"metadata":{"trusted":true,"_uuid":"20f4517741660727e50ef9536723e3b6b29eae19"},"cell_type":"code","source":"ngs_conc['unique_id'] = [i + j + k for i, j, k in zip([str(x) for x in ngs_conc.GameKey], [str(x) for x in ngs_conc.PlayID], [str(x) for x in ngs_conc.Season_Year])]\nids = ngs_conc.unique_id.unique()\nline_set = ngs_conc[ngs_conc.Event == \"line_set\"]\n\n### There is no \"line_set\" event for this punt so I use ball_snap instead to get starting formation\nline_set = line_set.append(ngs_conc[(ngs_conc.Event == 'ball_snap') & (ngs_conc.unique_id == '56714072017')])\n\nx_vals = []\ny_vals = []\n\nfor i in np.arange(0, len(ids)):\n    temp = line_set[line_set.unique_id == ids[i]]\n    x_vals.append(temp.x.values)\n    y_vals.append(temp.y.values)\n    \n    \n\nngs_control = pd.read_csv('../input/punt-data/ngs_control.csv')\ncontrol_line_set = ngs_control[ngs_control.Event == 'line_set']\n\n### There is no \"line_set\" event for these punts so I use ball_snap instead to get starting formation\ncontrol_line_set = control_line_set.append(ngs_control[(ngs_control.Event == 'ball_snap') & ((ngs_control.unique_id == 42333332017) | (ngs_control.unique_id == 4276892017))])\n\ncontrol_ids = ngs_control.unique_id.unique()\n\ncontrol_x = []\ncontrol_y = []\n\nfor j in np.arange(0, len(control_ids)):\n    temp = control_line_set[control_line_set.unique_id == control_ids[j]]\n    control_x.append(temp.x.values)\n    control_y.append(temp.y.values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"309153aacb332b5de5dba9c1a646155ad63a6b25"},"cell_type":"code","source":"conc_paths = []\n\nfig, ax = plt.subplots(37, 2, figsize=(15, 150))\n\nfor i in range(len(ids)):\n    ax[i, 0].scatter(x_vals[i],y_vals[i])\n    ax[i, 0].set_xlim(0, 120)\n    ax[i, 0].set_ylim(0, 54)\n    \nfor index, rows in injury_data.iterrows():\n    temp = ngs_conc[(ngs_conc.GSISID == rows.GSISID) & (ngs_conc.unique_id == rows.unique_id)]\n    conc_paths.append(temp)\n    ax[index, 1].scatter(temp.x, temp.y)\n    ax[index, 1].set_xlim(0, 120)\n    ax[index, 1].set_ylim(0, 54)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4698bbeb2acae1216882584b34c50dae4bd059bb"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8, 150))\n\nfor i in range(len(control_ids)):\n    plt.subplot(37, 1, i+1)\n    plt.scatter(control_x[i], control_y[i])\n    plt.xlim(0, 120)\n    plt.ylim(0, 54)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.2"}},"nbformat":4,"nbformat_minor":1}