{"cells":[{"metadata":{"slideshow":{"slide_type":"skip"},"_uuid":"35b133d5b77de3074be3aa84a9297f8ee047a061"},"cell_type":"markdown","source":"# NFL - Punt Data Analytics \n _Kyle Killion, Former NFL Player and aspiring Data Scientist_\n\n<hr>\n\n__Introduction__\n\nThis notebook describes the datasets and variables provided for the analysis of football punt plays.  The data provided for analysis are specific to punt plays during the 2016 to 2017 seasons.  Four different data sources are provided which describe various elements of each punt play and player.  This notebook describes the specifics of each variable contained within the datasets as well as guidelines on the best approach to use for analysis.  \n\n## Data Description\n<hr>\n**Data Relation**<br>\nThe following datasets will be provided for NFL seasons 2016 to 2017.  Each dataset can be merged on the game, play or player level using the provided key variables (Table 1).  GameKey provides a unique identifier for a specific game which is unique across NFL seasons.  PlayID identifies a unique play within a specified GameKey.  GSISID provides a unique identifier for a player across all seasons\n![Data Relation](../input/informativePics/DataRelation.png \"Data Relationship\")\n\n**Game Data**<br>\nGame level data that specifies the type of season (pre, reg, post), week and the hosting city and team.  Each game is uniquely identified across all seasons using GameKey.  \n\n![Game Data](../input/informativePics/GameData.png \"Game Data Variables\")\n<br>\n\n**Play Information** <br>\nPlay level data that describes the type of play, possession team, score and a brief narrative of each play.  Plays are uniquely identified using a its PlayID along with the corresponding GameKey.  PlayIDs are not unique.  \n<br>\n![Play Data](../input/informativePics/PlayData.png \"Play Data Variables\")\n\n**Player Punt Data**<br>\nPlayer level data that specifies the traditional football position for each player.  Each player is identified using his GSISID.  \n<br>\n![Player Punt Data](../input/informativePics/PlayerPuntData.png \"Player Punt Data Variables\")\n\n__Play Player Role Data__<br>\nPlay and player level data that specifies a punt specific player role.  This dataset will specify each player that played in each play.  A player’s role in a play is uniquely defined by the Gamekey PlayID and GSISID.  \n\n![Play Player Role Data](../input/informativePics/PlayPlayerRoleData.png \"Play Player Role Data Variables\")\n\n__Video Review__<br>\nInjury level data that provides a detailed description of the concussion-producing event.  Video Review data are only available in cases in which the injury play can be identified.  Each video review case can be identified using a combination of GameKey, PlayID, and GSISID.  A brief narrative of the play events is provided.\n![Video Review Data](../input/informativePics/VideoReviewData1.png \"Video Review Data Variables\")\n![Video Review Data](../input/informativePics/VideoReviewData2.png \"Video Review Data Variables\")\n\n__NGS__ – __Next Gen Stats__<br>\nPlayer level data that describes the movement of each player during a play.  The NGS data is identified using GameKey, PlayID, and GSISID.  Player data for each play is provided as a function of time (Time) for the duration of the play.  \n![NGSData1 Data](../input/informativePics/NGSData1.png \"Player Movement Data Variables\")\n![NGSData1 Data](../input/informativePics/NGSData2.png \"Player Movement Data Variables\")"},{"metadata":{"slideshow":{"slide_type":"slide"},"trusted":true,"_uuid":"a7baa7ea907c8ecd8ec8858b7f471350c4acc175"},"cell_type":"code","source":"# Import Tools\nimport os\nimport re\nimport glob as glob\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n# KungFu \nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.cluster import SpectralClustering\n\n\n# Use some configs\n%matplotlib inline\nplt.rcParams['figure.dpi'] = 170","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"947f09f18def67ffe18ea68dd9ab308335f4b368"},"cell_type":"markdown","source":"# Exploring Data Set"},{"metadata":{"slideshow":{"slide_type":"slide"},"trusted":true,"_uuid":"475f66310475c16c1c7e54f48499c53cd61a0cd0"},"cell_type":"code","source":"import os\n\nos.chdir('../input/NFL-Punt-Analytics-Competition/')\n\n# Read in data\ndata = pd.read_csv('NGS-2017-post.csv', parse_dates=True)\n\n# Fill the Events Column Forward\ndata.fillna(method='ffill',inplace=True)\n\n# Get Roles\nrole = pd.read_csv('play_player_role_data.csv')\n\n# Get the Concussion data\nevents = pd.read_csv('video_review.csv')\n\n# Take a summary\nprint(data.info(),'\\n\\n\\n')\nprint(role.info(),'\\n\\n\\n')\nprint(events.info(),'\\n\\n\\n')","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"slideshow":{"slide_type":"slide"},"trusted":true,"_uuid":"156fe406a965d32efb13996367474f1dd6a81b26"},"cell_type":"code","source":"# First Merge the movement data with the Player Roles\nposFrame = data.merge(role, how='inner', on=['Season_Year','GameKey','PlayID','GSISID'])\n\nprint(posFrame.info())\nposFrame[(posFrame.PlayID.isin(events.PlayID))].sort_values('Time').head()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"slideshow":{"slide_type":"slide"},"trusted":true,"_uuid":"43a16978207e9c20b6489e4212f85970f14ab994"},"cell_type":"code","source":"# Look to see what Plays Resulted in Concussions\nprint('Number of Plays: ', len(events.PlayID.unique()))\nprint(events.PlayID.unique())\n\n\n\n# Look at all the players involved with Blocking or being Blocked\nevents[(events.Primary_Partner_Activity_Derived == 'Blocking') | (events.Player_Activity_Derived == 'Blocking')].head()","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"slide"},"trusted":true,"_uuid":"b63ce246605faf0f21d16ddc4d42f87893f7f112"},"cell_type":"code","source":"# The Roles and Movement during those particular plays\n# Need to look at the only players involved in concussions\nconFrame = posFrame[(posFrame.PlayID.isin(events.PlayID))]\n\n\nprint('\\nList of the Plays in the Frame\\n')\nconFrame = conFrame[(conFrame.PlayID.isin(events.PlayID)) & (~conFrame['x'].isna())]\nprint(conFrame.PlayID.unique())\nprint('\\nList of the Positions/Role in the Frame\\n')\nprint(conFrame.Role.unique())\nprint('\\n###############################################################\\n')\n\nprint(conFrame.info())\nconFrame.head()","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"slide"},"trusted":true,"_uuid":"519653c8d0d8501f689e1b5096eda23b93211019"},"cell_type":"code","source":"conFrame.loc[:,['x','y']]\\\n.fillna(method='ffill').dropna()\\\n.plot(x='x', y='y',kind='scatter')\n\nendZone1 = [0, -10]\nendZone2 = [100, 110]\n\nplt.axvspan(endZone1[0], endZone1[1], alpha=.5, lw=0, color='red')\nplt.axvspan(endZone2[0], endZone2[1], alpha=.5, lw=0, color='green')\nplt.xticks(np.arange(0, 101, 10))\nplt.grid()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1552d8caa7d763c059ff71df4e49b8c94e2a3d61"},"cell_type":"markdown","source":"# Data preprocessing - working with the whole Data Set"},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"87b1b09c38feef9b4e4506b39f1a8197b0a10a16"},"cell_type":"code","source":"# Get the Concussion data\nevents = pd.read_csv('video_review.csv')\n\n# Get Roles\nrole = pd.read_csv('play_player_role_data.csv')\n\n# Run through all the data here to find all plays of interest\nfileList = glob.glob(os.getcwd() + \"/*.csv\")\n\n# Regex to match for NGS data\nmatch = ['NGS']\n\n# The Frame to build\nbigFrame = pd.DataFrame()\n\n\nfor s in fileList:\n    \n    # Get all the movement NGS Data\n    if (re.findall(r\"(?=(\"+'|'.join(match)+r\"))\", s)):\n        \n        print('Processing %s...' % s.split('\\\\')[-1])\n\n        df = pd.read_csv(s, \n                    parse_dates=['Time'],\n                    infer_datetime_format=True,\n                    dtype = {'Event' : str}) \n        \n        # Carry play Event forward\n        df.Event.fillna(method='ffill', inplace=True)\n        \n        \n        # Filter for only Concussion plays\n        df = df[(df.PlayID.isin(list(events.PlayID.values)))].sort_values('Time')\n        \n        # Now get the players involved\n        df = df[df.GSISID.isin(list(events.GSISID.values))].sort_values('Time')\n\n        \n        # Fill in the Roles/Positions\n        df = df.merge(role, how='inner', on=['Season_Year','GameKey','PlayID','GSISID'])\n        df.reset_index(inplace=True, drop=True)\n        \n        # Concatenate the Data with bigFrame\n        bigFrame = pd.concat([bigFrame, df], sort=False)\n\nprint('\\n\\nFinal DataFrame:\\n\\n', bigFrame.info())\nbigFrame.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"719604aced816737d7bd413a45cff0727da8be96"},"cell_type":"code","source":"# Sanity Check\nprint(sorted([int(x) for x in bigFrame.GSISID.unique()]))\nprint('\\n\\n', sorted(events.GSISID.unique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41e8c22787fb227c2ea1f35d2a3a45a54629c474"},"cell_type":"code","source":"# Merge Events \nbigFrame1 = bigFrame.merge(events, how='inner', on=['Season_Year','GameKey','PlayID','GSISID'])\nbigFrame1 = bigFrame1[bigFrame1.Primary_Partner_GSISID != 'Unclear'].dropna().apply(pd.to_numeric, errors='ignore')\nbigFrame1.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5ccc98254f0cfafce4dd3036858bf3e09c57dc1f"},"cell_type":"code","source":"bigFrame2 = pd.get_dummies(data=bigFrame1, columns=['Role', \n                                                    'Player_Activity_Derived',\n                                                    'Primary_Impact_Type', \n                                                    'Primary_Partner_Activity_Derived'])\n\n# Remove for clustering analytics\ndel bigFrame2['Event']\ndel bigFrame2['Turnover_Related']\ndel bigFrame2['Friendly_Fire']\n\n\nbigFrame2.info()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"a0c078016b9d105eca5aa4d047f19c3ba177955d"},"cell_type":"code","source":"# Normalize Movement data\nscaler = StandardScaler()\nclusterFrame = bigFrame2\nclusterFrame[['dir','dis','o','x','y']] = scaler.fit_transform(bigFrame2[['dir','dis','o','x','y']])\n\nclusterFrame.set_index('Time', inplace=True)\nprint(clusterFrame.info())\nclusterFrame.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"db0e952ed6f67764dc6a6c385d679ee3ab62de5f"},"cell_type":"markdown","source":"# Spectral Clustering"},{"metadata":{"trusted":true,"_uuid":"5081f95acd51dc5ae6dcf4de2903a7943ed278b7"},"cell_type":"code","source":"# Spectral Clustering\nspecCluster = SpectralClustering(n_clusters=4, eigen_solver='arpack', random_state=43, assign_labels='discretize')\n\nclusterFrame['clusters'] = specCluster.fit_predict(clusterFrame.values)\nclusterFrame.info()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a15e663c30627e1ca3c06ba568d63c22ecb0e256"},"cell_type":"markdown","source":"# Cluster Analytics "},{"metadata":{"trusted":true,"_uuid":"6e8d9d2c043552d50c92556f4c4d2343c1974243"},"cell_type":"code","source":"clusterFrame.groupby(['clusters'])['Primary_Impact_Type_Helmet-to-helmet',\n                                  'Primary_Impact_Type_Helmet-to-body']\\\n.agg(['sum','count','max','mean','min','first','last'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c7b7d734b2aeaef7df502b3ace63a50f79bde56"},"cell_type":"markdown","source":"## Primary Partner Activity"},{"metadata":{"trusted":true,"_uuid":"a6202d56b52f984c251f631acaab98cf1a98b68d"},"cell_type":"code","source":"clusterFrame.groupby(['clusters'])['Primary_Partner_Activity_Derived_Blocked',\n                                  'Primary_Partner_Activity_Derived_Blocking']\\\n.agg(['sum','count','max','mean','min','first','last'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"350342f57ca0a29a8fbfb47451ddb131b1d1be77"},"cell_type":"code","source":"clusterFrame.groupby(['clusters'])['Primary_Partner_Activity_Derived_Tackled',\n                                  'Primary_Partner_Activity_Derived_Tackling']\\\n.agg(['sum','count','max','mean','min','first','last'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4bf8186ade4b82061042f674295fd8a016157587"},"cell_type":"markdown","source":"## Player Activity"},{"metadata":{"trusted":true,"_uuid":"af410b285761fd2e3fbce9e623be26cea3647fea"},"cell_type":"code","source":"clusterFrame.groupby(['clusters'])['Player_Activity_Derived_Tackled',\n                                  'Player_Activity_Derived_Tackling']\\\n.agg(['sum','count','max','mean','min','first','last'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"56a40cdb181ee14c723803f287eeaecfd0867934"},"cell_type":"code","source":"clusterFrame.groupby(['clusters'])['Player_Activity_Derived_Blocked',\n                                  'Player_Activity_Derived_Blocking']\\\n.agg(['sum','count','max','mean','min','first','last'])","execution_count":null,"outputs":[]},{"metadata":{"scrolled":false,"trusted":true,"_uuid":"00c43cf9ca43130050edab2d2d229e50ffa43c7d"},"cell_type":"code","source":"# Look through the clusters\n\nsumFrame = pd.DataFrame()\n\nfor cluster in [0,1,2,3]:\n    interestFrame = clusterFrame[clusterFrame.clusters == cluster]\n\n    print('\\n\\n')\n    print('Number of Plays in Cluster %s :' % cluster, len(interestFrame['PlayID'].unique()))\n    print('Plays from Cluster %s : ' % cluster, interestFrame['PlayID'].unique())\n    print('Players from Cluster %s : ' % cluster, interestFrame['GSISID'].unique(), interestFrame.Primary_Partner_GSISID.unique())\n\n    \n    players = interestFrame.Primary_Partner_GSISID.unique()\n    \n    activityFrame = interestFrame[interestFrame.Primary_Partner_GSISID.isin([int(x) for x in players])]\\\n          .loc[:,['PlayID',\n                  'Primary_Partner_GSISID',\n                  'GSISID',\n                  'Player_Activity_Derived_Blocked',\n                  'Player_Activity_Derived_Blocking',             \n                  'Player_Activity_Derived_Tackled',              \n                  'Player_Activity_Derived_Tackling',\n                  'Primary_Partner_Activity_Derived_Blocked',\n                  'Primary_Partner_Activity_Derived_Blocking',\n                  'Primary_Partner_Activity_Derived_Tackled',\n                  'Primary_Partner_Activity_Derived_Tackling',\n                  'Primary_Impact_Type_Helmet-to-helmet',\n                  'Primary_Impact_Type_Helmet-to-body']]\n\n\n    print('\\n\\n')\n    results = activityFrame[activityFrame > 0].groupby(['PlayID','GSISID', 'Primary_Partner_GSISID']).last().sum()\n    print(results)\n    \n    activityFrame.groupby(['PlayID','GSISID', 'Primary_Partner_GSISID'])['Player_Activity_Derived_Blocked',\n                  'Player_Activity_Derived_Blocking',             \n                  'Player_Activity_Derived_Tackled',              \n                  'Player_Activity_Derived_Tackling',\n                  'Primary_Partner_Activity_Derived_Blocked',\n                  'Primary_Partner_Activity_Derived_Blocking',\n                  'Primary_Partner_Activity_Derived_Tackled',\n                  'Primary_Partner_Activity_Derived_Tackling',\n                  'Primary_Impact_Type_Helmet-to-helmet',\n                  'Primary_Impact_Type_Helmet-to-body']\\\n    .last().sum().plot(kind='bar', legend=False)\n    \n    plt.show()\n    \n    filter_cols = [col for col in clusterFrame if col.startswith('Role')]\n    filter_cols.append('clusters')\n    filter_cols.append('PlayID')\n\n    interestFrame.loc[:,filter_cols].groupby(['clusters','PlayID'])\\\n    .last().sum().plot(kind='bar',legend=False)\n    \n    plt.show()\n    \n    sumFrame = sumFrame.append(results, ignore_index=True)\n    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d32880a89fc6e895130c3a00ed75155efaf98159"},"cell_type":"markdown","source":"# Overall Cluster Sums "},{"metadata":{"trusted":true,"_uuid":"ef5d4207d408dff5dd4891fed68c24d4d9a7ee50"},"cell_type":"code","source":"sumFrame.sum().plot(kind='bar')\nprint(sumFrame.loc[:,['Player_Activity_Derived_Tackled',              \n                  'Player_Activity_Derived_Tackling',\n                'Primary_Partner_Activity_Derived_Tackled',\n                  'Primary_Partner_Activity_Derived_Tackling']].sum())\nsumFrame.loc[:,['Player_Activity_Derived_Blocked','Player_Activity_Derived_Blocking',\n                'Primary_Partner_Activity_Derived_Blocked','Primary_Partner_Activity_Derived_Blocking']].sum()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eaff6275f029a9718f38497a04007a57a5bf2e48"},"cell_type":"markdown","source":"# Investigated Instance\n\nhttp://a.video.nfl.com//films/vodzilla/153280/Wing_37_yard_punt-cPHvctKg-20181119_165941654_5000k.mp4"},{"metadata":{"trusted":true,"_uuid":"31a7fcd59ce96fa72b4f0b3fdc5e5434b7ac01c3"},"cell_type":"code","source":"from IPython.display import HTML\n\nHTML('<video width=\"560\" height=\"315\" controls> <source src=\"http://a.video.nfl.com//films/vodzilla/153280/Wing_37_yard_punt-cPHvctKg-20181119_165941654_5000k.mp4\" type=\"video/mp4\"></video>')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b0cf413f080d9b912f777843d96dc10fdafc4b2e"},"cell_type":"markdown","source":"# Conclusion"},{"metadata":{"_uuid":"ce6c4d263d287d7aa061d748530ee6bcf90fc619"},"cell_type":"markdown","source":"After going through the data and deriving an unsupervised Spectral clustering approach, the results were more than interesting to look through. There can be a wide variety of insights drawn and improvements could have been made to possibly strengthen the analysis. For instance, going back and grouping the two sides of the ball from each other so the algorithm could be aware of this relationship. Another would be to introduce punt hang time in seconds into the study since this highly impacts decisions made by players. \n\n__Clustering Summary:__\n\nI selected an arbitrary number of 4 clusters for Spectral Clustering which are as follows:<br>\n\n__Cluster 0__<br>\nThis was a Cluster that favored predominately tackling with 10 plays. These tackling concussions were mainly from the Primary Partner Activity. Also to note that there was zero Primary Partners conducting any blocking. This cluster also preferred more of a helmet-to-body impact along with singling out 3 plays involving the Punt Returner. \n\n__Cluster 1__ <br>\nThis Cluster as well preferred an activity of tackling, however, the activity was significantly derived from the Player Activity Tackling and with 5 out of the 8 total plays in the cluster. \n\n__Cluster 2 __<br>\nThis Cluster was fairly uniform with a slight preference for tackling with 4 out of the 7 plays involving injury.  It was an odd collection of plays on the cluster with long change of field returns and a fake punt which resulted in the punter suffering a concussion on the tackle. Of the tackling, the Primary Partner was heavily more common than the Player Activity. \n\n__Cluster 3__ \nThis Cluster we predominately a blocking/blocked cluster with 5 out of the 7 total plays. It was also noted that 3 of the plays involved wall blocking schemes. The cluster also did not have any Primary Partners conducting any tackles. The PlayID 1683 was a textbook Helmet-to-Helmet defenseless tackler. \n\nBut while considering all that was in front of me, I knew upholding the game was important to me. The first time the rule was enforced on a defenseless receiver was called I became frustrated. How was this not imposed on Special Teams then if we are just going to be doing it just for the Offense? It’s a third of the game and these plays are specifically designed to take someone’s head off. There are Right and Left Wedges/Walls and snipe blocks coming off the heels targeting guys.\n\nSo with this in mind and going through the game film (specifically PlayID 1683 in the investigated instance) and data, the most fair and possible imposed rule is to protect a defenseless tackler from a helmet-to-helmet, especially the guys inside the box. This would impact roughly 50% of the concussions during punts in a significant way. This would also most importantly preserve the game while also maintaining equality of safety during the course of the football game. \n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}