{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# A Metric for Identifying Individual Performance in Punt Coverage\n\n#### Ethan Lee (University of California, Irvine)\n\nThe focus on the 2021 - 2022 Big Data Bowl is special teams. This notebook submission attempts to create a new special teams metric and subsequently rank teams and players on punt coverage plays\n\n[Introduction](#whatis) - explains the new metric of interest and how it can be used to analyze members of a punt coverage unit\n\n[Read Data](#ReadData) - read the tracking / non-tracking data\n\n[Clean Data](#CleanData) - filter for plays of interest and merge datapoints\n\n[Query Data](#QueryData) - make queries for the new metric of interest\n\n[Visualize Data](#Visualize) - rank teams and players","metadata":{}},{"cell_type":"markdown","source":"<a id=\"whatis\"></a>\n# 1. Introduction: Player Distance to Returner at Punt Receive\n\nOn NFL punt plays, the punt unit will (1) block for the punter and (2) sprint down the field to get to the ball. In order to limit return yards, punt coverage teams want to get down the field as fast as possible to be in the best possible position to force a fair catch, tackle the returner, or down the ball where it lands.\n\nThus, the \"closeness\" of the distance between punt coverage players and the returner at the exact moment the ball is received can be a valuable metric to predict how many yards the returner might gain.\n\n<img src=\"https://www3.pictures.zimbio.com/gi/New+York+Jets+vs+Detroit+Lions+CdGAYEVdzw-x.jpg\" style=\"width:500px;\">","metadata":{}},{"cell_type":"markdown","source":"<a id=\"ReadData\"></a>\n# 2. Read Data\n\nBefore we can do anything, we must read in the data. Note that we will only be using data from the 2018 season.","metadata":{}},{"cell_type":"code","source":"# Useful libraries\nimport math\nimport numpy as np \nimport pandas as pd\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:08:01.369227Z","iopub.execute_input":"2022-01-07T00:08:01.369583Z","iopub.status.idle":"2022-01-07T00:08:01.398033Z","shell.execute_reply.started":"2022-01-07T00:08:01.369473Z","shell.execute_reply":"2022-01-07T00:08:01.397328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pff_data = pd.read_csv('../input/nfl-big-data-bowl-2022/PFFScoutingData.csv')\nplay_data = pd.read_csv('../input/nfl-big-data-bowl-2022/plays.csv')\ngame_data = pd.read_csv('../input/nfl-big-data-bowl-2022/games.csv')\n\n# 2018 tracking data only for this case\ntracking_data = pd.read_csv('../input/nfl-big-data-bowl-2022/tracking2018.csv')","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:08:01.40036Z","iopub.execute_input":"2022-01-07T00:08:01.400928Z","iopub.status.idle":"2022-01-07T00:08:45.10304Z","shell.execute_reply.started":"2022-01-07T00:08:01.400885Z","shell.execute_reply":"2022-01-07T00:08:45.102144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"CleanData\"></a>\n# 3. Clean Data\n\nThis section is broken into four parts: (1) understanding the problem with the given \"punt_received\" event, (2) finding the \"true\" punt received event, (3) merging additional data like distance and return yardage, and (4) building our main dataset\n\n### 3.1 The Problem with the \"punt_received\" Event\n\nWe want to analyze all punt return plays at the moment the returner catches the ball. While there is a given \"punt_received\" event in the tracking data, we want to ensure whether or not that is the correct time frame. Let's test the accuracy of this event by finding the distance between the returner and the ball at that frame.\n\nWe will use Andre Roberts' 78-yard punt return touchdown as an example:\n\nAt the \"punt_received\" event at frame 74, the returner Andre Roberts was 11.4 yards away from the ball - This shows that the given \"punt_received\" label may not be completely accurate for our desired analysis","metadata":{}},{"cell_type":"code","source":"def get_play_tracking(gameId, playId):\n    return tracking_data[(tracking_data['gameId']==gameId)&(tracking_data['playId']==playId)]\n\nandre_roberts_td = get_play_tracking(2018091000, 2626)\n\ndef get_returner_id(gameId, playId):\n    retId = play_data[(play_data['gameId'] == gameId) & (play_data['playId']==playId)].iloc[0].returnerId\n    if type(retId) == str:\n        retId = float(retId.split(';')[0])\n    if pd.isnull(retId):\n        return None\n    return retId\n\nandre_roberts_id = get_returner_id(2018091000, 2626)\n\ndef get_distance(frame_df, playerId, player2Id):\n    player = frame_df[frame_df['nflId'] == playerId].iloc[0]\n    if player2Id == 'football':\n        player2 = frame_df[frame_df['displayName']=='football'].iloc[0]\n    else:\n        player2 = frame_df[frame_df['nflId'] == player2Id].iloc[0]\n    X2 = player2['x']\n    Y2 = player2['y']\n    distance = math.sqrt((player['x']-X2)**2 + (player['y']-Y2)**2)\n    return distance\n\nreceive_df = andre_roberts_td[andre_roberts_td['event'] == 'punt_received']\nreceive_frame = receive_df.iloc[0]['frameId']\nprint(receive_frame, get_distance(receive_df, andre_roberts_id, 'football'))\nreceive_df[receive_df['displayName'].apply(lambda x: x=='Andre Roberts' or x=='football')]","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:08:45.104406Z","iopub.execute_input":"2022-01-07T00:08:45.104801Z","iopub.status.idle":"2022-01-07T00:08:45.205375Z","shell.execute_reply.started":"2022-01-07T00:08:45.104754Z","shell.execute_reply":"2022-01-07T00:08:45.204552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3.2 Finding the \"True\" Receive Time\n\nTo find the \"true\" receive timeframe, we can start at the perceived \"punt_received\" event frame and move on to the next timeframe until the ball is at a reasonably close distance to the returner.","metadata":{}},{"cell_type":"code","source":"def get_true_receive_frame(play_df, returner_id, receive_frame):\n    original = play_df[play_df['frameId']==receive_frame]\n    found = False\n    while True:\n        receive_df = play_df[play_df['frameId']==receive_frame]\n        try:\n            distance = get_distance(receive_df, returner_id, 'football')\n        except:\n            break\n        \n        if distance < 0.6:\n            found = True\n            break\n        \n        receive_frame += 1\n        \n    if found:\n        df = play_df[play_df['frameId']==receive_frame]\n    else:\n        df = original\n    \n    # To identify players of interest = members of coverage team\n    try:\n        punter_team = df[df['position']=='P'].iloc[0]['team']\n    except:\n        punter_team = df[df['position']=='K'].iloc[0]['team']\n    coverageTeam = [1 if i == punter_team else 0 for i in df['team']]\n    df.insert(0, 'isCoverage', coverageTeam, True)\n    \n    # Add teams abbreviations for play\n    game_id = df.iloc[0]['gameId']\n    game_df = game_data[game_data['gameId']==game_id]\n    if len(game_df) > 1:\n        print('ERROR')\n    else:\n        game = game_df.iloc[0]\n    teams = [game['homeTeamAbbr'] if i == 'home' else game['visitorTeamAbbr'] for i in df['team']]\n    df.insert(0, 'teamAbbr', teams, True)\n                                                                                                    \n    return_team = df[df['isCoverage']==0].iloc[0]['teamAbbr']  \n    returnTeams = [return_team] * len(df)\n    df.insert(1, 'returnTeam', returnTeams, True)\n    \n    return df\n\ntrue_receive_df = get_true_receive_frame(andre_roberts_td, andre_roberts_id, receive_frame)\ntrue_receive_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:08:45.20659Z","iopub.execute_input":"2022-01-07T00:08:45.206909Z","iopub.status.idle":"2022-01-07T00:08:45.268488Z","shell.execute_reply.started":"2022-01-07T00:08:45.206872Z","shell.execute_reply":"2022-01-07T00:08:45.267909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following code sample demonstrates why I chose the threshold of 0.6 yards for being a reasonably close distance between returner and ball. By visualizing the distance between the returner and the ball over each time frame, we can tell that there is a clear point in which the distance stays at a low number - the first of those should be the exact time frame for our analysis. \n\nAt the \"true_receive\" frame 80, the returner Andre Roberts was 0.24 yards away from the ball.","metadata":{}},{"cell_type":"code","source":"def graph_returner_to_ball_distance_over_time(play_df, returner_id):\n    distances = []\n    found = False\n    for i in sorted(play_df['frameId'].unique()):\n        frame_df = play_df[play_df['frameId']==i]\n        distance = get_distance(frame_df, returner_id, 'football')\n        if not found and distance < 0.6:\n            index = i\n            found = True\n        distances.append((i, distance))\n    plt.bar([i[0] for i in distances], [i[1] for i in distances])\n    return distances[index-9:index+8]\ngraph_returner_to_ball_distance_over_time(andre_roberts_td, andre_roberts_id)","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:08:45.270278Z","iopub.execute_input":"2022-01-07T00:08:45.270616Z","iopub.status.idle":"2022-01-07T00:08:46.618538Z","shell.execute_reply.started":"2022-01-07T00:08:45.270588Z","shell.execute_reply":"2022-01-07T00:08:46.617675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3.3 Merging Additional Data\n\nNext, we will label the gunners on the play. Gunners are the position group with the sole role to rush toward the returner on a punt to force a fair catch (0 return yards), make a tackle on a return, or down the punt themselves. Since they are the coverage players who are most likely to be closest to the returner on a punt receive, we will be ranking individual gunners and non-gunners later on by the % of plays where they were \"close\" to the returner. ","metadata":{}},{"cell_type":"code","source":"# Label who the gunners are from the pff_data\ndef add_gunners(play_df):\n    game = play_df.iloc[0]['gameId']\n    play = play_df.iloc[0]['playId']\n    gunners = pff_data[(pff_data['gameId']==game) & (pff_data['playId']==play)].iloc[0]['gunners']\n    \n    # the value = nan = there are no recorded gunners\n    if type(gunners) == float:\n        play_df.insert(0, 'gunner', [0]*len(play_df.index), True)\n    else:\n    \n        try:\n            punter_team = play_df[play_df['position']=='P'].iloc[0]['teamAbbr']\n        except:\n            punter_team = play_df[play_df['position']=='K'].iloc[0]['teamAbbr']\n            #raise ValueError(\"Long Snapper not found\")\n        is_gunner = []\n        for team, num in zip(play_df['teamAbbr'], play_df['jerseyNumber']):\n            found = False\n\n            for g in gunners.split('; '):\n                try:\n                    if str(int(num)) == g.split()[1] and team == g.split()[0]:\n                        found = True\n                        break\n                except:\n                    found = False\n                    break\n\n            if found:\n                is_gunner.append(1)\n            else:\n                is_gunner.append(0)\n            \n        play_df.insert(0, 'gunner', is_gunner, True)\n        \n    return play_df\n\ntrue_receive_df = add_gunners(true_receive_df)\ntrue_receive_df[true_receive_df['gunner']==1]","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:08:46.619666Z","iopub.execute_input":"2022-01-07T00:08:46.619912Z","iopub.status.idle":"2022-01-07T00:08:46.661057Z","shell.execute_reply.started":"2022-01-07T00:08:46.619875Z","shell.execute_reply":"2022-01-07T00:08:46.660198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we will add the distances from the players to the returner.","metadata":{}},{"cell_type":"code","source":"# Get data for player distances to the returner at the true receive frame\n# If there is no labeled returner, then find the distance to the ball\ndef add_player_distances(play_df, returner_id):\n    game_id = play_df.iloc[0]['gameId']\n    distances = []\n    for index, player in play_df.iterrows():\n        if player['displayName'] == 'football':\n            continue\n        \n        if returner_id:\n            distances.append(get_distance(play_df, player['nflId'], returner_id))\n        else:\n            distances.append(get_distance(play_df, player['nflId'], 'football'))\n        \n    play_df = play_df[play_df['displayName'] != 'football']\n    play_df.insert(0, 'distanceToReturner', distances, True)\n    return play_df\n    \ntrue_receive_df = add_player_distances(true_receive_df, andre_roberts_id)\ntrue_receive_df.sort_values(by='distanceToReturner').head()","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:08:46.662254Z","iopub.execute_input":"2022-01-07T00:08:46.662473Z","iopub.status.idle":"2022-01-07T00:08:46.72363Z","shell.execute_reply.started":"2022-01-07T00:08:46.662446Z","shell.execute_reply":"2022-01-07T00:08:46.722607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We also bring in an established metric of punt return success: return yardage.","metadata":{}},{"cell_type":"code","source":"def add_return_yardage(play_df):\n    game_id = play_df.iloc[0]['gameId']\n    play_id = play_df.iloc[0]['playId']\n    \n    play = play_data[(play_data['gameId'] == game_id) & (play_data['playId']==play_id)].iloc[0]\n    play_df.insert(0, 'kickReturnYardage', [play['kickReturnYardage']] * len(play_df), True)\n    return play_df\n\ntrue_receive_df = add_return_yardage(true_receive_df)\ntrue_receive_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:08:46.725048Z","iopub.execute_input":"2022-01-07T00:08:46.725448Z","iopub.status.idle":"2022-01-07T00:08:46.758926Z","shell.execute_reply.started":"2022-01-07T00:08:46.725252Z","shell.execute_reply":"2022-01-07T00:08:46.758334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3.4 Obtaining Our Main Dataset\n\nNow we apply this example to all punt returns to obtain our main dataset. This includes muffed punts since a muff (or fumble) counts as an attempted return. ","metadata":{}},{"cell_type":"code","source":"# Takes 1 minute to run for ~850 punts\ndistance_data = pd.DataFrame()\npunts = play_data[(play_data['gameId'].apply(lambda x: str(x)[:4]=='2018'))&(play_data['specialTeamsPlayType']=='Punt')]\npunts = punts[(punts['specialTeamsResult']=='Return')|(punts['specialTeamsResult']=='Muffed')]\npunt_ids = [(i,j) for i, j in zip(punts.gameId, punts.playId)]\n\ncount = 0\nfor game, play in zip(punts['gameId'], punts['playId']):\n    play_df = get_play_tracking(game, play)\n    returner_id = get_returner_id(game, play)\n    # some plays do not have the \"punt_received\" event, so for those we will set a base timeframe at 50 and go from there to find the \"true\" frame\n    try:\n        temp = play_df[play_df['event']=='punt_received']\n        frame = temp.iloc[0]['frameId']\n    except:\n        frame=50\n    true_receive_frame = get_true_receive_frame(play_df, returner_id, frame)\n    true_receive_df = add_gunners(true_receive_frame)\n    true_receive_df = add_player_distances(true_receive_df, returner_id)\n    true_receive_df = add_return_yardage(true_receive_df)\n    distance_data = pd.concat([distance_data, true_receive_df])\n    \ndistance_data","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:08:46.759793Z","iopub.execute_input":"2022-01-07T00:08:46.760457Z","iopub.status.idle":"2022-01-07T00:09:51.805717Z","shell.execute_reply.started":"2022-01-07T00:08:46.760423Z","shell.execute_reply":"2022-01-07T00:09:51.804847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"QueryData\"></a>\n# 4. Query Data\n\nNow we have for each punt return play:\n* Player team\n* Who the return team is\n* Distance from players to returner\n* Kick return yardage\n\n### 4.1 \"Close\" Players\n\nGiven this information, we predict that a \"close\" player is a punt coverage player who has the effect of lowering return yards by attempting to zone out or tackle the returner.\n\nWe will establish the threshold of a \"close\" player being within 10 yards of the returner at the moment the ball is caught.\n\nNow we will find the number of \"close\" coverage players for every punt return.","metadata":{}},{"cell_type":"code","source":"punt_coverage_data = distance_data[distance_data['isCoverage']==1]\n\n# Show that the metric of \"close\" players is related to the play result = punt return yardage\n# For each play, list the return yardage vs the number of \"close\" players to the returner and the number of return yards there were\n\nCLOSE_THRESHOLD = 10\nTEAMS = sorted(punt_coverage_data['teamAbbr'].unique())\n\nunique_plays = punt_coverage_data.groupby(['gameId', 'playId', 'teamAbbr', 'returnTeam']).size().reset_index()\nyards, close_players = [], []\nfor game_id, play_id, team in zip(unique_plays['gameId'], unique_plays['playId'], unique_plays['teamAbbr']):\n    play = punt_coverage_data[(punt_coverage_data['gameId'] == game_id) & (punt_coverage_data['playId']==play_id) & (punt_coverage_data['teamAbbr']==team)]\n    yards.append(play.iloc[0]['kickReturnYardage'])\n    close = play[play['distanceToReturner'].apply(lambda x: x > .5 and x < CLOSE_THRESHOLD)]\n    close_players.append(len(close))\nunique_plays['kickReturnYardage']=yards\nunique_plays['closePlayers']=close_players\nunique_plays.sample(5)","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:09:51.807214Z","iopub.execute_input":"2022-01-07T00:09:51.807619Z","iopub.status.idle":"2022-01-07T00:09:54.515602Z","shell.execute_reply.started":"2022-01-07T00:09:51.80757Z","shell.execute_reply":"2022-01-07T00:09:54.513916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By finding the Pearson Correlation Coefficient between \"close\" players and return yardage, we find that there is an approximately 0% likelihood that the true value of r is 0. That is to say, it is very unlikely that there is no correlation between \"close\" players and return yardage. We can conclude that as the number of \"close\" players increases on a punt return, the return yardage will tend to be lower. ","metadata":{}},{"cell_type":"code","source":"from scipy import stats\nunique_plays['kickReturnYardage'] = unique_plays['kickReturnYardage'].fillna(0)\nstats.pearsonr(unique_plays['closePlayers'], unique_plays['kickReturnYardage'])","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:09:54.517354Z","iopub.execute_input":"2022-01-07T00:09:54.517664Z","iopub.status.idle":"2022-01-07T00:09:55.107086Z","shell.execute_reply.started":"2022-01-07T00:09:54.517622Z","shell.execute_reply":"2022-01-07T00:09:55.106208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we will predict the Expected Return Yards using a simple linear model between the number of \"close\" players and the return yardage. For example, if there are 3 \"close\" players within the returner at the moment the ball is received, then the returner is expected to gain about 6.2 yards. Then we can find Return Yards Over Expected by calculating the actual return yards minus the expected.","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nX = unique_plays.loc[:, 'closePlayers'].values.reshape(-1, 1)\nY = unique_plays.loc[:, 'kickReturnYardage'].values.reshape(-1, 1)\nlinear_regressor = LinearRegression() \nlinear_regressor.fit(X, Y) \nY_pred = linear_regressor.predict(X)\n\nunique_plays['expectedReturn'] = Y_pred\nunique_plays['returnOverExpected'] = unique_plays['kickReturnYardage']-unique_plays['expectedReturn']\n\nplt.scatter(X, Y)\nplt.plot(X, Y_pred, color='red')\nplt.show()\nprint(sorted(unique_plays['expectedReturn'].unique(), reverse=True))","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:09:55.108233Z","iopub.execute_input":"2022-01-07T00:09:55.108448Z","iopub.status.idle":"2022-01-07T00:09:55.501367Z","shell.execute_reply.started":"2022-01-07T00:09:55.108423Z","shell.execute_reply":"2022-01-07T00:09:55.500475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.2 Querying By Team\n\nWe can now take the data for every unique punt return play and group by team to rank the performance of special teams units in 2018. \n\nWe want to cover:\n   * Punt Coverage - how well do teams get close to the opposing returner\n   * Punt Return Coverage - how well do teams' opponents get close to the returner\n   \nFor Punt Coverage:\n * Number of Punts\n * Average \"Close\" Players\n * Opponent Average Return Yardage\n * Opponent Average Return Yardage Over Expected\n * Percentage of Plays with 1+ \"Close\" Players\n * Percentage of Plays with Opposing Return Yardage of Less than 5\n\nFor Punt Return Coverage:\n * Number of Punt Returns\n * Opponent Average \"Close\" Players\n * Average Return Yardage\n * Average Return Yardage Over Expected\n * Percentage of Plays with 1+ Opposing \"Close\" Players\n * Percentage of Plays with Return Yardage of Less than 5","metadata":{}},{"cell_type":"code","source":"num_punt_coverages, avg_close_players, opp_avg_returns, num_punt_receives, opp_avg_close_players, avg_returns = [], [], [], [], [], []\navg_over_expected, opp_avg_over_expected  = [], []\nat_least_one_close, opp_5_yards, opp_at_least_one_close, return_5_yards = [], [], [], []\nfor team in TEAMS:\n    # Punt Coverages\n    df = unique_plays[unique_plays['teamAbbr']==team]\n    num_punt_coverages.append(len(df))\n    avg_close_players.append(df['closePlayers'].mean())\n    opp_avg_returns.append(df['kickReturnYardage'].mean())\n    at_least_one_close.append(len(df[df['closePlayers']>=1])/len(df)*100)\n    opp_5_yards.append(len(df[df['kickReturnYardage']<5])/len(df)*100)\n    \n    opp_avg_over_expected.append(df['returnOverExpected'].mean())\n    \n    # Punt Return Coverages\n    df = unique_plays[unique_plays['returnTeam']==team]\n    num_punt_receives.append(len(df))\n    opp_avg_close_players.append(df['closePlayers'].mean())\n    avg_returns.append(df['kickReturnYardage'].mean())\n    opp_at_least_one_close.append(len(df[df['closePlayers']>=1])/len(df)*100)\n    return_5_yards.append(len(df[df['kickReturnYardage']<5])/len(df)*100)\n    \n    avg_over_expected.append(df['returnOverExpected'].mean())\n\nteam_df = pd.DataFrame({'puntCoverages':num_punt_coverages, 'avgClosePlayers':avg_close_players, 'oppAvgReturns':opp_avg_returns, 'oppAvgOverExpected':opp_avg_over_expected, '%1+Close':at_least_one_close, 'opp%-5Return':opp_5_yards,\n                        'puntReceives':num_punt_receives, 'oppAvgClosePlayers':opp_avg_close_players, 'avgReturns':avg_returns, 'avgOverExpected':avg_over_expected, 'opp%1+Close':opp_at_least_one_close, '%-5Return':return_5_yards})\nteam_df.index = TEAMS\nteam_df","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:09:55.502511Z","iopub.execute_input":"2022-01-07T00:09:55.502732Z","iopub.status.idle":"2022-01-07T00:09:55.643778Z","shell.execute_reply.started":"2022-01-07T00:09:55.502703Z","shell.execute_reply":"2022-01-07T00:09:55.643198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To identify the most relevant data points from above, we will use a heatmap to test correlations between 2 variables. What sticks out in particular is the correlation between the percentage of plays where there is 1+ \"close\" coverage players versus the % of plays where the opposing return yardage is 5 or less: there is a pearson correlation coefficient of .55 with a 0.001% chance that r = 0. ","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\ncorr_df = team_df.corr()\nheatmap = sns.heatmap(corr_df)\n\nprint(stats.pearsonr(team_df['%1+Close'], team_df['opp%-5Return']))","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:09:55.646133Z","iopub.execute_input":"2022-01-07T00:09:55.64687Z","iopub.status.idle":"2022-01-07T00:09:56.160546Z","shell.execute_reply.started":"2022-01-07T00:09:55.646819Z","shell.execute_reply":"2022-01-07T00:09:56.159665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.3 Querying By Individual\n\nFor each individual coverage player with over 10 punt coverages, we find their average distance to the returner (at the moment the ball is caught) and percentage of plays where they were \"close\" to the returner.\n\nWe also find average return yardage they give up, although that cannot be entirely predicted by the performance of a single player.","metadata":{}},{"cell_type":"code","source":"def query_individual_data(min_plays = 10, gunner=None):\n    if gunner == True:\n        individual_df = punt_coverage_data[punt_coverage_data['gunner'] == 1]\n    elif gunner == False:\n        individual_df = punt_coverage_data[punt_coverage_data['gunner'] == 0]\n    elif gunner == None:\n        individual_df = punt_coverage_data\n    total_plays = individual_df.groupby(['nflId']).size()\n    total_plays = total_plays[total_plays >= min_plays]\n\n    avg_distances = []\n    names = []\n    teams = []\n    returns = []\n    percent_close = []\n    for i in total_plays.keys():\n        df = punt_coverage_data[punt_coverage_data['nflId']==i]\n        avg_distances.append(df['distanceToReturner'].mean())\n        names.append(df.iloc[0]['displayName'])\n        teams.append(df['teamAbbr'].unique()[0])\n        returns.append(df['kickReturnYardage'].mean())\n        \n        close_df = df[df['distanceToReturner']<10]\n        percent_close.append(len(close_df)/len(df))\n\n    individual_df = pd.DataFrame({'totalPlays':total_plays, 'avgDistanceToReturner':avg_distances, 'Name':names, \n                                  'Team':teams, 'avgReturnYardage':returns, 'percentClose':percent_close})\n    individual_df = individual_df.sort_values(by='avgReturnYardage')\n    return individual_df\n\nindividual_df = query_individual_data()\nindividual_df = individual_df.sort_values(by='avgReturnYardage').head(10)\nindividual_df","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:09:56.161768Z","iopub.execute_input":"2022-01-07T00:09:56.162029Z","iopub.status.idle":"2022-01-07T00:09:56.683084Z","shell.execute_reply.started":"2022-01-07T00:09:56.161997Z","shell.execute_reply":"2022-01-07T00:09:56.682254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"Visualize\"></a>\n# 5. Visualize Data\n\nVisualization templates from: https://gist.github.com/Deryck97/dff8d33e9f841568201a2a0d5519ac5e\n\n### 5.1 Team Logos Setup\n\nWe will import the team logo images according to the nflfastR-Python guide above.","metadata":{}},{"cell_type":"code","source":"# Import Visualization Libraries\nimport os\nimport urllib.request\nfrom matplotlib.offsetbox import OffsetImage, AnnotationBbox\nfrom PIL import Image\n\n# Create folder for logo images\nlogos_path = os.getcwd() + '/logos/'\nif not os.path.exists(logos_path):\n    os.makedirs(logos_path)\n\nurls = pd.read_csv('https://raw.githubusercontent.com/statsbylopez/BlogPosts/master/nfl_teamlogos.csv')\n\nLOGO_PATHS = {}\nfor i in range(0,len(urls)):\n    team = urls['team_code'].iloc[i]\n    path = logos_path + team + '.png'\n    urllib.request.urlretrieve(urls['url'].iloc[i], path)\n    LOGO_PATHS[team] = path","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:09:56.684211Z","iopub.execute_input":"2022-01-07T00:09:56.684425Z","iopub.status.idle":"2022-01-07T00:10:10.534765Z","shell.execute_reply.started":"2022-01-07T00:09:56.684399Z","shell.execute_reply":"2022-01-07T00:10:10.534173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5.2 Team Rankings: Punt Coverage\n\nFirst we will graph teams based on the average number of players \"close\" to the punt returner on punt coverages\n\nIn other words, every time team X punts, about how many of their special teams players will be within 10 feet of the returner at the moment the returner catches the ball?\n\nBased on the graph, we expect the Colts and Falcons in particular to have good punt coverage (thus having the least amount of avg. return yards) while the Ravens and Rams are expected to have poor punt coverage (the highest amount of avg. return yards). ","metadata":{}},{"cell_type":"code","source":"BASE_SIZE = 100\ndef getImage(path, basewidth=BASE_SIZE, zoom=0.5): \n    if 'BAL.png' in path:\n        basewidth= int(basewidth*1.25)\n    elif 'IND.png' in path:\n        basewidth = int(basewidth*0.9)\n    \n    # Normalizing logo size\n    img = Image.open(path)\n    wpercent = (basewidth/float(img.size[0]))\n    hsize = int((float(img.size[1])*float(wpercent)))\n    img = img.resize((basewidth,hsize), Image.ANTIALIAS)\n    img.save(path, bbox_inches='tight')\n    \n    # this lets you add images onto a chart\n    return OffsetImage(plt.imread(path), zoom=zoom)\n\n# Bar Graph Visualization\nCOLORS = {'ARI':'#97233F','ATL':'#A71930','BAL':'#241773','BUF':'#00338D','CAR':'#0085CA','CHI':'#00143F',\n          'CIN':'#FB4F14','CLE':'#FB4F14','DAL':'#B0B7BC','DEN':'#002244','DET':'#046EB4','GB':'#24423C',\n          'HOU':'#C9243F','IND':'#003D79','JAX':'#136677','KC':'#CA2430','LA':'#002147','LAC':'#2072BA',\n          'LV':'#C4C9CC','MIA':'#0091A0','MIN':'#4F2E84','NE':'#0A2342','NO':'#A08A58','NYG':'#192E6C',\n          'NYJ':'#203731','OAK':'#C4C9CC','PHI':'#014A53','PIT':'#FFC20E','SEA':'#7AC142','SF':'#C9243F','TB':'#D40909',\n          'TEN':'#4095D1','WAS':'#FFC20F'}\n\nCLOSE_FOOTNOTE = '\"Close\" = Within ' + str(CLOSE_THRESHOLD) + ' yards'\n\ndef teams_bar_graph(df, col, title, ycol, footnote='', ascending=False):\n    df = df.sort_values(by=col, ascending=ascending)\n    fig, ax = plt.subplots(figsize=(30,10))\n\n    #Create league average line\n    ax.axhline(y=df[col].mean(), linestyle='--', color='black')\n\n    #Add team logos\n    for x0, y0, path in zip(np.arange(0,32), df[col]+.005, [LOGO_PATHS[i] if i != 'OAK' else LOGO_PATHS['LV'] for i in df.index]):\n        ab = AnnotationBbox(getImage(path), (x0, y0), frameon=False, fontsize=4)\n        ax.add_artist(ab)\n\n    #Add bar chart, x axis is an array from 0-31 (length of 32, one per team)\n    ax.bar(np.arange(0,32), df[col], color=[COLORS[i] for i in df.index], width=.5)\n\n    #Add a grid across the y-axis\n    ax.grid(zorder=0,alpha=.6,axis='y')\n    ax.set_axisbelow(True)\n    ax.set_xticks(np.arange(0,32))\n    #Add team abbreviations as x tick labels\n    ax.set_xticklabels(df.index,fontsize=16)\n\n    ax.set_ylabel(ycol, fontsize=20, labelpad=20)\n    ax.set_title(title, fontsize=26, pad=20)\n    plt.figtext(.8, .06, footnote, fontsize=14)\n    plt.figtext(.8, .04, 'By: Ethan Lee', fontsize=14)\n    plt.text(31, df[col].mean()+.017, 'NFL Average', fontsize=12)\n    \nteams_bar_graph(team_df, 'avgClosePlayers', 'Average Number Of \"Close\" Players Per Punt Coverage', 'Avg. \"Close\" Players', CLOSE_FOOTNOTE)\n# teams_bar_graph(team_df, 'oppAvgOverExpected', 'Average Opposing Punt Return Yardage Over Expected Per Punt Coverage', 'Yards')","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:11:36.663762Z","iopub.execute_input":"2022-01-07T00:11:36.664548Z","iopub.status.idle":"2022-01-07T00:11:37.63773Z","shell.execute_reply.started":"2022-01-07T00:11:36.664506Z","shell.execute_reply":"2022-01-07T00:11:37.636845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the below scatterplot, we see that while the Colts do have the lowest average punt return, teams like the Broncos and Browns do not have the predicted effect of lowering return yardage through a higher number of \"close\" players.","metadata":{}},{"cell_type":"code","source":"# Scatterplot Visualization\ndef teams_scatterplot(df, xcol, ycol, xlabel, ylabel, title, xlow=0, xhigh=30, ylow=0, yhigh=30, extra=0, footnote=''):\n    # Adding logos to the chart\n    fig, ax = plt.subplots(figsize=(20,15))\n    for team, row in df.iterrows():\n        x0 = row[xcol]\n        y0 = row[ycol]\n\n        if team == 'OAK':\n            img = getImage(LOGO_PATHS['LV'], 125) \n        # TEN has a white background - can't seem to get rid of\n        else:\n            img = getImage(LOGO_PATHS[team], 125)\n        ab = AnnotationBbox(img, (x0, y0), xycoords='data', frameon=False, fontsize=4)\n        ax.add_artist(ab)\n\n    plt.xlim(xlow, xhigh)\n    plt.ylim(ylow, yhigh)\n    \n    ax.axvline(x=df[xcol].mean(), linestyle='--', alpha=.5, color='black')\n    ax.axhline(y=df[ycol].mean(), linestyle='--', alpha=.5, color='black')\n\n    # Add a grid\n    ax.grid(zorder=0,alpha=.4)\n    ax.set_axisbelow(True)\n\n    # Adding labels and text\n    ax.set_xlabel(xlabel, fontsize=16)\n    ax.set_ylabel(ylabel, fontsize=16)\n    ax.set_title(title, fontsize=20)\n    plt.figtext(.79,-.008,'By: Ethan Lee',fontsize=12)\n    plt.figtext(.79,.005, footnote, fontsize=12)\n    \n    # Label Logos\n    for x0, y0, name in zip(df[xcol], df[ycol], df.index):\n        plt.text(x=x0+extra , y = y0 , s=f\"{name}\" , fontdict=dict(fontsize=14), zorder=100)\n        plt.tight_layout()\n\n# Average Opponent Return Yards vs Average Number of Close Players\nteams_scatterplot(team_df, 'avgClosePlayers', 'oppAvgReturns', 'Avg. \"Close\" Players', 'Avg. Opp. Return Yards', \n                 'Average Opponent Return Yards vs Average Number of \"Close\" Players', .2, 1.7, 3.9, 12.25, 0.04, footnote=CLOSE_FOOTNOTE)\n\n#teams_scatterplot(team_df, 'avgClosePlayers', 'oppAvgOverExpected', 'Avg. \"Close\" Players', 'Avg. Opp. Return Yards', \n#                  'Average Opponent Return Yards Over Expected vs Average Number of Close Players', 0, 2, -4, 4)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-01-07T00:11:47.915306Z","iopub.execute_input":"2022-01-07T00:11:47.916037Z","iopub.status.idle":"2022-01-07T00:11:49.63393Z","shell.execute_reply.started":"2022-01-07T00:11:47.915996Z","shell.execute_reply":"2022-01-07T00:11:49.63311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we analyze the metrics with the highest correlations found in the heatmap earlier: ","metadata":{}},{"cell_type":"code","source":"teams_scatterplot(team_df, '%1+Close', 'opp%-5Return', '% of Plays with 1+ \"Close\" Players', '% of Plays with Opponent Returns of <5 Yards', \n                  'Percentage of Plays with 1+ \"Close\" Players versus Percentage of Plays with Opponent Returns of <5 Yards' , 20, 90, 20, 65, 1.75, footnote=CLOSE_FOOTNOTE)","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:10:53.865017Z","iopub.execute_input":"2022-01-07T00:10:53.865469Z","iopub.status.idle":"2022-01-07T00:10:55.467594Z","shell.execute_reply.started":"2022-01-07T00:10:53.865436Z","shell.execute_reply":"2022-01-07T00:10:55.466655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5.3 Team Rankings: Punt Return Coverage\n\nThe idea here is the same - as there are more \"close\" opponent players to the returner, there should be lower return yardage. Or, as there are less \"close\" opposing players to the returner, the returner will tend to gain more yardage. Based off of the graph, we expect the 49ers and Giants to have the highest opposing return yardage and the Panthers and Falcons to have the lowest opposing return yardage.","metadata":{}},{"cell_type":"code","source":"teams_bar_graph(team_df, 'oppAvgClosePlayers', 'Average Number Of \"Close\" Opposing Players Per Punt Return', 'Avg. \"Close\" Players', '\"Close\" = Within ' + str(CLOSE_THRESHOLD) + ' yards')","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:13:58.458986Z","iopub.execute_input":"2022-01-07T00:13:58.45927Z","iopub.status.idle":"2022-01-07T00:13:59.297811Z","shell.execute_reply.started":"2022-01-07T00:13:58.45924Z","shell.execute_reply":"2022-01-07T00:13:59.296951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"While the 49ers and Giants do have low average return yards, the Falcons and Panthers also have a below-average average return yards. There seems to be little correlation that the average return yards is affected by the average number of opposing \"close\" players.","metadata":{}},{"cell_type":"code","source":"# Average Return Yards vs Average Number of \"Close\" Opposing Players\nteams_scatterplot(team_df, 'oppAvgClosePlayers', 'avgReturns', 'Avg. \"Close\" Opp. Players', 'Avg. Return Yards', \n                  'Average Return Yards vs Average Number of \"Close\" Opposing Players', .45, 1.4, 3.8, 14, 0.025, footnote=CLOSE_FOOTNOTE)","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:14:18.170637Z","iopub.execute_input":"2022-01-07T00:14:18.170939Z","iopub.status.idle":"2022-01-07T00:14:19.785952Z","shell.execute_reply.started":"2022-01-07T00:14:18.170909Z","shell.execute_reply":"2022-01-07T00:14:19.784908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5.4 Individual Gunner Rankings\n\nHere we rank the top 20 gunners by the average distance from the returner (minimum 10 plays). ","metadata":{}},{"cell_type":"code","source":"# Return Yardage vs Gunner Closing Distance\n\ngunner_df = query_individual_data(min_plays=10, gunner=True)\n\ndef individual_scatterplot(df, xcol, ycol, title, xlab, ylab, close=False):\n    fig, ax = plt.subplots(figsize=(15,15))\n\n    #Create vertical and horizontal lines for averages of each metric\n    x=df[xcol]\n    y=df[ycol]\n    ax.axvline(x=x.mean(), linestyle='--', alpha=.5, color='black')\n    ax.axhline(y=y.mean(), linestyle='--', alpha=.5, color='black')\n\n    #Create scatter plot\n    #s stands for size, the dot size is proportional to the QBs number of plays\n    ax.scatter(x, y, \n               s=40, \n               alpha=.7, \n               color=[COLORS[i] for i in df['Team']])\n\n    #Add grid\n    ax.grid(zorder=0,alpha=.4)\n    ax.set_axisbelow(True)\n\n    #Remove top and right boundary lines\n    ax.spines['right'].set_visible(False)\n    ax.spines['top'].set_visible(False)\n\n    #Add title, labels, and source\n    ax.set_title(title,fontsize=20,pad=15)\n    ax.set_xlabel(xlab,fontsize=16,labelpad=15)\n    ax.set_ylabel(ylab,fontsize=16,labelpad=15)\n    plt.figtext(.78,.01,'By: Ethan Lee',fontsize=12)\n    if close:\n        plt.figtext(.78, .025, CLOSE_FOOTNOTE, fontsize=12)\n\n    # Label Logos\n    for x0, y0, name in zip(x, y, df['Name']):\n        plt.text(x=1.01*x0 , y = y0 , s=f\"{name}\" , fontdict=dict(fontsize=14), zorder=100)\n        plt.tight_layout()\n        \nindividual_scatterplot(gunner_df, 'avgDistanceToReturner', 'avgReturnYardage',\n                      'Average Return Yards Per Gunner vs Gunner Average Distance to Returner',\n                      'Avg. Distance to Returner', 'Avg. Return Yards')","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:15:46.380597Z","iopub.execute_input":"2022-01-07T00:15:46.380915Z","iopub.status.idle":"2022-01-07T00:15:49.879287Z","shell.execute_reply.started":"2022-01-07T00:15:46.380876Z","shell.execute_reply":"2022-01-07T00:15:49.878379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we rank the top 20 gunners by the percentage of plays where they were \"close\" to the returner on punt receive (minimum 10 plays).","metadata":{}},{"cell_type":"code","source":"individual_scatterplot(gunner_df, 'percentClose', 'avgReturnYardage',\n                      'Average Opposing Return Yards vs Percentage of \"Close\" Plays Per Gunner',\n                      '% of Plays \"Close\" To Returner', 'Avg. Opp. Return Yards', True)","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:15:49.881279Z","iopub.execute_input":"2022-01-07T00:15:49.881583Z","iopub.status.idle":"2022-01-07T00:15:53.117379Z","shell.execute_reply.started":"2022-01-07T00:15:49.881545Z","shell.execute_reply":"2022-01-07T00:15:53.116415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we identify the top 20 gunners based off of how often (or the % of plays, minimum 10) they were \"close\" to the returner at the moment the ball was received.","metadata":{}},{"cell_type":"code","source":"def individual_bar_graph(df, col, title, ycol, footnote='', ascending=True):\n    df = df.sort_values(by=col, ascending=True)\n    fig, ax = plt.subplots(figsize=(15,10))\n\n    #Add team logos\n    for y0, x0, path in zip(np.arange(0, len(df)), df[col]+.005, [LOGO_PATHS[i] if i != 'OAK' else LOGO_PATHS['LV'] for i in df['Team']]):\n        ab = AnnotationBbox(getImage(path, 70), (x0, y0), frameon=False, fontsize=4)\n        ax.add_artist(ab)\n\n    #Add bar chart, x axis is an array from 0-31 (length of 32, one per team)\n    ax.barh(df['Name'], df[col], color=[COLORS[i] for i in df['Team']])\n    \n    ax.set_title(title, fontsize=24)\n    ax.set_xlabel(ycol, fontsize=18, labelpad=20)\n                  \n    plt.figtext(.756, .075, footnote, fontsize=12)\n    plt.figtext(.756, .05, 'By: Ethan Lee', fontsize=12)\n    \ntop_gunners = gunner_df.sort_values(by='percentClose', ascending=False).head(20)\nindividual_bar_graph(top_gunners, 'percentClose', 'Gunners Who Were Most Often \"Close\" To The Returner', '% of Punt Plays \"Close\" To Returner', CLOSE_FOOTNOTE)","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:15:53.118514Z","iopub.execute_input":"2022-01-07T00:15:53.118734Z","iopub.status.idle":"2022-01-07T00:15:53.691169Z","shell.execute_reply.started":"2022-01-07T00:15:53.118706Z","shell.execute_reply":"2022-01-07T00:15:53.690272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lastly, here is the ranking for the top 20 non-gunner coverage players, although their percentages are lower because it is harder for them to get \"close\" to the returner when they are first expected to block for the punter. ","metadata":{}},{"cell_type":"code","source":"top_non_gunners = query_individual_data(gunner=False).sort_values(by='percentClose', ascending=False).head(20)\nindividual_bar_graph(top_non_gunners, 'percentClose', 'Non-Gunners Who Were Most Often \"Close\" To The Returner', '% of Punt Plays \"Close\" To Returner', CLOSE_FOOTNOTE)","metadata":{"execution":{"iopub.status.busy":"2022-01-07T00:15:53.693124Z","iopub.execute_input":"2022-01-07T00:15:53.693358Z","iopub.status.idle":"2022-01-07T00:15:54.643132Z","shell.execute_reply.started":"2022-01-07T00:15:53.693329Z","shell.execute_reply":"2022-01-07T00:15:54.642232Z"},"trusted":true},"execution_count":null,"outputs":[]}]}